1. CS621 : Artificial Intelligence Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 25: Backpropagation and NN based IR

2. Backpropagation algorithm Fully connected feed forward network Pure FF network (no jumping of connections over layers) Hidden layers Input layer (n i/p neurons) Output layer (m o/p neurons) j i w ji ….

3. Gradient Descent Equations

4. Backpropagation – for outermost layer

5. Backpropagation for hidden layers Hidden layers Input layer (n i/p neurons) Output layer (m o/p neurons) j i …. k  k is propagated backwards to find value of  j

6. Backpropagation – for hidden layers

7. General Backpropagation Rule General weight updating rule: Where for outermost layer for hidden layers

8. Issues in the training algorithm Greedy Algorithm Always changes weight such that E reduces. The algorithm may get stuck up in a local minimum. If we observe that E is not getting reduced anymore, the following may be the reasons:

9. Issues in the training algorithm contd. 1.Stuck in local minimum. 2.Network paralysis. (High –ve or +ve i/p makes neurons to saturate.) 3. (learning rate) is too small.

10. Diagnostics in action (1) 1) If stuck in local minimum, try the following: Re-initializing the weight vector. Increase the learning rate. Introduce more neurons in the hidden layer.

11. Diagnostics in action (2) 2) Observe the outputs: If they are close to 0 or 1, try the following: 1.Scale the inputs or divide by a normalizing factor. 2.Change the shape and size of the sigmoid.

12. Kolmogorov Statement pertaining to Hidden Layer Design Kolgomorov statement: A feedforward network with three layers (input, output and hidden) with appropriate I/O relation that can vary from neuron to neuron is sufficient to compute any function.  However, More hidden layers reduce the size of individual layers.

13. Momentum factor 1.To increase speed of learning Introduce momentum factor  Accelerates the movement out of the trough.  Dampens the oscillation inside the trough.  Choosing : If is large, we may jump over the global minimum.

14. An application in Medical Domain

15. Expert System for Skin Diseases Diagnosis Bumpiness and scaliness of skin Mostly for symptom gathering and for developing diagnosis skills Not replacing doctor’s diagnosis

16. Architecture of the FF NN 96-20-10 96 input neurons, 20 hidden layer neurons, 10 output neurons Inputs: skin disease symptoms and their parameters –Location, distribution, shape, arrangement, pattern, number of lesions, presence of an active norder, amount of scale, elevation of papuls, color, altered pigmentation, itching, pustules, lymphadenopathy, palmer thickening, results of microscopic examination, presence of herald path, result of dermatology test called KOH

17. Output 10 neurons indicative of the diseases: –psoriasis, pityriasis rubra pilaris, lichen planus, pityriasis rosea, tinea versicolor, dermatophytosis, cutaneous T-cell lymphoma, secondery syphilis, chronic contact dermatitis, soberrheic dermatitis

18. Training data Input specs of 10 model diseases from 250 patients 0.5 if some specific symptom value is not known Trained using standard error backpropagation algorithm

19. Testing Previously unused symptom and disease data of 99 patients Correct diagnosis achieved for 70% of papulosquamous group skin diseases Success rate above 80% for the remaining diseases except for psoriasis psoriasis diagnosed correctly only in 30% of the cases Psoriasis resembles other diseases within the papulosquamous group of diseases, and is somewhat difficult even for specialists to recognise.

20. Explanation capability Rule based systems reveal the explicit path of reasoning through the textual statements Connectionist expert systems reach conclusions through complex, non linear and simultaneous interaction of many units Analysing the effect of a single input or a single group of inputs would be difficult and would yield incor6rect results

21. Explanation contd. The hidden layer re-represents the data Outputs of hidden neurons are neither symtoms nor decisions

23. Discussion Symptoms and parameters contributing to the diagnosis found from the n/w Standard deviation, mean and other tests of significance used to arrive at the importance of contributing parameters The n/w acts as apprentice to the expert

24. Neural Network for IR Slides taken from lecture notes material accompanying the book Modern Information Retrieval, Ricardo Baeza Yates

25. Neural Network for IR: from the work by Wilkinson & Hingston, SIGIR’91 Document Terms Query Terms Documents kaka kbkb kckc kaka kbkb kckc k1k1 ktkt d1d1 djdj d j+1 dNdN

26. Three layers network Query terms issue the first inputs –These inputs propagate accross the network to reach the document nodes Second level of propagation: –Document nodes might themselves generate new iputs which affect the document term nodes –Document term nodes might respond with new inputs of their own

27. Weight Assignments Weight associated with an edge from a query term node ki to a document term node ki

28. Next layer weight assignment Weight associated with an edge from a document term node ki to a document node dj:

29. Document Ranking After the first level of input propagation, the activation level of a document node d j is given by: which is exactly the ranking of the Vector model The feedback has the effect of thesaurus based expansion

30. Terrier Assignment corpus link http://www.cfilt.iitb.ac.in/~ashish/irasgn/trec_wsj _1989.tar.gzhttp://www.cfilt.iitb.ac.in/~ashish/irasgn/trec_wsj _1989.tar.gz 50 topics link http://www.cfilt.iitb.ac.in/~ashish/irasgn/topics.ws j.150-200http://www.cfilt.iitb.ac.in/~ashish/irasgn/topics.ws j.150-200