1. CS621 : Artificial Intelligence Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 25: Backpropagation and Application

2. The biological neuron has evidence of MLP and BP Pyramidal neuron, from the amygdala (Rupshi et al. 2005) A CA1 pyramidal neuron (Mel et al. 2004)

3. Biological MLP … Computation at SOMA Computation at DENDRITES

4. Backpropagation algorithm Fully connected feed forward network Pure FF network (no jumping of connections over layers) Hidden layers Input layer (n i/p neurons) Output layer (m o/p neurons) j i w ji ….

5. Gradient Descent Equations

6. Backpropagation – for outermost layer

7. Backpropagation for hidden layers Hidden layers Input layer (n i/p neurons) Output layer (m o/p neurons) j i …. k  k is propagated backwards to find value of  j

8. Backpropagation – for hidden layers

9. General Backpropagation Rule General weight updating rule: Where for outermost layer for hidden layers

10. Issues in the training algorithm Greedy Algorithm Always changes weight such that E reduces. The algorithm may get stuck up in a local minimum. If we observe that E is not getting reduced anymore, the following may be the reasons:

11. Issues in the training algorithm contd. 1.Stuck in local minimum. 2.Network paralysis. (High –ve or +ve i/p makes neurons to saturate.) 3. (learning rate) is too small.

12. Diagnostics in action (1) 1) If stuck in local minimum, try the following: Re-initializing the weight vector. Increase the learning rate. Introduce more neurons in the hidden layer.

13. Diagnostics in action (2) 2) Observe the outputs: If they are close to 0 or 1, try the following: 1.Scale the inputs or divide by a normalizing factor. 2.Change the shape and size of the sigmoid.

14. Kolmogorov Statement pertaining to Hidden Layer Design Kolgomorov statement: A feedforward network with three layers (input, output and hidden) with appropriate I/O relation that can vary from neuron to neuron is sufficient to compute any function.  However, More hidden layers reduce the size of individual layers.

15. Momentum factor 1.To increase speed of learning Introduce momentum factor  Accelerates the movement out of the trough.  Dampens the oscillation inside the trough.  Choosing : If is large, we may jump over the global minimum.

16. An application in Medical Domain

17. Expert System for Skin Diseases Diagnosis Bumpiness and scaliness of skin Mostly for symptom gathering and for developing diagnosis skills Not replacing doctor’s diagnosis

18. Architecture of the FF NN 96-20-10 96 input neurons, 20 hidden layer neurons, 10 output neurons Inputs: skin disease symptoms and their parameters –Location, distribution, shape, arrangement, pattern, number of lesions, presence of an active norder, amount of scale, elevation of papuls, color, altered pigmentation, itching, pustules, lymphadenopathy, palmer thickening, results of microscopic examination, presence of herald path, result of dermatology test called KOH

19. Output 10 neurons indicative of the diseases: –psoriasis, pityriasis rubra pilaris, lichen planus, pityriasis rosea, tinea versicolor, dermatophytosis, cutaneous T-cell lymphoma, secondery syphilis, chronic contact dermatitis, soberrheic dermatitis

20. Training data Input specs of 10 model diseases from 250 patients 0.5 if some specific symptom value is not known Trained using standard error backpropagation algorithm

21. Testing Previously unused symptom and disease data of 99 patients Correct diagnosis achieved for 70% of papulosquamous group skin diseases Success rate above 80% for the remaining diseases except for psoriasis psoriasis diagnosed correctly only in 30% of the cases Psoriasis resembles other diseases within the papulosquamous group of diseases, and is somewhat difficult even for specialists to recognise.

22. Explanation capability Rule based systems reveal the explicit path of reasoning through the textual statements Connectionist expert systems reach conclusions through complex, non linear and simultaneous interaction of many units Analysing the effect of a single input or a single group of inputs would be difficult and would yield incor6rect results

23. Explanation contd. The hidden layer re-represents the data Outputs of hidden neurons are neither symtoms nor decisions

25. Discussion Symptoms and parameters contributing to the diagnosis found from the n/w Standard deviation, mean and other tests of significance used to arrive at the importance of contributing parameters The n/w acts as apprentice to the expert