1. Computational Intelligence Semester 2 Neural Networks Lecture 2 out of 4

2. UFCE3H-20-1: Steve Cayzer History of Neural Network Research 1940s: McCulloch & Pitts: Logical Calculus 1960s: Rosenblatt: Perceptron 1980s: Rumelhart, McClelland & Hinton: Multi Layer Perceptrons 1990s: Neural Networks go mainstream

3. UFCE3H-20-1: Steve Cayzer McCulloch and Pitts The Logical Calculus (1940’s) – UNIT (cf neurone) with inputs and ‘all or none’ output – OUTPUT (cf Action Potential) – CONNECTION (cf synapse) - weighted – THRESHOLD = sum of all weighted inputs needs to meet this for unit to fire

4. UFCE3H-20-1: Steve Cayzer McCulloch and Pitts The Logical Calculus – INPUT: 0 or 1 – OUTPUT: 0 or 1 – SYNAPSE = weighted connection: +1 or -1 – THRESHOLD = 0 – BIAS INPUT – clamped at 1

5. UFCE3H-20-1: Steve Cayzer How the logical calculus works OR AND NOR NAND XOR (You had a go at this in practical 1)

6. UFCE3H-20-1: Steve Cayzer But… How do we choose the weights? Can we get the computer to do it for us? … Rosenblatt claimed to have the answer in the 1960s…

7. UFCE3H-20-1: Steve Cayzer The Rosenblatt Perceptron similar to the billiard ball weights can be any real number. However only 1 'layer' – just inputs and units. – units cannot be connected to other units. – So there is 1 unit per output.

8. UFCE3H-20-1: Steve Cayzer The Perceptron Algorithm Rosenblatt proposed an algorithm to get the computer to choose the weights. Once this was done, perceptrons can handle problems – eg image recognition (letters) The idea is to supervise the procedure – give the perceptron a list of inputs and DESIRED outputs. – the perceptron will learn from that and generalize to previously unseen inputs.

9. UFCE3H-20-1: Steve Cayzer Perceptron Training Law ∆ω = ε · i · α change in weight = error X input X learning rate. So this means that – Error = target-actual, can be negative – weights only change when there is an error – It also means that only active inputs are changed. Inactive (x=0) inputs are not changed at all (which makes sense since they did not contribute to the error). See AI illuminated p297+ for a worked example

10. UFCE3H-20-1: Steve Cayzer The Promise of Perceptrons If the perceptron can handle a problem, then the perceptron is guaranteed to find an answer The perceptron convergence theorem Works for OR, AND, NOT and many demonstration problems…

11. UFCE3H-20-1: Steve Cayzer The problem with perceptrons… The Minsky & Papert book 'perceptrons' showed in detail the limitations of perceptrons It only deals with linearly separable tasks. – So cannot deal with XOR… – … and pretty much all real world problems. Rosenblatt was aware of this but didn't know how to fix it … … neural network research went into a decline in the 1970s.

12. UFCE3H-20-1: Steve Cayzer Linear Separability

13. UFCE3H-20-1: Steve Cayzer Multilayer perceptron Ironically, the answer to the Rosenblatt's problem had been discovered by 2 researchers not working in the NN field – Bryson & Ho – in the same year – 1969 – that the perceptrons book came out. In the 1980s this was 'rediscovered' and popularised in a seminal work called 'parallel distributed programming' by Rumelhart, Mclleland and Hinton. They showed how it was possible to extend learning to multiple layers.

14. UFCE3H-20-1: Steve Cayzer Multilayer Perceptron Architecture

15. UFCE3H-20-1: Steve Cayzer Architecture Notes Standard MLP is is feedforward, fully connected 3 layers – 1st (input) is a cheat – it just distributes the inputs to the hidden nodes. – The 2nd is 'hidden' – 3rd is the output layer. Sometimes there are 4 layers – 2 hidden. Many other architectures are also possible – eg recurrent (feedback) – we won't consider those here.

16. UFCE3H-20-1: Steve Cayzer Backpropagation Recall perceptron training law: ∆ω = ε · i · α We know what the output error is (supervised learning) We also know what the input is (just measure network activity in response to the input) So we can apply perceptron training to the output units (more or less) BUT... how do we know the error of the hidden units?

17. UFCE3H-20-1: Steve Cayzer The backpropagation ‘secret sauce’ The intuition is to feed back the error from the output nodes BACK to the hidden layer – (hence backpropagation) follow the distribution of hidden layer outputs. So if you are an output node, then feed back most of your error to the hidden node that gave you the most input.

18. UFCE3H-20-1: Steve Cayzer Some tweaks to get backpropagation to work ‘step’ threshold smoothed out to a sigmoid Output is no longer 0 or 1 – it is a real number between 0 and 1. Search is gradient descent – or greedy search Can handle more difficult problems – BUT convergence no longer guaranteed

19. UFCE3H-20-1: Steve Cayzer Example: NETTalk 1980s: Terrence Sejnowski and Charles Rosenberg.Terrence Sejnowski It is a particularly fascinating neural network because hearing the audio examples of the neural network as it progresses through training seems to progress from a baby babbling to what sounds like a young child reading a kindergarten text, making the occasional mistake, but clearly demonstrating learned the major rules of reading.audio examples Audio: http://www.cnl.salk.edu/ParallelNetsPronounce/nettal k.mp3 http://www.cnl.salk.edu/ParallelNetsPronounce/nettal k.mp3

20. UFCE3H-20-1: Steve Cayzer Example: Prediction of Osteoporosis UWE work (Praminda Caleb-Solly) Uses Kohonen networks (an advanced ANN) to detect whether, from health records, a woman had osteoporosis. Gives a prediction, or warning, to women who had not yet been diagnosed but were nevertheless at risk.... http://www.cems.uwe.ac.uk/~pcalebso/medicalcollab.html

21. UFCE3H-20-1: Steve Cayzer Next Up… More technical details on: – Linear Separability – Perceptron convergence – Neural Network Architecture – Backpropagation