1. CSC 4510 – Machine Learning Dr. Mary-Angela Papalaskari Department of Computing Sciences Villanova University Course website: www.csc.villanova.edu/~map/4510/ 7: Introduction to Neural Networks 1 CSC 4510 - M.A. Papalaskari - Villanova University Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course http://www.cs.manchester.ac.uk/ugt/COMP24111/http://www.cs.manchester.ac.uk/ugt/COMP24111/ The Stanford online ML course http://www.ml-class.org/http://www.ml-class.org/ Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course http://www.cs.manchester.ac.uk/ugt/COMP24111/http://www.cs.manchester.ac.uk/ugt/COMP24111/ The Stanford online ML course http://www.ml-class.org/http://www.ml-class.org/ Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course http://www.cs.manchester.ac.uk/ugt/COMP24111/http://www.cs.manchester.ac.uk/ugt/COMP24111/ The Stanford online ML course http://www.ml-class.org/http://www.ml-class.org/ Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course http://www.cs.manchester.ac.uk/ugt/COMP24111/http://www.cs.manchester.ac.uk/ugt/COMP24111/ The Stanford online ML course http://www.ml-class.org/http://www.ml-class.org/ Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course http://www.cs.manchester.ac.uk/ugt/COMP24111/http://www.cs.manchester.ac.uk/ugt/COMP24111/ The Stanford online ML course http://www.ml-class.org/http://www.ml-class.org/ Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course http://www.cs.manchester.ac.uk/ugt/COMP24111/http://www.cs.manchester.ac.uk/ugt/COMP24111/ The Stanford online ML course http://www.ml-class.org/http://www.ml-class.org/ Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course http://www.cs.manchester.ac.uk/ugt/COMP24111/http://www.cs.manchester.ac.uk/ugt/COMP24111/ The Stanford online ML course http://www.ml-class.org/http://www.ml-class.org/ Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course http://www.cs.manchester.ac.uk/ugt/COMP24111/http://www.cs.manchester.ac.uk/ugt/COMP24111/ The Stanford online ML course http://www.ml-class.org/http://www.ml-class.org/ Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course http://www.cs.manchester.ac.uk/ugt/COMP24111/http://www.cs.manchester.ac.uk/ugt/COMP24111/ The Stanford online ML course http://www.ml-class.org/http://www.ml-class.org/

2. Machine learning problems Supervised Learning – Classification – Regression Unsupervised learning Others: Reinforcement learning, recommender systems. Also talk about: Practical advice for applying learning algorithms. CSC 4510 - M.A. Papalaskari - Villanova University 2

3. Motivation – part 1 Learning a non-linear function CSC 4510 - M.A. Papalaskari - Villanova University 3 Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course http://www.cs.manchester.ac.uk/ugt/COMP24111/http://www.cs.manchester.ac.uk/ugt/COMP24111/ The Stanford online ML course http://www.ml-class.org/http://www.ml-class.org/

4. You see this: But the camera sees this: What is this?

5. Computer Vision: Car detection Testing: What is this? Not a car Cars

6. Learning Algorithm pixel 1 pixel 2 pixel 1 pixel 2 Raw image Cars “Non”-Cars

7. pixel 1 pixel 2 Raw image Cars “Non”-Cars Learning Algorithm pixel 1 pixel 2

8. pixel 1 pixel 2 Raw image Cars “Non”-Cars Learning Algorithm pixel 1 pixel 2 Why not apply logistic regression?

9. Non-linear decision boundaries x1x1 x2x2 Predict “ “ if 1 1 Reminder: logistic regression can do non-linear Add some higher-order terms?

10. pixel 1 pixel 2 Raw image Cars “Non”-Cars 50 x 50 pixel images→ 2500 pixels (7500 if RGB) pixel 1 intensity pixel 2 intensity pixel 2500 intensity Quadratic features ( ): ≈3 million features Learning Algorithm pixel 1 pixel 2

11. Motivation – part 2 Take inspiration from the brain CSC 4510 - M.A. Papalaskari - Villanova University 11

12. Neural Networks Origins: Algorithms that try to mimic the brain. Was very widely used in 80s and early 90s; popularity diminished in late 90s. Recent resurgence: State-of-the-art technique for many applications

13. [Roe et al., 1992] Auditory cortex learns to see Auditory Cortex The “one learning algorithm” hypothesis [Roe et al., 1992]

14. Somatosensory cortex learns to see Somatosensory Cortex The “one learning algorithm” hypothesis [Metin & Frost, 1989]

15. Seeing with your tongue Human echolocation (sonar) Haptic belt: Direction senseImplanting a 3 rd eye Sensor representations in the brain [BrainPort; Welsh & Blasch, 1997; Nagel et al., 2005; Constantine-Paton & Law, 2009]

16. Neuron in the brain Input signals sent from other neurons If enough sufficient signals accumulate, the neuron fires a signal. Connection strengths determine how the signals are accumulated

17. Neurons in the brain [Credit: US National Institutes of Health, National Institute on Aging]

18. Comparing Carbon & Silicon Human Brain Computational Units: O(10 11 ) neurons Storage Units: O(10 11 ) neurons, O(10 14 ) synapses Cycle Time: O(10 -3 ) sec, Bandwidth: O(10 14 ) bits/sec Neuron Updates/sec: O(10 14 ) Computer Computational Units: 1 CPU, 10 7 gates Storage Units: O(10 11 ) bits RAM, O(10 12 ) bits disk Cycle Time: O(10 -8 ) sec, Bandwidth: O(10 8 ) bits/sec Neuron Updates/sec: O(10 8 )

19. The neuron model McCulloch & Pitts 1943 Compute weighted sum of inputs and “fire” if above threshold value CSC 4510 - M.A. Papalaskari - Villanova University 19

20. input signals ‘ x ’ and coefficients ‘ w ’ are multiplied weights correspond to connection strengths signals are added up – if they are enough, FIRE! incoming signal connection strength activation level output signal

21. Activation Functions

22. A neuron can compute….

23. The perceptron Rosenblatt 1958: Training algorithm Neural Net A single neuron Adjustable synaptic weights CSC 4510 - M.A. Papalaskari - Villanova University 23

24. Perceptron learning 1.Initialize weights and thresholds to random numbers between -0.5 and 0.5 2.Activate perceptron 3.Update weights: w i (p+1) = w i (p) + α * x i (p) * err(p) 4.Iterate until Convergence

25. Example: Perceptron learning logical AND 25

26. Representation Limits for perceptron Linear Separability