CSC 4510 – Machine Learning Dr. Mary-Angela Papalaskari Department of Computing Sciences Villanova University Course website: 7: Introduction to Neural Networks 1 CSC M.A. Papalaskari - Villanova University Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course The Stanford online ML course Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course The Stanford online ML course Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course The Stanford online ML course Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course The Stanford online ML course Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course The Stanford online ML course Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course The Stanford online ML course Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course The Stanford online ML course Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course The Stanford online ML course Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course The Stanford online ML course

Machine learning problems Supervised Learning – Classification – Regression Unsupervised learning Others: Reinforcement learning, recommender systems. Also talk about: Practical advice for applying learning algorithms. CSC M.A. Papalaskari - Villanova University 2

Motivation – part 1 Learning a non-linear function CSC M.A. Papalaskari - Villanova University 3 Some of the slides in this presentation are adapted from: Prof. Frank Klassner’s ML class at Villanova the University of Manchester ML course The Stanford online ML course

You see this: But the camera sees this: What is this?

Computer Vision: Car detection Testing: What is this? Not a car Cars

Learning Algorithm pixel 1 pixel 2 pixel 1 pixel 2 Raw image Cars “Non”-Cars

pixel 1 pixel 2 Raw image Cars “Non”-Cars Learning Algorithm pixel 1 pixel 2

pixel 1 pixel 2 Raw image Cars “Non”-Cars Learning Algorithm pixel 1 pixel 2 Why not apply logistic regression?

Non-linear decision boundaries x1x1 x2x2 Predict “ “ if 1 1 Reminder: logistic regression can do non-linear Add some higher-order terms?

pixel 1 pixel 2 Raw image Cars “Non”-Cars 50 x 50 pixel images→ 2500 pixels (7500 if RGB) pixel 1 intensity pixel 2 intensity pixel 2500 intensity Quadratic features ( ): ≈3 million features Learning Algorithm pixel 1 pixel 2

Motivation – part 2 Take inspiration from the brain CSC M.A. Papalaskari - Villanova University 11

Neural Networks Origins: Algorithms that try to mimic the brain. Was very widely used in 80s and early 90s; popularity diminished in late 90s. Recent resurgence: State-of-the-art technique for many applications

[Roe et al., 1992] Auditory cortex learns to see Auditory Cortex The “one learning algorithm” hypothesis [Roe et al., 1992]

Somatosensory cortex learns to see Somatosensory Cortex The “one learning algorithm” hypothesis [Metin & Frost, 1989]

Seeing with your tongue Human echolocation (sonar) Haptic belt: Direction senseImplanting a 3 rd eye Sensor representations in the brain [BrainPort; Welsh & Blasch, 1997; Nagel et al., 2005; Constantine-Paton & Law, 2009]

Neuron in the brain Input signals sent from other neurons If enough sufficient signals accumulate, the neuron fires a signal. Connection strengths determine how the signals are accumulated

Neurons in the brain [Credit: US National Institutes of Health, National Institute on Aging]

Comparing Carbon & Silicon Human Brain Computational Units: O(10 11 ) neurons Storage Units: O(10 11 ) neurons, O(10 14 ) synapses Cycle Time: O(10 -3 ) sec, Bandwidth: O(10 14 ) bits/sec Neuron Updates/sec: O(10 14 ) Computer Computational Units: 1 CPU, 10 7 gates Storage Units: O(10 11 ) bits RAM, O(10 12 ) bits disk Cycle Time: O(10 -8 ) sec, Bandwidth: O(10 8 ) bits/sec Neuron Updates/sec: O(10 8 )

The neuron model McCulloch & Pitts 1943 Compute weighted sum of inputs and “fire” if above threshold value CSC M.A. Papalaskari - Villanova University 19

input signals ‘ x ’ and coefficients ‘ w ’ are multiplied weights correspond to connection strengths signals are added up – if they are enough, FIRE! incoming signal connection strength activation level output signal

Activation Functions

A neuron can compute….

The perceptron Rosenblatt 1958: Training algorithm Neural Net A single neuron Adjustable synaptic weights CSC M.A. Papalaskari - Villanova University 23

Perceptron learning 1.Initialize weights and thresholds to random numbers between -0.5 and Activate perceptron 3.Update weights: w i (p+1) = w i (p) + α * x i (p) * err(p) 4.Iterate until Convergence

Example: Perceptron learning logical AND 25

Representation Limits for perceptron Linear Separability