1. Introduction to Neural Networks
Holger Wittel

2. Outline Motivation Neural Networks & Topologies
Perceptron
Multi-layered networks
Training
Time dependence
Neural Networks for Filters
Example usages
Summary

3. Motivation
Computers are fast, still certain tasks are difficult for them
Your brain took ~100 cycles (0.1s) to classify the picture as ‘cat‘, what can your computer do in 100 cycles?
brain
computer
No. of processing units
1011
109
Type of processing units
neurons
transistors
Type of calculation
massively parallel
usually serial
Switching time
10-3 sec
10-9 sec
Possible switching operations
1014 per sec
1018 per sec
Table taken from: D.Kriesel: Neuronal Netze,

4. A real Neuron
Dendrite
Axon terminal
Node of Ranvier
Soma
Axon
# neurons
roundworm
302
ant
104
fly
105
mouse
5·106
cat
3·108
human
1011
elephant
2·1011
Schwann
cell
Myelin sheath
Nucleus
SEM image of a neuron, cut open to show sacks of neurotransmitters
Images:

5. Neuron Model ..... Out = resp(Σ inputi · wi) Input1 Input2 Inputi
weight w1
Input2
weight w2
response
function
Out = resp(Σ inputi · wi)
.....
weight wi
Inputi
Most important response functions:
Linear: resp(x)= x
Sigmoid: resp(x)= 𝑒 −𝑥
ReLU: resp(x)=max(0, x)
Sgn: resp(x) = +1 or -1

6. The Perceptron / Input Neurons
Output sigmoid → use for classification
The magic happens in the connections & weights!
Input neurons
Output neurons
weight w1
weight w2
Output =response(Σ inputi · wi)

7. The Bias Neuron -1 Input neurons Output neurons
No inputs, constantly outputs bias
Often omitted in drawings
Implement as input neuron
Input neurons
Output neurons
weight w1
weight w2
weight w3 -1

8. Perceptron Categorises input vectors as being in one of two categories
A single perceptron can be trained to separate inputs into two linearly separable categories
Category 1
Category 2

9. Training a Perceptron Need a training set of input/output pairs
Initialise weights and bias (randomly or to zero)
Calculate output
Adjust the weights and bias in proportion to the difference between actual and expected values
Repeat until termination criteria is reached
Rosenblatt (1962) showed that the weights and bias will converge to fixed values after a finite number of iterations (if the categories are linearly separable)

10. Perceptron Example
We want to classify points in R2 into those points for which y≥x+1 and those for which y<x+1 y
y=x+1 x

11. Perceptron Example Initialise bias/weight vector to (0,0,0)
Input is the point (-1,-1) (below the line) – expressed as (1,-1,-1)
s = 0x1+0x-1+0x-1 = 0
Actual output is sgn(0) = +1
Expected output is -1 (below the line)

12. Perceptron Example Error (expected-actual) is -2
Constant learning rate of 0.25
So new weight vector is (0,0,0)

13. Perceptron Example Error (expected-actual) is -2
Constant learning rate of 0.25
So new weight vector is (0,0,0)

14. Perceptron Example Error (expected-actual) is -2
Constant learning rate of 0.25
So new weight vector is (0,0,0) (-2)

15. Perceptron Example Error (expected-actual) is -2
Constant learning rate of 0.25
So new weight vector is (0,0,0) (-2)(1,-1,-1)

16. Perceptron Example Error (expected-actual) is -2
Constant learning rate of 0.25
So new weight vector is (0,0,0) (-2)(1,-1,-1) = (-0.5,0.5,0.5)

17. Perceptron Example New bias/weight vector is (-0.5,0.5,0.5)
Input is the point (0,2) (above the line) – expressed as (1,0,2)
s = -0.5x1+0.5x0+0.5x2 = 0.5
Actual output is sgn(0.5) = +1
Expected output is +1 (above the line) – no change to weight

18. Perceptron Example
Eventually, this will converge to the correct answer of (-a,-a,a) for some a>0
Generally, we won’t know the correct answer!

19. History: OR / XOR 1958: Perceptron
1969: Minsky & Papert: Perceptron can only classify linearly separable data
This killed nearly all research on NNs for the next 15 years
No XOR as popular example:
XOR i1 i2 or
xor 1 OR 1 1 1 1

20. Multi-layered Networks
Adding a middle (‘hidden‘) layer solves the problem!
At least one nonlinear layer
A 3-layered perceptron can represent virtually anything! -1 -1

21. Feed forward network (fully connected)
Network Topologies
Feed forward network (fully connected)
with shortcuts -1 -1

22. Recurrent network with lateral connections
Network Topologies
Recurrent network with lateral connections -1 -1

23. Recurrent network with direct feedbacks
Network Topologies
Recurrent network with direct feedbacks -1 -1

24. Recurrent network with indirect feedback
Network Topologies
Recurrent network with indirect feedback -1 -1

25. Fully connected network (omitted some connections)
Network Topologies
Fully connected network (omitted some connections)

26. Three-Layer Perceptron
We consider a 3-layer feed forward Perceptron
By far most used NN in practice -1 -1

27. Training How to determine the right connection weights?
Prepare training data set: inputs & desired outputs
Optimize weights :
Backpropagation
Standard text book method
Gradient based
Useful for adaptive networks
May get stuck in local minimum
local minimum
Genetic algorithm
Biology inspired
Finds global minimum
May be slow
global minimum

28. Back-Propagation . Operates similarly to perceptron learning Input
Output

29. Back-Propagation . Inputs are fed forward through the network Input
Output

30. Back-Propagation . Inputs are fed forward through the network Input
Output

31. Back-Propagation . Inputs are fed forward through the network Input
Output

32. Back-Propagation . Inputs are fed forward through the network Input
Output

33. Back-Propagation . Inputs are fed forward through the network Input
Output
Compare to expected

34. Back-Propagation
Errors are propagated back
Input .
Output

35. Back-Propagation
Errors are propagated back
Input .
Output

36. Back-Propagation
Errors are propagated back
Input .
Output

37. Back-Propagation
Errors are propagated back
Input .
Output

38. Back-Propagation
Adjust weights based on errors
Input .
Output

39. Back-Propagation Training
Weights might be updated after each pass or after multiple passes
Need a comprehensive training set
Network cannot be too large for the training set
No guarantees the network will learn
Network design and learning strategies impact the speed and effectiveness of learning

40. Excurs: Genetic Algorithms
Start with random population
Evaluate each individual & select best
Make next generation:
Migration – copy best individuals
Mutation – copy best and randomly change some attributes
Crossover – copy best and mix their attributes
Next Generation

41. Uses of NNs abcdef ghijk ? Pattern recognition: Predicting the future
Face recognition
Character recognition
Sound recognition
Predicting the future
Stock market
Weather
Signal processing
abcdef
ghijk ?
DAX today

42. Example: NETTALK Written language → spoken language
Nontrivial problem:
pronounce ‘a‘ in have, brave, read
Simple Network: 3-layer perceptron
After words trained
Reading a new text
Training
Ref: Sejnowsk, Rosenberg: ‘Parallel Networks that Learn to Pronounce English Text’,
Complex Systems 1 (1987)

43. Problems with Neural Networks
Interpretation of Hidden Layers
Overfitting

44. Interpretation of Hidden Layers
What are the hidden layers doing?!
Feature Extraction
The non-linearities in the feature extraction can make interpretation of the hidden layers very difficult.
This leads to Neural Networks being treated as black boxes.

45. Overfitting in Neural Networks
Neural Networks are especially prone to overfitting.
Recall Perceptron Error
Zero error is possible, but so is more extreme overfitting
Logistic
Regression
Perceptron

47. Perceptron vs. Logistic Regression
Logistic Regression has a hard time eliminating all errors
Errors: 2
Perceptrons often do better. Increased importance of errors.
Errors: 0