1. Neural Networks An Introduction
Kasin Prakobwaitayakit
Department of Electrical Engineering
Chiangmai University

2. Brain and Machine The Brain The Machine Pattern Recognition
Association
Complexity
Noise Tolerance
The Machine
Calculation
Precision
Logic

3. The contrast in architecture
The Von Neumann architecture uses a single processing unit;
Tens of millions of operations per second
Absolute arithmetic precision
The brain uses many slow unreliable processors acting in parallel

4. Features of the Brain Ten billion neurons
Average several thousand connections
Hundreds of operations per second
Reliability low
Die off frequently (never replaced)
Compensates for problems by massive parallelism

5. The biological inspiration
The brain has been extensively studied by scientists.
Vast complexity prevents all but rudimentary understanding.
Even the behavior of an individual neuron is extremely complex

6. The biological inspiration
Single “percepts” distributed among many neurons
Localized parts of the brain are responsible for certain well-defined functions (e.g.. vision, motion).
Which features are integral to the brain's performance?
Which are incidentals imposed by the fact of biology?

7. The Structure of Neurones

8. The Structure of Neurones
A neurone has a cell body, a branching input
structure (the dendrIte) and a branching output
structure (th axOn)
Axons connect to dendrites via synapses.
Electro-chemical signals are propagated from the dendritic input, through the cell body, and down the axon to other neurons

9. The Structure of Neurones
A neurone only fires if its input signal exceeds a certain amount (the threshold) in a short time period.
Synapses vary in strength
Good connections allowing a large signal
Slight connections allow only a weak signal.
Synapses can be either excitatory or inhibitory.

10. A Classic Artifical Neuron(1)Sj
f (Sj) Xj ao a1 a2 an +1
wj0
wj1
wj2
wjn

11. A Classic Artifical Neuron(2)
All neurons contain an activation function which determines whether the
signal is strong enough to produce an output.
Shows several functions that could be used as an activation function.

12. Learning
When the output is calculated, the desire output is then given to the program to modify the weights.
After modifications are done, the same inputs given will produce the outputs desired.
Formula :
Weight N = Weight N + learning rate * (Desire Output-Actual Output) * Input N * Weight N

13. Tractable Architectures
Feedforward Neural Networks
Connections in one direction only
Partial biological justification
Complex models with constraints (Hopfield and ART).
Feedback loops included
Complex behaviour, limited by constraining architecture

14. Fig. 1: Multilayer Perceptron
Output Values
Input Signals (External Stimuli)
Output Layer
Adjustable
Weights
Input Layer

15. Types of Layer The input layer. The hidden layer(s).
Introduces input values into the network.
No activation function or other processing.
The hidden layer(s).
Perform classification of features
Two hidden layers are sufficient to solve any problem
Features imply more layers may be better

16. Types of Layer (continued)
The output layer.
Functionally just like the hidden layers
Outputs are passed on to the world outside the neural network.

17. A Simple Model of a Neuron
w1j
w2j
w3j
wij y1 y2 y3 yi O
Each neuron has a threshold value
Each neuron has weighted inputs from other neurons
The input signals form a weighted sum
If the activation level exceeds the threshold, the neuron “fires”

18. An Artificial Neuron O f(x) w1j w2j w3j wij y1 y2 y3 yi
Each hidden or output neuron has weighted input connections from each of the units in the preceding layer.
The unit performs a weighted sum of its inputs, and subtracts its threshold value, to give its activation level.
Activation level is passed through a sigmoid activation function to determine output.

19. Mathematical Definition
Number all the neurons from 1 up to N
The output of the j'th neuron is oj
The threshold of the j'th neuron is qj
The weight of the connection from unit i to unit j is wij
The activation of the j'th unit is aj
The activation function is written as f(x)

20. Mathematical Definition
Since the activation aj is given by the sum of the weighted inputs minus the threshold, we can write:
aj = S ( wijoi ) - qj i
oj = f(aj )

21. Activation functions Transforms neuron’s input into output.
Features of activation functions:
A squashing effect is required
Prevents accelerating growth of activation levels through the network.
Simple and easy to calculate
Monotonically non-decreasing
order-preserving

22. Standard activation functions
The hard-limiting threshold function
Corresponds to the biological paradigm
either fires or not
Sigmoid functions ('S'-shaped curves)
The logistic function
The hyperbolic tangent (symmetrical)
Both functions have a simple differential
Only the shape is important
f(x) = 1
1 + e -ax

23. Training Algorithms
Adjust neural network weights to map inputs to outputs.
Use a set of sample patterns where the desired output (given the inputs presented) is known.
The purpose is to learn to generalize
Recognize features which are common to good and bad exemplars

24. Back-Propagation
A training procedure which allows multi-layer feedforward Neural Networks to be trained;
Can theoretically perform “any” input-output mapping;
Can learn to solve linearly inseparable problems.

25. Activation functions and training
For feedforward networks:
A continuous function can be differentiated allowing gradient-descent.
Back-propagation is an example of a gradient-descent technique.
Reason for prevalence of sigmoid

26. Training versus Analysis
Understanding how the network is doing what it does
Predicting behaviour under novel conditions

27. Applications
The properties of neural networks define where they are useful.
Can learn complex mappings from inputs to outputs, based solely on samples
Difficult to analyze: firm predictions about neural network behavior difficult;
Unsuitable for safety-critical applications.
Require limited understanding from trainer, who can be guided by heuristics.

28. Engine management
The behaviour of a car engine is influenced by a large number of parameters
temperature at various points
fuel/air mixture
lubricant viscosity.
A major company have used neural networks to dynamically tune an engine depending on current settings.

29. Signature recognition
Each person's signature is different.
There are structural similarities which are difficult to quantify.
One company have manufactured a machine which recognizes signatures to within a high level of accuracy.
Considers speed in addition to gross shape.
Makes forgery even more difficult.

30. Sonar target recognition
Distinguish mines from rocks on sea-bed
The neural network is provided with a large number of parameters which are extracted from the sonar signal.
The training set consists of sets of signals from rocks and mines.

31. Stock market prediction
“Technical trading” refers to trading based solely on known statistical parameters; e.g. previous price
Neural networks have been used to attempt to predict changes in prices.
Difficult to assess success since companies using these techniques are reluctant to disclose information.

32. Mortgage assessment Assess risk of lending to an individual.
Difficult to decide on marginal cases.
Neural networks have been trained to make decisions, based upon the opinions of expert underwriters.
Neural network produced a 12% reduction in delinquencies compared with human experts.

33. Types of Problem Pattern Classification Regression
Assign patterns to one of two or more classes
Regression
Predict value of a continuous variable

34. Pattern Classification
Decide which class a particular pattern belongs to.
In the most common case, there are only two classes.
This implies that the neural network is modelling a step-function
The most common use of Neural Networks.

35. Pattern Classification
A feature is a measurement of some kind (a real number).
Corresponds to inputs of neural network.
A pattern is called a feature vector
Points in N-dimensional space.
Bifurcate feature space.
Division is based on sample patterns.

36. Decision boundaries
In simple cases, divide feature space by drawing a hyper-plane across it.
Known as a decision boundary.
Discriminant function: returns different values on opposite sides.
Problems which can be thus classified are linearly separable.

37. Linear Separability Decision Boundary X1 A A A A A A A X2 B B B B B B

38. Nearest neighbour
New pattern is assigned the same class as its nearest neighbour.
Can be improved by taking k nearest neighbours and assigning to the majority

39. Hyper-plane partitions
A single Perceptron (i.e. output unit) with connections from each input can perform, and learn, a linear separation.
Perceptrons have a step function activation.
Units with a sigmoid activation also act as a linear discriminant, if interpreted correctly.
Use activation mid-point

40. Hyper-plane partitions
An extra layer models a convex hull
“An area with no dents in it”
Perceptron models, but can’t learn
Sigmoid function learning of convex hulls
Two layers add convex hulls together
Sufficient to classify anything “sane”.
In theory, further layers add nothing
In practice, extra layers may be better

41. Different Non-Linearly Separable Problems
Types of
Decision Regions
Exclusive-OR
Problem
Classes with
Meshed regions
Most General
Region Shapes
Structure
Single-Layer
Half Plane
Bounded By
Hyperplane A B B A
Two-Layer
Convex Open Or
Closed Regions A B B A
Abitrary
(Complexity
Limited by No.
of Nodes)
Three-Layer A B B A

42. Over-training
With sufficient nodes can classify any training set exactly
May have poor generalisation ability.
Cross-validation with some patterns
Typically 50% of training patterns
Validation set error is checked each epoch
Stop training if validation error goes up

43. Training time How many epochs of training?
Stop if the error fails to improve (has reached a minimum)
Stop if the rate of improvement drops below a certain level
Stop if the error reaches an acceptable level
Stop when a certain number of epochs have passed

44. Rugby players & Ballet dancers2
Height (m)
Ballet? 1 50
100
Weight (Kg)

45. Clustering K means Use Euclidean distance ||x - mean ||
randomly assign each point to 1 of K sets
calculate mean vector of each set
reassign points to set with closed mean vector
repeat until no further changes
Use Euclidean distance ||x - mean ||
Caution - scaling inputs is important

46. MLPs versus RBFs MLPs separate classes using hyper-planes MLP
RBF’s separate classes using hyper-spheres.
MLPs may have one or more hidden layers, RBFs have just one. X2
MLP X1 X2
RBF X1

47. MLPs versus RBIs 2
MLPs use distributed learning, RBFs use localized learning.
RBFs usually require more hidden units to model the problem.
RBFs said to be more robust for novel data and they train faster.
However, RBFs can suffer from ‘curse of dimensionality.

48. 3-Class Problem classifying New Data using MLP
LEGEND:
Class A
Class B
Class C
New Data
(Unknown Class)
Decision
Boundary

49. 3-Class Problem Classifying New Data using RBFN
LEGEND:
Class A
Class B
Class C
New Data
(Unknown Class)
Decision
Boundary

50. Radial Basis Function Architecture
Centre pattern, stored in 1st layer weights
Distance measure, determines how far an input pattern is from the centre.
Gaussian transfer function.
Outputs
2nd layer
weights
Radial
Units
1st layer
weights
Inputs

51. A Radial Basis Function Neuron+1 X1
ck1
ck2 X2
ck3 X3
Euclidean
summation
Ik=||X-ck||
Transfer
function
vk= j (Ik) vk
ckn Xn

52. RBF 3 Stage Training
1. Find cluster centres by e.g. K-means clustering.
2. Find the width of the function (deviation) e.g. K-nearest neighbours.
3. Supervised training phase. Adjust 2nd layer weights to map input patterns onto the known output values.

53. Selecting Radial Centres
Radial Sampling (or sub-sampling)
randomly select centres from training points
K-means centre assignment
k means clustering
Kohonen Self Organising Maps
competitive learning (ref: Kohonen lecture)

54. Calculating Centre Widths
Explicit Assignment
K Nearest Neighbours - need to specify size of K
Isotropic - determined by the number of centres and how spread out they are
d is distance between most distant centres
k is number of centres

55. Supervised phase Optimize second layer weights using the known outputs
linear optimization using Pseudo inverse
Can use backpropagation, quick propagation or delta-bar-delta instead

56. Number of Patterns A simple formula gives a reasonable guideline:
Use w/e training patterns, w = number of weights, e = desired accuracy
Train until the error is less than e/2 on the training set
The mathematical justification is quite complex (omitted!)
In practice, can RARELY meet this criterion

57. Advantage & Disadvantage
Pattern recognition
Solve problems with many inputs
Damage to the network does not screw up the output completely
Disadvantage:
Slow, compare to other computers
Black box model

58. Conclusion
True AI can be achieved, but our understand of human brain and lack of technologies do not enable us to study this field further.
It is highly possible that androids like Data could be created.
What would happen if the androids are superior than us?