2. Derivation of a Learning Rule for Perceptrons
Neural Networks
Single Layer Perceptrons
Derivation of a Learning Rule for Perceptrons x1 x2 xm
wk1
wk2
wkm . 
Adaline
(Adaptive Linear Element)
Widrow [1962]
Goal:

3. Least Mean Squares (LMS)
Neural Networks
Single Layer Perceptrons
Least Mean Squares (LMS)
The following cost function (error function) should be minimized:
i : index of data set, the ith data set
j : index of input, the jth input

4. Adaline Learning Rule With then As already obtained before, Defining
Neural Networks
Single Layer Perceptrons
Adaline Learning Rule
With
then
As already obtained before,
Weight Modification Rule
Defining
we can write

5. Adaline Learning Modes
Neural Networks
Single Layer Perceptrons
Adaline Learning Modes
Batch Learning Mode
Incremental Learning Mode

6. Tangent Sigmoid Activation Function
Neural Networks
Single Layer Perceptrons
Tangent Sigmoid Activation Function x1 x2 xm
wk1
wk2
wkm . 
Goal:

7. Logarithmic Sigmoid Activation Function
Neural Networks
Single Layer Perceptrons
Logarithmic Sigmoid Activation Function x1 x2 xm
wk1
wk2
wkm . 
Goal:

8. Derivation of Learning Rules
Neural Networks
Single Layer Perceptrons
Derivation of Learning Rules
For arbitrary activation function,

9. Derivation of Learning Rules
Neural Networks
Single Layer Perceptrons
Derivation of Learning Rules
Depends on the activation function used

10. Derivation of Learning Rules
Neural Networks
Single Layer Perceptrons
Derivation of Learning Rules
Linear function
Tangent sigmoid
function
Logarithmic sigmoid
function

11. Derivation of Learning Rules
Neural Networks
Single Layer Perceptrons
Derivation of Learning Rules

12. Homework 3  x1 x2 w11 w12 [x1;x2]=[2;3] [x1;x2] =[[2 1];[3 1]]
Neural Networks
Single Layer Perceptrons
Homework 3
Given a neuron with linear activation function (a=0.5), write an m-file that will calculate the weights w11 and w12 so that the input [x1;x2] can match output y1 the best.
Use initial values w11=1 and w12=1.5, and η = 0.01.
Determine the required number of iterations.
Note: Submit the m-file in hardcopy and softcopy. x1 x2
w11
w12 
Case 1
[x1;x2]=[2;3]
Case 2
[x1;x2] =[[2 1];[3 1]]
[y1] =[5]
[y1]=[5 2]
Odd-numbered Student ID
Even-numbered Student ID

13. Homework 3A  x1 x2 w11 w12 [x1] =[0.2 0.5 0.4] [x2] =[0.5 0.8 0.3]
Neural Networks
Single Layer Perceptrons
Homework 3A
Given a neuron with a certain activation function, write an m-file that will calculate the weights w11 and w12 so that the input [x1;x2] can match output y1 the best.
Use initial values w11=0.5 and w12=–0.5, and η = 0.01.
Determine the required number of iterations.
Note: Submit the m-file in hardcopy and softcopy. x1 x2
w11
w12 
[x1] =[ ]
[x2] =[ ]
[y1] =[ ] ?
Even Student ID: Tangent sigmoid function
Odd Student ID: Logarithmic sigmoid function

14. MLP Architecture x1 y1 x2 y2 x3 wlk wji wkj
Neural Networks
Multi Layer Perceptrons
MLP Architecture
Hidden layers
Input
layer x1 x2 x3
wji
wkj
wlk
Output
layer y1
Inputs
Outputs y2
Possesses sigmoid activation functions in the neurons to enable modeling of nonlinearity.
Contains one or more “hidden layers”.
Trained using the “Backpropagation” algorithm.

15. MLP Design Consideration
Neural Networks
Multi Layer Perceptrons
MLP Design Consideration
What activation functions should be used?
How many inputs does the network need?
How many hidden layers does the network need?
How many hidden neurons per hidden layer?
How many outputs should the network have?
There is no standard methodology to determine these values. Even there is some heuristic points, final values are determinate by a trial and error procedure.

16. Neural Networks
Multi Layer Perceptrons
Advantages of MLP
MLP with one hidden layer is a universal approximator.
MLP can approximate any function within any preset accuracy
The conditions: the weights and the biases are appropriately assigned through the use of adequate learning algorithm. x1 x2 x3
wji
wkj
wlk
MLP can be applied directly in identification and control of dynamic system with nonlinear relationship between input and output.
MLP delivers the best compromise between number of parameters, structure complexity, and calculation cost.

17. Learning Algorithm of MLP
Neural Networks
Multi Layer Perceptrons
Learning Algorithm of MLP
Function signal
Error signal
f(.)
Computations at each neuron j:
Neuron output, yj
Vector of error gradient, ¶E/¶wji
Forward propagation
“Backpropagation
Learning Algorithm”
Backward propagation

18. Backpropagation Learning Algorithm
Neural Networks
Multi Layer Perceptrons
Backpropagation Learning Algorithm
If node j is an output node,
yi(n)
wji(n)
netj(n)
f(.)
yj(n) -1
dj(n)
ej(n)

19. Backpropagation Learning Algorithm
Neural Networks
Multi Layer Perceptrons
Backpropagation Learning Algorithm
If node j is a hidden node,
yi(n)
wji(n)
netj(n)
f(.)
yj(n)
wkj(n)
netk(n)
yk(n) -1
dk(n)
ek(n)

20. MLP Training i j k Left Right Forward Pass Fix wji(n) Compute yj(n)
Neural Networks
Multi Layer Perceptrons
MLP Training i j k
Left
Right
Forward Pass
Fix wji(n)
Compute yj(n)
Backward Pass
Calculate dj(n)
Update weights wji(n+1) i j k
Left
Right