Derivation of a Learning Rule for Perceptrons Neural Networks Single Layer Perceptrons Derivation of a Learning Rule for Perceptrons x1 x2 xm wk1 wk2 wkm .  Adaline (Adaptive Linear Element) Widrow [1962] Goal:

Least Mean Squares (LMS) Neural Networks Single Layer Perceptrons Least Mean Squares (LMS) The following cost function (error function) should be minimized: i : index of data set, the ith data set j : index of input, the jth input

Adaline Learning Rule With then As already obtained before, Defining Neural Networks Single Layer Perceptrons Adaline Learning Rule With then As already obtained before, Weight Modification Rule Defining we can write

Adaline Learning Modes Neural Networks Single Layer Perceptrons Adaline Learning Modes Batch Learning Mode Incremental Learning Mode

Tangent Sigmoid Activation Function Neural Networks Single Layer Perceptrons Tangent Sigmoid Activation Function x1 x2 xm wk1 wk2 wkm .  Goal:

Logarithmic Sigmoid Activation Function Neural Networks Single Layer Perceptrons Logarithmic Sigmoid Activation Function x1 x2 xm wk1 wk2 wkm .  Goal:

Derivation of Learning Rules Neural Networks Single Layer Perceptrons Derivation of Learning Rules For arbitrary activation function,

Derivation of Learning Rules Neural Networks Single Layer Perceptrons Derivation of Learning Rules Depends on the activation function used

Derivation of Learning Rules Neural Networks Single Layer Perceptrons Derivation of Learning Rules Linear function Tangent sigmoid function Logarithmic sigmoid function

Derivation of Learning Rules Neural Networks Single Layer Perceptrons Derivation of Learning Rules

Homework 3  x1 x2 w11 w12 [x1;x2]=[2;3] [x1;x2] =[[2 1];[3 1]] Neural Networks Single Layer Perceptrons Homework 3 Given a neuron with linear activation function (a=0.5), write an m-file that will calculate the weights w11 and w12 so that the input [x1;x2] can match output y1 the best. Use initial values w11=1 and w12=1.5, and η = 0.01. Determine the required number of iterations. Note: Submit the m-file in hardcopy and softcopy. x1 x2 w11 w12  Case 1 [x1;x2]=[2;3] Case 2 [x1;x2] =[[2 1];[3 1]] [y1] =[5] [y1]=[5 2] Odd-numbered Student ID Even-numbered Student ID

Homework 3A  x1 x2 w11 w12 [x1] =[0.2 0.5 0.4] [x2] =[0.5 0.8 0.3] Neural Networks Single Layer Perceptrons Homework 3A Given a neuron with a certain activation function, write an m-file that will calculate the weights w11 and w12 so that the input [x1;x2] can match output y1 the best. Use initial values w11=0.5 and w12=–0.5, and η = 0.01. Determine the required number of iterations. Note: Submit the m-file in hardcopy and softcopy. x1 x2 w11 w12  [x1] =[0.2 0.5 0.4] [x2] =[0.5 0.8 0.3] [y1] =[0.1 0.7 0.9] ? Even Student ID: Tangent sigmoid function Odd Student ID: Logarithmic sigmoid function

MLP Architecture x1 y1 x2 y2 x3 wlk wji wkj Neural Networks Multi Layer Perceptrons MLP Architecture Hidden layers Input layer x1 x2 x3 wji wkj wlk Output layer y1 Inputs Outputs y2 Possesses sigmoid activation functions in the neurons to enable modeling of nonlinearity. Contains one or more “hidden layers”. Trained using the “Backpropagation” algorithm.

MLP Design Consideration Neural Networks Multi Layer Perceptrons MLP Design Consideration What activation functions should be used? How many inputs does the network need? How many hidden layers does the network need? How many hidden neurons per hidden layer? How many outputs should the network have? There is no standard methodology to determine these values. Even there is some heuristic points, final values are determinate by a trial and error procedure.

Neural Networks Multi Layer Perceptrons Advantages of MLP MLP with one hidden layer is a universal approximator. MLP can approximate any function within any preset accuracy The conditions: the weights and the biases are appropriately assigned through the use of adequate learning algorithm. x1 x2 x3 wji wkj wlk MLP can be applied directly in identification and control of dynamic system with nonlinear relationship between input and output. MLP delivers the best compromise between number of parameters, structure complexity, and calculation cost.

Learning Algorithm of MLP Neural Networks Multi Layer Perceptrons Learning Algorithm of MLP Function signal Error signal f(.) Computations at each neuron j: Neuron output, yj Vector of error gradient, ¶E/¶wji Forward propagation “Backpropagation Learning Algorithm” Backward propagation

Backpropagation Learning Algorithm Neural Networks Multi Layer Perceptrons Backpropagation Learning Algorithm If node j is an output node, yi(n) wji(n) netj(n) f(.) yj(n) -1 dj(n) ej(n)

Backpropagation Learning Algorithm Neural Networks Multi Layer Perceptrons Backpropagation Learning Algorithm If node j is a hidden node, yi(n) wji(n) netj(n) f(.) yj(n) wkj(n) netk(n) yk(n) -1 dk(n) ek(n)

MLP Training i j k Left Right Forward Pass Fix wji(n) Compute yj(n) Neural Networks Multi Layer Perceptrons MLP Training i j k Left Right Forward Pass Fix wji(n) Compute yj(n) Backward Pass Calculate dj(n) Update weights wji(n+1) i j k Left Right