11 1 Backpropagation

11 2 Multilayer Perceptron R – S 1 – S 2 – S 3 Network

11 3 Example

11 4 Elementary Decision Boundaries First Subnetwork First Boundary: Second Boundary:

11 5 Elementary Decision Boundaries Third Boundary: Fourth Boundary: Second Subnetwork

11 6 Total Network

11 7 Function Approximation Example Nominal Parameter Values

11 8 Nominal Response

11 9 Parameter Variations

11 10 Multilayer Network

11 Performance Index Training Set Mean Square Error Vector Case Approximate Mean Square Error (Single Sample) Approximate Steepest Descent

11 12 Chain Rule Example Application to Gradient Calculation

11 13 Gradient Calculation Sensitivity Gradient

11 14 Steepest Descent s m F ˆ  n m   F ˆ  n 1 m  F ˆ  n 2 m   F ˆ  n S m m  = Next Step: Compute the Sensitivities (Backpropagation)

11 15 Jacobian Matrix F Ý m n m  f Ý m n 1 m  0  0 0f Ý m n 2 m  0  00  f Ý m n S m m  =

11 16 Backpropagation (Sensitivities) The sensitivities are computed by starting at the last layer, and then propagating backwards through the network to the first layer.

11 17 Initialization (Last Layer) a i  n i M  a i M  n i M  f M n i M  n i M  f Ý M n i M  === s i M 2t i a i –  –f Ý M n i M  =

11 18 Summary Forward Propagation Backpropagation Weight Update

11 19 Example: Function Approximation Network + - t a e p

11 20 Network Network a p

11 21 Initial Conditions

11 22 Forward Propagation

11 23 Transfer Function Derivatives

11 24 Backpropagation

11 25 Weight Update

11 26 Choice of Architecture Network i = 1i = 2 i = 4i = 8

11 27 Choice of Network Architecture

11 28 Convergence

11 29 Generalization