Backpropagation

Multilayer Perceptron R – S1 – S2 – S3 Network

Example

Elementary Decision Boundaries First Boundary: Second Boundary: First Subnetwork

Elementary Decision Boundaries Third Boundary: Fourth Boundary: Second Subnetwork

Total Network

Function Approximation Example Nominal Parameter Values

Nominal Response

Parameter Variations

Multilayer Network

Performance Index Training Set Mean Square Error Vector Case Approximate Mean Square Error (Single Sample) Approximate Steepest Descent

Application to Gradient Calculation Chain Rule Example Application to Gradient Calculation

Gradient Calculation Sensitivity Gradient

Next Step: Compute the Sensitivities (Backpropagation) Steepest Descent s m F ˆ ¶ n - º 1 2 ¼ S = Next Step: Compute the Sensitivities (Backpropagation)

Jacobian Matrix n m 1 + ¶ - 2 ¼ S º

Backpropagation (Sensitivities) The sensitivities are computed by starting at the last layer, and then propagating backwards through the network to the first layer.

Initialization (Last Layer)

Summary Forward Propagation Backpropagation Weight Update

Example: Function Approximation - e + 1-2-1 Network a

Network 1-2-1 Network a p

Initial Conditions

Forward Propagation

Transfer Function Derivatives

Backpropagation

Weight Update

Choice of Architecture 1-3-1 Network i = 1 i = 2 i = 4 i = 8

Choice of Network Architecture 1-2-1 1-3-1 1-4-1 1-5-1

Convergence 5 1 5 3 3 4 2 4 2 1

Generalization 1-2-1 1-9-1