1. © Copyright 2004 ECE, UM-Rolla. All rights reserved A Brief Overview of Neural Networks By Rohit Dua, Samuel A. Mulder, Steve E. Watkins, and Donald C. Wunsch Copyright Notice: © Copyright 2004 Electrical and Computer Engineering Department, University of Missouri-Rolla. All rights reserved. Permission is freely given to receive and store this material for personal educational use by educational institutions only. Not to be reproduced, linked, distributed, or sold in any form or media without express written permission of the authors

2. © Copyright 2004 ECE, UM-Rolla. All rights reserved Overview Relation to Biological Brain: Biological Neural Network The Artificial Neuron Types of Networks and Learning Techniques Supervised Learning & Backpropagation Training Algorithm Learning by Example Applications Questions

3. © Copyright 2004 ECE, UM-Rolla. All rights reserved Biological Neuron

4. © Copyright 2004 ECE, UM-Rolla. All rights reserved Artificial Neuron Σ f(n) W W W W Outputs Activation Function INPUTSINPUTS W =Weight Neuron

5. © Copyright 2004 ECE, UM-Rolla. All rights reserved Transfer Functions 1 0 Input Output

6. © Copyright 2004 ECE, UM-Rolla. All rights reserved Types of networks Multiple Inputs and Single Layer Multiple Inputs and layers

7. © Copyright 2004 ECE, UM-Rolla. All rights reserved Types of Networks – Contd. Feedback Recurrent Networks

8. © Copyright 2004 ECE, UM-Rolla. All rights reserved Learning Techniques Supervised Learning: Inputs from the environment Neural Network Actual System Σ Error + - Expected Output Actual Output Training

9. © Copyright 2004 ECE, UM-Rolla. All rights reserved Multilayer Perceptron Inputs First Hidden layer Second Hidden Layer Output Layer

10. © Copyright 2004 ECE, UM-Rolla. All rights reserved Signal Flow Backpropagation of Errors Function Signals Error Signals

11. © Copyright 2004 ECE, UM-Rolla. All rights reserved Learning by Example Hidden layer transfer function: Sigmoid function = F(n)= 1/(1+exp(-n)), where n is the net input to the neuron. Derivative= F’(n) = (output of the neuron)(1- output of the neuron) : Slope of the transfer function. Output layer transfer function: Linear function= F(n)=n; Output=Input to the neuron Derivative= F’(n)= 1

12. © Copyright 2004 ECE, UM-Rolla. All rights reserved Learning by Example Training Algorithm: backpropagation of errors using gradient descent training. Colors: –Red: Current weights –Orange: Updated weights –Black boxes: Inputs and outputs to a neuron –Blue: Sensitivities at each layer

13. © Copyright 2004 ECE, UM-Rolla. All rights reserved First Pass 0.5 1 0.6225 0.6508 Error=1-0.6508=0.3492 G3=(1)(0.3492)=0.3492 G2= (0.6508)(1- 0.6508)(0.3492)(0.5)=0.0397 G1= (0.6225)(1- 0.6225)(0.0397)(0.5)(2)=0.0093 Gradient of the neuron= G =slope of the transfer function ×[Σ{ (weight of the neuron to the next neuron) × ( output of the neuron)}] Gradient of the output neuron = slope of the transfer function × error

14. © Copyright 2004 ECE, UM-Rolla. All rights reserved Weight Update 1 New Weight=Old Weight + {(learning rate)(gradient)(prior output)} 0.5+(0.5)(0.3492)(0.6508) 0.6136 0.5124 0.6136 0.5124 0.5047 0.5+(0.5)(0.0397)(0.6225) 0.5+(0.5)(0.0093)(1)

15. © Copyright 2004 ECE, UM-Rolla. All rights reserved Second Pass 0.5047 0.5124 0.6136 0.5047 0.5124 1 0.5047 0.6391 0.6236 0.8033 0.6545 0.8033 Error=1-0.8033=0.1967 G3=(1)(0.1967)=0.1967 G2= (0.6545)(1- 0.6545)(0.1967)(0.6136)=0.0273 G1= (0.6236)(1- 0.6236)(0.5124)(0.0273)(2)=0.0066

16. © Copyright 2004 ECE, UM-Rolla. All rights reserved Weight Update 2 New Weight=Old Weight + {(learning rate)(gradient)(prior output)} 0.6136+(0.5)(0.1967)(0.6545) 0.6779 0.5209 0.6779 0.5209 0.508 0.5124+(0.5)(0.0273)(0.6236) 0.5047+(0.5)(0.0066)(1)

17. © Copyright 2004 ECE, UM-Rolla. All rights reserved Third Pass 0.508 0.5209 0.6779 0.508 0.5209 1 0.508 0.6504 0.6243 0.8909 0.6571 0.8909

18. © Copyright 2004 ECE, UM-Rolla. All rights reserved Weight Update Summary W1: Weights from the input to the input layer W2: Weights from the input layer to the hidden layer W3: Weights from the hidden layer to the output layer

19. © Copyright 2004 ECE, UM-Rolla. All rights reserved Training Algorithm The process of feedforward and backpropagation continues until the required mean squared error has been reached. Typical mse: 1e-5 Other complicated backpropagation training algorithms also available.

20. © Copyright 2004 ECE, UM-Rolla. All rights reserved Why Gradient? O1 O2 O = Output of the neuron W = Weight N = Net input to the neuron W1 W2 N = (O1×W1) +(O2×W 2) O3 = 1/[1+exp(-N)] Error = Actual Output – O3 To reduce error: Change in weights: o Learning rate o Rate of change of error w.r.t rate of change of weight  Gradient: rate of change of error w.r.t rate of change of ‘N’  Prior output (O1 and O2) 0 Input Output 1

21. © Copyright 2004 ECE, UM-Rolla. All rights reserved Gradient in Detail Gradient : Rate of change of error w.r.t rate of change in net input to neuron o For output neurons  Slope of the transfer function × error o For hidden neurons : A bit complicated ! : error fed back in terms of gradient of successive neurons  Slope of the transfer function × [Σ (gradient of next neuron × weight connecting the neuron to the next neuron)]  Why summation? Share the responsibility!! o Therefore: Credit Assignment Problem

22. © Copyright 2004 ECE, UM-Rolla. All rights reserved An Example 1 0.4 0.731 0.598 0.5 0.6645 0.66 1 0 Error = 1-0.66 = 0.34 Error = 0-0.66 = -0.66 G1=0.66×(1-0.66)×(-0.66)= -0.148 G1=0.66×(1-0.66)×(0.34)= 0.0763 Reduce more Increase less

23. © Copyright 2004 ECE, UM-Rolla. All rights reserved Improving performance Changing the number of layers and number of neurons in each layer. Variation in Transfer functions. Changing the learning rate. Training for longer times. Type of pre-processing and post- processing.

24. © Copyright 2004 ECE, UM-Rolla. All rights reserved Applications Used in complex function approximations, feature extraction & classification, and optimization & control problems Applicability in all areas of science and technology.

25. © Copyright 2004 ECE, UM-Rolla. All rights reserved Application Detection and Classification of Impact Induced Damage in Composite Plates using Neural Networks: ANNs as classifiers Intelligent Strain Sensing on a Smart Composite Wing using Extrinsic Fabry-Perot Interferometric Sensors and Neural Networks: ANNs as function approximators

26. © Copyright 2004 ECE, UM-Rolla. All rights reserved App 1:Damage Classification Important part of Health Monitoring of a Composite Structure. Study - concentrated on “Low Velocity Impact Behavior” Low velocity impact events can induce localized delamination - significantly reduce the compression strength of composite structures. Many times - damage from impact due to low velocity events cannot be detected by visual inspection techniques. The experimental determination of impact-induced strain profiles can help predict the extent of damage in composite plates. Visual inspection techniques (Surface inspection) may not indicate the severity and extent of the internal damage such as cracking and delamination Artificial neural networks can be incorporated for real time monitoring of composites for damage detection and classification.

27. © Copyright 2004 ECE, UM-Rolla. All rights reserved Type of Input Data The strain in time is sampled every 4 µs and stored in files for every experiment performed. Only Strain ‘X’ and ‘Y’ used as inputs. They are down sampled and normalized.

28. © Copyright 2004 ECE, UM-Rolla. All rights reserved Type of Outputs A total of Seven (7) Classifications were decided upon by visually inspecting the composite plates and the Kinetic Energy of the falling mass. The classification is coded using “GRAY CODE”

29. © Copyright 2004 ECE, UM-Rolla. All rights reserved ANN Details Multi-layered Feed-forward Network is used 10 neurons in the Hidden Layer As we have 7 Classifications implemented in Gray Code, we have 3 neurons in the output layer Transfer Functions of both hidden and output layer is “LOGSIG”

30. © Copyright 2004 ECE, UM-Rolla. All rights reserved Training Algorithm Backprpogation using several training algorithms were used to train the network. Levenberg Marquardt couldn’t be used because of the huge memory requirements for 503 elements of the input vector. “Conjugate Gradient Method” & “One Step Secant Method”, used to train the network give good results, with the former converging earlier. Conjugate Gradient Method is suited for large size input vectors. One step secant method requires more storage space than conjugate gradient method, therefore takes a longer time to converge. The network was trained for 4000 epochs to obtain the required mean squared error.

31. © Copyright 2004 ECE, UM-Rolla. All rights reserved Results Number of Errors for one of the locations

32. © Copyright 2004 ECE, UM-Rolla. All rights reserved App 2: Strain Prediction Aerodynamic parameter prediction  Strain: different points on wing Varying conditions  Angle-of-attack & air speed Neural network modeling Stall Prediction

33. © Copyright 2004 ECE, UM-Rolla. All rights reserved Experimentation Key Strain points Measured Variation in Pressure: 0 to 460 Pa Variation in angle-of-attack: -1.627 0 to 4.31 0

34. © Copyright 2004 ECE, UM-Rolla. All rights reserved Neural Network Modeling Neural network trained on two types of data Max and Min strain Average Strain

35. © Copyright 2004 ECE, UM-Rolla. All rights reserved Results Average errors in the test set