1. Neural Networks Primer Dr Bernie Domanski The City University of New York / CSI 2800 Victory Blvd 1N-215 Staten Island, New York 10314 drbernie@optonline.net http://domanski.cs.csi.cuny.edu

2. © B. Domanski, 2000-2001. All Rights Reserved. Slide 2 What is a Neural Network? Artificial Neural Networks – (ANN) learning Provide a general, practical method for learning – real valued functions – discrete valued functions – vector valued functions from examples Algorithms “tune” input parameters to best fit a training set of input-output pairs

3. © B. Domanski, 2000-2001. All Rights Reserved. Slide 3 What is a Neural Network?  ANN Learning is robust to errors in the training set data  ANN have been applied to problems like Interpreting visual scenes Speech recognition Learning robot control strategies Recoginizing handwriting Face recognition

4. © B. Domanski, 2000-2001. All Rights Reserved. Slide 4 Biological Motivation neurons  ANNs are built out of a densely interconnected set of simple units (neurons) Each neuron takes a number of real- valued inputs Produces a single real-valued output Inputs to a neuron may be the outputs of other neurons. A neuron’s output may be used as input to many other neurons

5. © B. Domanski, 2000-2001. All Rights Reserved. Slide 5 Biological Analogy Human Brain: 10 11 neurons Each neuron is connected to 10 4 neurons inhibitedexcited Neuron Activity is inhibited or excited through interconnections with other neurons Neuron switching times = 10 -3 (human) Time to recognize mom = 10 -1 seconds neuron firings Implies only several hundred neuron firings

6. © B. Domanski, 2000-2001. All Rights Reserved. Slide 6 Complexity of the Biological System  Speculation: parallel Highly parallel processes must be operating on representations that are distributed over many neurons. slow Human neuron switching speeds are slow Motivation is for ANN to capture this highly parallel computation based on a distributed representation Motivation is for ANN to capture this highly parallel computation based on a distributed representation

7. © B. Domanski, 2000-2001. All Rights Reserved. Slide 7 A Simple Neural Net Example Input Nodes Input Nodes Output Neurons LINK Weight

8. How Does the Network Work? weights to each input-link Assign weights to each input-link Multiply each weight by the input value (0 or 1) Sum all the weight-firing input combinations If Sum > Threshold for the Neuron then  Output = +1  Else Output = -1 So for the X=1, Y=1 case – IF w1*X+w2*Y > 99 THEN OUTPUT =Z= +1 IF w1*X+w2*Y > 99 THEN OUTPUT =Z= +1 50*1+50*1 > 99  50*1+50*1 > 99  IF w3*X+w4*Y+w5*Z > 59 THEN OUTPUT = +1 IF w3*X+w4*Y+w5*Z > 59 THEN OUTPUT = +1 30*1+30*1+(-30)*1 > 59  ELSE OUTPUT = -1 30*1+30*1+(-30)*1 > 59  ELSE OUTPUT = -1

9. OR 100 99 XYoutput 00-1 011 101 111

10. Exclusive OR XYXY Output Neurons LINK W1 XYoutput 00-1 011 101 11-1 50 99 -30 30 59 Exclusive-OR W2 W3 W4 W5

11. © B. Domanski, 2000-2001. All Rights Reserved. Slide 11 Appropriate Problems for Neural Networks vectors  Instances where there are vectors of many defined features (eg. meaurements)  Output may be a discrete value or a vector of discrete values  Training examples may contain errors  Non-trivial training sets imply non-trivial time for training  Very fast application of the learned network to a subsequent instance  We don’t have to understand the learned function – only the learned rules

12. © B. Domanski, 2000-2001. All Rights Reserved. Slide 12 How Are ANNs Trained?  Initially choose small random weights (w i ) Set threshold = 1 Choose small learning rate (r)  Apply each member of the training set to the neural net model using the training rule to adjust the weights

13. © B. Domanski, 2000-2001. All Rights Reserved. Slide 13 The Training Rule Explained  Modify the weights (w i ) according to the Training Rule:  Here – r is the learning rate (eg. 0.2) t = target output a = actual output x i = i-th input value w i = w i +  w i where  w i = r * (t – a) * x i

14. © B. Domanski, 2000-2001. All Rights Reserved. Slide 14 Training for ‘OR’ Training Set: X 1 X 2 target 0 0-1 0 1 1 1 0 1 1 1 1 Initial Random Weights W1 =.3 W2 =.7 Learning Rate r =.2

15. © B. Domanski, 2000-2001. All Rights Reserved. Slide 15 Applying the Training Set for OR - 1 X1X1 X2X2 1 a 0 0 = -1 0 1 = -1 X  w 1 = r * (t – a) * x 1 =.2 * (1-(-1)) * x 1 =.2 * (2) * 0 = 0  w 2 =.2 * (1-(-1)) * x 2 =.2 * (2) * 1 =.4 w 1 = w 1 +  w 1 =.3 + 0 =.3 w 2 = w 2 +  w 2 =.7 +.4 = 1.1.3.7

16. © B. Domanski, 2000-2001. All Rights Reserved. Slide 16 Applying the Training Set for OR - 2 X1X1 X2X2 1 a 0 0 = -1 0 1 = +1 1 0 = -1 X  w 1 = r * (t – a) * x 1 =.2 * (1-(-1)) * x 1 =.2 * (2) * 1 =.4  w 2 =.2 * (1-(-1)) * x 2 =.2 * (2) * 0 = 0 w 1 = w 1 +  w 1 =.3 +.4 =.7 w 2 = w 2 +  w 2 = 1.1+0 = 1.1.3 1.1

17. © B. Domanski, 2000-2001. All Rights Reserved. Slide 17 Applying the Training Set for OR - 3 X1X1 X2X2 1 a 0 0 = -1 0 1 = +1 1 0 = -1 X  w 1 = r * (t – a) * x 1 =.2 * (1-(-1)) * x 1 =.2 * (2) * 1 =.4  w 2 =.2 * (1-(-1)) * x 2 =.2 * (2) * 0 = 0 w 1 = w 1 +  w 1 =.7+.4 = 1.1 w 2 = w 2 +  w 2 = 1.1+0 = 1.1.7 1.1

18. © B. Domanski, 2000-2001. All Rights Reserved. Slide 18 Applying the Training Set for OR - 4 X1X1 X2X2 1 a 0 0 = -1 0 1 = +1 1 0 = +1 1 1 = +1  1.1

19. © B. Domanski, 2000-2001. All Rights Reserved. Slide 19 Training for ‘AND’ Training Set: X 1 X 2 target 0 0-1 0 1-1 1 0-1 1 1 1 Initial Random Weights W1 =.3 W2 =.7 Learning Rate r =.2

20. © B. Domanski, 2000-2001. All Rights Reserved. Slide 20 Applying the Training Set for AND - 1 X1X1 X2X2 1 a 0 0 = -1 0 1 = -1 1 0 = -1 1 1 = -1 X  w 1 = r * (t – a) * x 1 =.2 * (1-(-1)) * 1 =.4  w 2 =.2 * (1-(-1)) * 1 =.4 w 1 = w 1 +  w 1 =.3 +.4 =.7 w 2 = w 2 +  w 2 =.7 +.4 = 1.1.3.7

21. © B. Domanski, 2000-2001. All Rights Reserved. Slide 21 Applying the Training Set for AND - 2 X1X1 X2X2 1 a 0 0 = -1 0 1 = +1 X  w 1 = r * (t – a) * x 1 =.2 * (-1-(+1)) * 0 = 0  w 2 =.2 * (-1-(+1)) * 1 = -.4 w 1 = w 1 +  w 1 =.7 + 0 =.7 w 2 = w 2 +  w 2 = 1.1 -.4 =.7.7 1.1

22. © B. Domanski, 2000-2001. All Rights Reserved. Slide 22 Applying the Training Set for AND - 3 X1X1 X2X2 1 a 0 0 = -1 0 1 = -1 1 0 = -1 1 1 = +1 .7

23. © B. Domanski, 2000-2001. All Rights Reserved. Slide 23 Applying the Technology

24. © B. Domanski, 2000-2001. All Rights Reserved. Slide 24 Start the Neugent Wizard Choose Namespace Name the instance to be trained

25. © B. Domanski, 2000-2001. All Rights Reserved. Slide 25 Select The Data Set Choose data for the Neugent

26. © B. Domanski, 2000-2001. All Rights Reserved. Slide 26 Select The Output That You Want to Predict Choose Inputs Identify the Outputs

27. © B. Domanski, 2000-2001. All Rights Reserved. Slide 27 Train And Validate the Neugent Choose Action to be performed: Create the model (Quick Train) Train & Validate (to understand the predictive capability) Investigate the data (Export to Excel or Data Analysis)

28. © B. Domanski, 2000-2001. All Rights Reserved. Slide 28 Validate the Neugent With the Data Set Selecting Training Data – Select a random sample percentage Use the entire data set

29. © B. Domanski, 2000-2001. All Rights Reserved. Slide 29 Neugent Model is Trained, Tested, and Validated Training Results – Model Fit: 99.598% (trained model quality) Predictive Capability: 99.598% (tested model quality)

30. © B. Domanski, 2000-2001. All Rights Reserved. Slide 30 View The Results in Excel Consult trained Neugent for prediction Save results using Excel

31. © B. Domanski, 2000-2001. All Rights Reserved. Slide 31 Data Analysis Stats & Filtering: mean, min, max, std dev, filtering constraints Ranking: input significance Correlation Matrix: corr. between all fields

32. © B. Domanski, 2000-2001. All Rights Reserved. Slide 32 Importance Ranking

33. © B. Domanski, 2000-2001. All Rights Reserved. Slide 33 Correlation Matrix The closer to 1, the stronger the indication that the information represented by the two fields is the same NetBusy vs #Trans =.9966

34. © B. Domanski, 2000-2001. All Rights Reserved. Slide 34 Actual Vs Predicted

35. © B. Domanski, 2000-2001. All Rights Reserved. Slide 35 Actual Vs Predicted

36. © B. Domanski, 2000-2001. All Rights Reserved. Slide 36 Summary Neural Networks  Modeled after neurons in the brain  Artificial neurons are simple  Neurons can be trained  Networks of neurons can be taught how to respond to input  Models can be built quickly  Accurate predictions can be made

37. © B. Domanski, 2000-2001. All Rights Reserved. Slide 37 Questions?  Questions, comments, … ??  Finding me – Dr Bernie Domanski Email: domanski@postbox.csi.cuny.edu Website: http://domanski.cs.csi.cuny.edu Phone: (718) 982-2850 Fax: 2356  Thanks for coming and listening !