1. Copyright Brad Morantz 2003 Neural Networks An Overview Brad Morantz PhD Viral Immunology Center Georgia State University bradscientist@ieee.org www.geocities.com/bradscientist

2. Copyright Brad Morantz 2003 Neural Networks I think, therefore I am

3. Copyright Brad Morantz 2003 Contents 1. Introduction 2. Expert Systems 3. Neural Networks 4. Prediction 5. Classification 6. Pattern Recognition 7. Overview 8. Comparisons 9. How a NN works 10. Examples 11. Applications 12. Future 13. Resources 14. Appendices 1.Degrees of freedom 2.Machine Learning

4. Copyright Brad Morantz 2003 The flight to Dallas is cancelled That’s Wonderful!

5. Copyright Brad Morantz 2003 Why is it wonderful? 4 Either there is something wrong with : –The airplane –The Pilot –Dallas If there is something wrong with any of these, I don’t want to be on the airplane This example courtesy of Zig Ziglear

6. Copyright Brad Morantz 2003 Has Anyone been turned down on a credit card transaction? 4 For reasons other than late payments?

7. Copyright Brad Morantz 2003 I was ID’d buying a VCR 4 This is what I normally do: –I normally buy gas once a week –I usually spend about $10 –I buy a book once a month 4 This is MY pattern

8. Copyright Brad Morantz 2003 I need to see your ID That’s Wonderful! Scene at the Store

9. Copyright Brad Morantz 2003 Why did the Salesperson ask for ID? 4 The credit card machine told her to 4 Why did the machine ask for it? –They have a pattern of my normal activity –Buying a VCR was not part of that pattern –Expensive purchases were not normal for me –It wanted to be sure that it was not a stolen card –Criminals with stolen cards often buy expensive electronic items like VCR’s –It is a pattern for a stolen credit card

10. Copyright Brad Morantz 2003 How did the credit card machine do this? 4 It used Artificial Intelligence 4 Computer program –Expert System –Neural Network –Hybrid or combination

11. Copyright Brad Morantz 2003 Prediction 4 Last week, 4 apples cost $2.00 4 Yesterday, 6 apples cost $3.00 4 Today, 10 apples cost $5.00 4 How much will 5 apples cost tomorrow? $2.50 Notice that the response is on a value that was not in the training set This is linear, but an ANN can handle nonlinear equally well

12. Copyright Brad Morantz 2003 Classification Problem # wheelslength Noise # seats what is it 4 short quiet 4car 2 short loud 1 motorcycle 6 long loud lots bus 18v long loud 2semi 4short loud 2 _____ With a few inputs, we classified the vehicle Notice that is in classes, not a specific vehicle, i.e. 300ZX car

13. Copyright Brad Morantz 2003 Pattern Recognition Your Friend MaryGeorge Washington These are very specific answers

14. Copyright Brad Morantz 2003 What do these examples have in common? 4 Learned 4 From experience 4 From historical data 4 By example 4 By organization

15. Copyright Brad Morantz 2003 What is an Expert System 4 Knowledge based system 4 Having an expert that you can ask questions 4 Computer program –Has set of rules or heuristics –From a human expert –Behaves like human expert

16. Copyright Brad Morantz 2003 Quick overview of expert system 4 Requires Expert 4 Requires Knowledge Engineer 4 Stores knowledge in set of rules 4 Must have rule for all possibilities 4 Contains bias, opinions, and emotions of expert 4 May not be current 4 Can ask it reason for its answer

17. Copyright Brad Morantz 2003 What is a Neural Network? 4 A human Brain 4 A porpoise brain 4 The brain in a living creature 4 A computer program –Emulates biological brain –Limited connections 4 Specialized computer chip

18. Copyright Brad Morantz 2003 Types of Function 4 Prediction –Like regression 4 Time Series Forecasting –Like autoregression 4 Classification –Sort into a class, like cluster analysis 4 Pattern Recognition –Fined tuned classification 4 Not constrained to linear or Gauss Normal distribution

19. Copyright Brad Morantz 2003 Advantages of Neural Network 4 No Expert needed 4 No Knowledge Engineer needed 4 Does not have bias of expert 4 Can interpolate for all cases 4 Learns from facts – data driven 4 Can resolve conflicts 4 Variables can be correlated (multicollinearity) 4 Works with unknown underlying relationships

20. Copyright Brad Morantz 2003 More Advantages 4 Learns input to output relationship 4 Can make good model with noisy or incomplete data 4 Can handle nonlinear or discontinuous data 4 Can Handle data of unknown or undefined distribution 4 Few to no apriori assumptions

21. Copyright Brad Morantz 2003 Disadvantages of Neural Net 4 Black Box –don’t know why or how –not sure of what it is looking at 4 Operator dependent -HITL 4 Don’t have knowledge in hand * Many of these disadvantages are being overcome

22. Copyright Brad Morantz 2003 Black Box 4 What happens inside the box is unknown 4 We can’t see into the box 4 We don’t know what it knows input output

23. Copyright Brad Morantz 2003 Key to Neural Networks 4 Learns input to output relationship

24. Copyright Brad Morantz 2003 Biological Neural Network 4 Human Brain has 4 x 10 10 to 10 11 Neurons 4 Each can have 10,000 connections* 4 Human baby makes 1 million connections per second until age 2 4 Speed of synapse is 1 kHz, much slower than computer (3.0gHz) 4 Massively parallel structure * Some estimates are much greater

25. Copyright Brad Morantz 2003 How does a neuron work? 4 It sums the weighted inputs –If it is enough, then neuron fires –There can be as many as 10,000 or more inputs

26. Copyright Brad Morantz 2003 Computer Neural Network 4 Von Neuman architecture 4 Serial machine with inherently parallel process 4 Series of mathematical equations 4 Simulates relatively small brain 4 Limited connectivity 4 Closely approximates complex nonlinear functions

27. Copyright Brad Morantz 2003 Computer Emulation of NN 4 Called Artificial Neural Network (ANN) 4 Two uses –Applications as shown Forecasting/Prediction Classification Pattern recognition –Use model to help understand biological neural network/ model brain function

28. Copyright Brad Morantz 2003 Neuron Activation 4 Weights can be positive or negative 4 Negative weight inhibits neuron firing 4 Sum = W 1 N 1 + W 2 N 2 + …. + W n N n 4 If sum is negative, neuron does not fire 4 If sum is positive neuron fires 4 Fire means an output from neuron 4 Non-linear function 4 Some models include a threshold

29. Copyright Brad Morantz 2003 Neuron Activation 4 Linear 4 Sigmoidal –1.0/(1.0+e -s ) where s = Σ inputs –0 or +1 result 4 Hyperbolic Tangent –(e s – e -s ) / (e s + e -s ) where s = Σ inputs –-1 or +1 result 4 Also called squashing or clamping function

30. Copyright Brad Morantz 2003 What does the network look like? 4 This is a computer model, not biological 4 Left has 11 neurons, sea slug has 100 Feed ForwardRecurrent or Feedback

31. Copyright Brad Morantz 2003 Macro View of Training 4 Setting all of the weights 4 To create optimal performance 4 Optimal adherence to training data 4 Really an optimization problem –Optimal methods depends on many variables –See optimization lecture 4 Need objective function 4 Beware of local minima!

32. Copyright Brad Morantz 2003 Training Methods 4 Back Propagation (most popular) 4 Gradient Descent 4 Generalized reduced gradient (GRG) 4 Simulated Annealing 4 Genetic Algorithm 4 Two or more output nodes –Multi objective optimization 4 Many more methods

33. Copyright Brad Morantz 2003 Training 4 Supervised –Data given, along with answer –Compare to prediction and go try again –Historical data and/or rules 4 Unsupervised –No teacher or feedback –Self organizing

34. Copyright Brad Morantz 2003 Training Data Set 4 Need more observations than weights –Positive # degrees freedom 4 More observations is usually better –Lower variance –More knowledge 4 Watch aging of data

35. Copyright Brad Morantz 2003 Dynamic Learning 4 Continuous learning –From mistakes and successes –From new information 4 Shooting baskets example –Too low. Learned: throw harder –Too high. Learned: throw softer, but not as soft as before –Basket! Learned: correct amount of “push” 4 Loaning $10 example

36. Copyright Brad Morantz 2003 inputs 4 One per input node 4 Ratio 4 Logical 4 Dummy 4 Categorical 4 Ordinal

37. Copyright Brad Morantz 2003 Outputs 4 One for single dependent variable 4 Multiple –Prediction –Classification –Pattern recognition

38. Copyright Brad Morantz 2003 Hidden Layer(s) 4 Increase complexity 4 Can increase accuracy 4 Can reduce degrees of freedom –Need larger data set 4 Presently architecture up to programmer 4 Source for error 4 In future will be more automatic

39. Copyright Brad Morantz 2003 Hybrids 4 Use advantages of both Expert and ANN 4 Uses more knowledge than just one type 4 Can seed system with expert knowledge and then update with data 4 Sometimes hard to get all parts to work together

40. Copyright Brad Morantz 2003 Example 4 You go some place that you have never been before, and get “bad vibes” –Atmosphere, temperature, lighting, smell, coloring, numerous things 4 For some reason, brain associates these together, possibly some past experience 4 Gives you “bad feeling”

41. Copyright Brad Morantz 2003 Example 4 Bat (yes, that little flying rodent) 4 Brain the size of a plum –With its SONAR, it can outperform our best airplanes, flying through an electric fan –Outperforms our best radar and computer system –Needs no 440V supply nor cooling system (unlike SGI Origins & IBM Regattas)

42. Copyright Brad Morantz 2003 Applications 4 Credit card fraud 4 Military: submarine, tank, sniper detect, or target recognition 4 Security 4 Classify stars & planets 4 Data mining 4 Natural language recognition

43. Copyright Brad Morantz 2003 Future 4 Rule extraction 4 Hybrids 4 Dynamic learning 4 Parallel processing 4 Dedicated chips 4 Bigger & more automatic 4 Machine Learning

44. Copyright Brad Morantz 2003 Contacts 4 IEEE Transactions on Neural Networks 4 IEEE Intelligent Systems Journal 4 IEEE Neural Network Society 4 www.aaai.org AAAI American Association for Artificial Intelligence 4 www.ieee.org 4 www.zsolutions.com 4 www.geocities.com/bradscientist www.geocities.com/bradscientist 4 Internet

45. Copyright Brad Morantz 2003 Appendix – degrees of freedom 4 Number of values free to vary –Complex math definition 4 Must watch, not less than zero 4 i.e. 3X + 4Y = Z –3 unknowns, if know Z only one other can vary –If have 3 equations, values can not vary 4 Large ANN needs big sample data

46. Copyright Brad Morantz 2003 Appendix - Machine Learning 4 Machine learns from experience 4 From experience and data it improves its knowledge (rule base or weights) 4 Shooting baskets example