Neural Networks William Lai Chris Rowlett

What are Neural Networks? A type of program that is completely different from functional programming. Consists of units that carry out simple computations linked together to perform a function Modeled after the decision making process of the biological network of neurons in the brain

The Biology of Neural Networks Neural Networks are models of neuron clusters in the brain Each Neuron has a: Dendrites Axon Terminal buds Synapse Action potential is passed down the axon, which causes the release of neurotransmitters

Types of Neural Networks: General Supervised During training, error is determined by subtracting output from actual value Unsupervised Nothing is known of results Used to classify complicated data Nonlearning Optimization

Types of Neural Networks: Specific Perceptrons A subset of feed-forward networks, containing only one input layer, one output layer, and each input unit links to only output units Feed-forward networks a.k.a. Directed Acyclic Graphs Each unit only links to units in subsequent layers Allows for hidden layers Recurrent networks Not very well understood Units can link to units in the same layer or even previous layers Example: The Brain

Neural Net Capabilities Neural Nets can do anything a normal digital computer can do (such as perform basic or complex computations) Functional Approximations/Mapping Classification Good at ignoring ‘noise’

Neural Net Limitations Problems similar to Y=1/X between (0,1) on the open interval (Pseudo)-random number predictors Factoring integers or determining prime numbers Decryption

History of Neural Networks McColloch and Pitts (1943) Co-wrote first paper on possible model for a neuron Widrow Hoff (1959) Developed MADALINE and ADALINE MADALINE was the first neural network to try to solve a real world problem Eliminates echo in phone lines vonNeumann architecture took over for about 20 years (60’s-80’s)

Early Applications Checkers (Samuel, 1952) At first, played very poorly as a novice With practice games, eventually beat its author ADALINE (Widrow and Hoff, 1959) Recognizes binary patterns in streaming data MADALINE (same) Multiple ADAptive LINear Elements Uses an adaptive filter that eliminates echoes on phone lines

Modern Practical Applications Pattern recognition, including Handwriting Deciphering Voice Understanding “Predictability of High-Dissipation Auroral Activity” Image analysis Finding tanks hiding in trees (cheating) Material Classification "A real-time system for the characterization of sheep feeding phases from acoustic signals of jaw sounds"

How Do Neural Networks Relate to Artificial Intelligence? Neural networks are usually geared towards some application, so they represent the practical action aspect of AI Since neural networks are modeled after human brains, they are an imitation of human action. However, than can be taught to act rationally instead. Neural networks can modify their own weights and learn.

The Future of Neural Networks Pulsed neural networks The AI behind a good Go playing agent Increased speed through the making of chips robots that can see, feel, and predict the world around them improved stock prediction common usage of self-driving cars Applications involving the Human Genome Project self-diagnosis of medical problems using neural networks

Past Difficulties Single-layer approach limited applications Converting Widrow-Hoff Technique for use with multiple layers Use of poorly chosen and derived learning function High expectations and early failures led to loss of funding

Recurring Difficulties Cheating Exactly what a neural net is doing to get its solutions is unknown and therefore, it can cheat to find the solution as opposed to find a reliable algorithm Memorization Overfitting without generalization

Describing Neural Net Units All units have input values, a j All input values are weighted, as in each a j is multiplied by the link’s weight, W j,i All weighted inputs are summed, generating in i The unit’s activation function is called on in i, generating the activation value a i The activation value is output to every destination of the current unit’s links.

Perceptrons Single layer neural networks Require linearly separable functions Guarantees the one solution OR XOR

Back-Propagation Back-propagation uses a special function to divide the error of the outputs to all the weights of the network The result is a slow-learning method for solving many real world problems

Organic vs. Artificial Computer cycle times are in the order of nanoseconds while neurons take milliseconds Computers compute the results of each neuron sequentially, while all neurons in the brain fire simultaneously every cycle Result: massive parallelism makes brains a billion times faster than computers, even though computer bits can cycle a million times faster than neurons

Questions?