1. Artificial Neural Networks KONG DA, XUEYU LEI & PAUL MCKAY

2. Neural Networks

3. What is Neural Networks An Artificial Neural Network is a computational simulation of a biological neural network.  composed of a large number of highly interconnected processing elements(neurons)  an information processing paradigm  learn by example

4. What is Neural Networks  Inspiration from brain Figure1 Neuron structure Figure 2 ANN structure Figure1: http://www.studyblue.com/notes/note/n/biological-foundations-neuron-communication-/deck/1025438 Figure2: http://www.gdl.cinvestav.mx/~edb/students/evazquez/index.html

5. History of ANNs 1943 Warren McCulloch and Walter Pitts modeled a simple neural network with electrical circuits 1949 Donald Hebb pointed out that neural pathways are strengthened each time that they are used in The Organization of Behavior The IBM research laboratories led the first effort to simulate a neural network 1950’s 1959 Bernard Widrow and Marcian Hoff of Stanford developed models called "ADALINE" and "MADALINE." ADALINE Could predict the next bit MADALINE The first neural network applied to a real world problem 1982 John Hopfield of Caltech created more useful machines by using bidirectional lines (Hopfield Network) The first recurrent network 1980 Convolutional neural networks were introduced in a 1980 paper by Kunihiko Fukushima Mid 2000s "deep learning" gained traction in the mid-2000s after a publication by Geoffrey Hinton and Ruslan Salakhutdinov 1985 A Boltzmann machine, a type of stochastic recurrent neural network is invented by Geoffrey Hinton and Terry Sejnowski Bayesian network Learning Vector Quantization Long short term memory network physical neural network Hierarchical temporal memory (HTM) …… NOW Companies are working on three types of neuro chips - digital, analog, and optical

6. History of ANNs Figure 3 MADALINE structure Figure 4 An example Boltzmann machine Figure 5 a maximally simple LSTM network Figure3:http://www.drdobbs.com/the-foundation-of-neural-networks-the-ad/184402585 Figure4:http://en.wikipedia.org/wiki/Boltzmann_machine Figure5: http://www.schraudolph.org/teach/NNcourse/lstm.htmlhttp://www.schraudolph.org/teach/NNcourse/lstm.html Figure 6 Neural chip http://www.gizmag.com/neuromorphic-chips/28586/

7. Types of ANNs ANNs Feed-forward Neural Networks Single- layer Multi- layer Recurrent Neural Networks Hopfield network Long short term memory network … Others Kohonen self- organizing network …

8. Perceptron

9. Activation function WikiBooks. Artificial Neural Networks/Activation Functions. 25 August 2014.Artificial Neural Networks/Activation Functions

10. Example 1 Çelebi, Ömer Cengiz. Neural Networks and Pattern Recognition Using MATLAB. Retrieved 25 August 2014.Neural Networks and Pattern Recognition Using MATLAB

11. Example 2 Çelebi, Ömer Cengiz. Neural Networks and Pattern Recognition Using MATLAB. Retrieved 25 August 2014.Neural Networks and Pattern Recognition Using MATLAB

12. Neural network topology  Multilayer  Interconnected  Feed forward/recurrent  Deep learning

13. Training  Back propagation  Learning rate  Batch learning  Stochastic gradient descent  Evolutionary algorithms Reference: LeCun et al. Efficient BackProp. 1998.Efficient BackProp

14. Application areas Function approximation http://www.eweb.unex.es/eweb/fisteor/santos/sby.html Classification Screenshot from https://www.youtube.com/watch?v=KuPai0ogiHk https://www.youtube.com/watch?v=KuPai0ogiHk

15. Application areas Data processing. Control http://www.seit.adfa.edu.au/research/details2.php?page_id =410&topic=Adaptive_Flight_Control http://www.cslu.ogi.edu/tutordemos/nnet_recog/recog.html

16. Application areas Robotics Screenshots from https://www.youtube.com/watch?v=V2ADU8YWIughttps://www.youtube.com/watch?v=V2ADU8YWIug