1. NEURAL NETWORK THEORY

2. TABLE OF CONTENTS Part 1: The Motivation and History of Neural Networks Part 2: Components of Artificial Neural Networks Part 3: Particular Types of Neural Network Architectures Part 4: Fundamentals on Learning and Training Samples Part 5: Applications of Neural Network Theory and Open Problems Part 6: Homework Part 7: Bibliography

3. PART 1: THE MOTIVATION AND HISTORY OF NEURAL NETWORKS

4. MOTIVATION Biologically inspired The organization of the brain is considered when constructing network configurations and algorithms

5. THE BRAIN A human Neuron has four elements: Dendrites – receive signals from other cells Synapses – where information is stored at the contact points between neurons Axons – output signals transmitted Cell body – produces all necessary chemicals for the neuron to function properly

6. ASSOCIATION TO NEURAL NETWORKS Artificial neurons have Input channels Cell body Output channel And synapses are simulated with a weight

7. MAIN CHARACTERISTICS ADAPTED FROM BIOLOGY Self-organization and learning capability Generalization capability Fault tolerance

8. THE 100-STEP RULE

9. WORD TO THE WISE We must be careful comparing the nervous system with a complicated contemporary device. In ancient times, the brain was compared to a pneumatic machine, in renaissance to a clockwork, and in the 1900's to a telephone network

10. HISTORY OF NEURAL NETWORK THEORY 1943 - Warren McCulloch and Walter Pitts introduced models of neurological networks 1947 - Pitts and McCulloch indicated a practical field of application for neural networks 1949 - Karl Lashley defended his thesis that brain information storage is realized as a distributed system.

11. HISTORY CONTINUED 1960 - Bernard Widrow and Marcian Hoff introduced the first fast and precise adaptive learning system. The first widely commercially used neural network. Hoff later became the co-founder of Intel Corporation. 1961 - Karl Steinbuch introduced technical realizations of associative memory which can be seen as predecessors of today's neural associative memories 1969 - Marvin Minsky and Seymour Papert published a precise analysis of the perceptron to show the perceptron model was not capable of representing many important problems and so, deduced that the field would be a research "dead end".

12. HISTORY PART 3 1973 - Christoph von der Malsburg used a neuron model that was non-linear and biologically more motivated 1974 - Harvard Werbos developed a learning procedure called backpropagation of error 1982 - Teuvo Kohonen described the self-organizing feature maps also known as Kohonen maps 1985 - John Hopfield published an article describing a way of finding acceptable solutions for the Travelling Salesman problem using Hopfield nets

13. SIMPLE EXAMPLE OF A NEURAL NETWORK Assume we have a small robot. This robot has n number of distance sensors from which it extracts input data. Each sensor provides a real numeric value at any time. In this example, the robot can "sense" when it is about to crash. So, it drives until one of its sensors denotes it is going to collide with an object. Neural networks allow the robot to "learn when to stop" by treating the neural network as a "black box", then we do not know its structure but just regard its behavior in practice. So, we show the robot when to drive on or when to stop. i.e. called training samples, and are taught to the neural network by learning procedures. Either an algorithm or a mathematical formula. From this, the neural network in the robot will generalize from these samples, and learn when to stop.

14. PART 2: COMPONENTS OF ARTIFICIAL NEURAL NETWORKS

15. FLYNN’S TAXONOMY OF COMPUTER DESIGN Single instruction streamMulti instruction streamSingle programMultiple program Single data streamSISDMISD Multiple data streamSIMDMIMDSPMDMPMD Neural Computers are a particular case of MSMID architecture Simplest case: an algorithm represents an operation of multiplying a large dimensionality vector or matrix by a vector The number of operation cycles in the problem solving process is determined by the physical entity and complexity of the problem

16. NEURAL “CLUSTERING” A “cluster” is a synchronously functioning group of single-bit processors that has a special organization that is close to the implementation of the main part of the algorithm This provides solutions to two additional problems 1) to minimize or eliminate the information interchange between nodes of the neural computer in the process of problem solving 2) to solve weakly formalized problems (e.g. learning for optimal pattern recognition, self-learning clusterization, etc)

17. DEFINTIONS

18. NEURONS

19. NEURAL NETWORKS

20. THE PROPAGATION FUNCTION

21. THRESHOLD FUNCTION

22. ACTIVATION FUNCTION

23. FURTHER PROPERTIES OF THE ACTIVATION FUNCTION

24. OPTIMAL CONTROL THEORY Zermelo’s problem and the handout Example problem

25. PART 3: PARTICULAR TYPES OF NEURAL NETWORK ARCHITECTURES

26. TRANSFER FROM LOGICAL BASIS TO THRESHOLD BASIS

27. FERMI OR LOGISTIC EQUATION AND TANH(X)

28. NEURAL NETWORK WITH DIRECT CONNECTIONS

29. NEURAL NETWORKS WITH CROSS CONNECTIONS

30. NEURAL NETWORKS WITH ORDERED BACKWARD CONNECTIONS

31. NEURAL NETWORKS WITH AMORPHOUS BACKWARD CONNECTIONS

32. MULTILAYER NEURAL NETWORKS WITH SEQUENTIAL CONNECTIONS

33. MULTILAYER NEURAL NETWORK

34. FEEDFORWARD NETWORKS

35. FEEDFORWARD NETWORK DIAGRAM

36. COMPLETELY LINKED NETWORKS (CLIQUE) Completely linked networks permit connections between all neurons except for direct recurrences. Furthermore, the connections must be symmetric. So, every neuron can become an input neuron. (Clique)

37. DIRECTED TERMS If the function to be computed by the feedforward neural network is thought to have a significant linear component, it may be useful to add linear terms (called directed terms) to the above structure

38. RECURRENT NETWORKS General Form: Neural networks that include cycles. Since the output of a neuron cannot be a function of itself, then we must explicitly take time into account. The output of a neuron cannot be a function of itself at the same instant of time, but can be a function of its past values. These are considered discrete-time systems Each connection of a recurrent neural network is assigned a delay value (possibly equal to zero) in addition to being assigned a weight as in feedforward networks.

39. CANONICAL FORM OF RECURRENT NETWORKS

40. CANONICAL FORM OF RECURRENT NETWORK DIAGRAM

41. THE ORDER OF NEURAL NETWORKS

42. WHEN TO USE NEURAL NETWORKS The fundamental property of neural networks with supervised training is the parsimonious approximation property. i.e. their ability of approximating any sufficiently regular function with arbitrary accuracy. Therefore, neural networks may be advantageous in any application that requires finding, in a machine learning framework, a nonlinear relation between numerical data To do so, make sure that 1) a nonlinear model is necessary 2) determine if a neural network is necessary instead of, for instance a polynomial approximation. i.e. when the number of variables is large (larger than or equal to 3)

43. PART 4: FUNDAMENTALS ON LEARNING AND TRAINING SAMPLES

44. THEORETICALLY, A NEURAL NETWORK COULD LEARN BY Developing new connections Deleting existing connections Changing connecting weights Changing the threshold values of neurons Varying one or more of the three neuron functions (activation, propagation, output) Developing new neurons Deleting neurons The change of connecting weight is the most common procedure.

45. DIFFERENT TYPES OF TRAINING Unsupervised learning - the training set only consists of input patterns, the network tries, by itself, to detect similarities and to generate pattern classes Reinforcement learning - the training set consists of input patterns, after completion of a sequence a value is returned to the network indicating whether the result was right or wrong, and possibly, how it was right or wrong. Supervised learning - the training set consists of input patterns with correct results so that the network can receive a precise error vector

46. SUPERVISED LEARNING STEPS Enter input pattern Forward propagation of the input by the network, generation of the output Comparing the output with the desired output and provide the error vector Corrections of the network are calculated based on the error vector Corrections are applied

47. ERROR VECTOR determined usually by the root mean square function (RMSE) Does not always guarantee global minimum, but may only find local minimum To calculate RMSE 1) take each error of each data point, square the value. 2) Sum the error squared terms 3) divide by the number of data values 4) take the square root of that value

48. PART 5: APPLICATIONS OF NEURAL NETWORK THEORY AND OPEN PROBLEMS

49. OPEN PROBLEMS Identifying if the neural network will converge in finite time Training the neural network to identify local versus global minimums Neural modularity

50. APPLICATIONS OF NEURAL NETWORK THEORY Traveling Salesman problem Image Compression Character Recognition Optimal Control Problems

51. PART 6: HOMEWORK

53. OPTIMAL CONTROL PROBLEM

54. FIND THE RMSE OF THE BELOW DATA SET SampleDataEstimation 11 247 331 40-2 597 611 71213 867 988 102017 11 9 121113 22 1400 1545 1699 1755 181314 191517 2010

55. PART 7: BIBLIOGRAPHY

56. WORKS CITED Dreyfus, G. Neural Networks: Methodology and Applications. Berlin: Springer, 2005. Print. Galushkin, A. I. Neural Networks Theory. Berlin: Springer, 2007. Print. Kriesel, David. "D. Kriesel." A Brief Introduction to Neural Networks []. Manuscript, n.d. Web. 28 Mar. 2016. Lenhart, Suzanne, and John T. Workman. Optimal Control Applied to Biological Models. Boca Raton: Chapman & Hall/CRC, 2007. Print. Ripley, Brian D. Pattern Recognition and Neural Networks. Cambridge: Cambridge UP, 1996. Print. Rojas, Raul. Neural Networks: A Systematic Introduction. Berlin: Springer-Verlag, 1996. Print. Wasserman, Philip D. Neural Computing: Theory and Practice. New York: Van Nostrand Reinhold, 1989. Print. https://www.researchgate.net/post/What_are_the_most_important_open_problems_in_the_field_of_artificial_neural_netw orks_for_the_next_ten_years_and_why