Presentation on theme: "Introduction to Neural Networks Andy Philippides Centre for Computational Neuroscience and Robotics (CCNR) School of Cognitive and Computing Sciences/School."— Presentation transcript:
Introduction to Neural Networks Andy Philippides Centre for Computational Neuroscience and Robotics (CCNR) School of Cognitive and Computing Sciences/School of Biological Sciences firstname.lastname@example.org Spring 2003
Lectures -- 2 per week Time Day Place 12:30 - 1:20 Mon Arun - 401 11:30 - 12:20 Wed Arun - 401 Seminar– 1 per week Group 1 3 – 3.50 Mon Pev1 2D4 Group 2 4 – 4.50 Mon Pev1 2D4 Group 3 2 – 2.50 Fri Arun 404B Group 4 3 – 3.50 Fri Arun 404B Office hour: Friday12.30-1.30, BIOLS room 3D10 Lecture will be available online soon
Todays Topics: Course summary Components of an artificial neural network A little bit math Single artificial neuron
The course will introduce the theory of several variants of artificial neural networks (ANNs) discuss how they are used/trained in practice Ideas will be illustrated using the example of ANNs used for function approximation Very common use of ANNs and also shows the major concepts nicely. Idea: Course Summary Pre- Processing Post- Processing Neural Net model + training method Course Summary Data Function approx [Will not specifically be using NNs as brain models (Computational Neuroscience)]
Topics covered 1. Introduction to neural networks 2. Basic concepts for network training 3. Single layer perceptron 4. Probability density estimation 5+6. Multilayer perceptron 7+8. Radial Basis Function networks 9+10. Support Vector machines 11+12. Pre-processing + Competitve Learning 13+14. Mixtures of Experts/Committee machines 15+16. Neural networks for robot control
Assessment 3rd years: All coursework Masters students: 50% coursework, 50 % exam (start of next term) Coursework is 2 programming projects first is 20% of coursework (details next week) due in week 6, second 80% due week 10. Coursework dealt with in seminars, some theoretical, some practical matlab sessions (programs can be in any language, but matlab is useful for in-built functions) This weeks seminar: light maths revision
Course Texts 1. Haykin S (1999). Neural networks. Prentice Hall International. Excellent but quite heavily mathematical 2. Bishop C (1995). Neural networks for pattern recognition. Oxford: Clarendon Press (good but a bit statistical, not enough dynamical theory) 3. Pattern Classification, John Wiley, 2001 R.O. Duda and P.E. Hart and D.G. Stork 4. Hertz J., Krogh A., and Palmer R.G. Introduction to the theory of neural computation (nice, but somewhat out of date)
5. Pattern Recognition and Neural Networks by Brian D. Ripley. Cambridge University Press. Jan 1996. ISBN 0 521 46086 7. 6. Neural Networks. An Introduction, Springer-Verlag Berlin, 1991 B. Mueller and J. Reinhardt As its quite a mathematical subject good to find the book that best suits your level Also for algorithms/mathematical detail see Numerical Recipes, Press et al. And appendices of Duda, Hart and Stork and Bishop
Uses of NNs Neural Networks Are For Applications Science Character recognition Neuroscience Optimization Physics, mathematics statistics Financial prediction Computer science Automatic driving Psychology.........................................................
What are biological NNs? UNITs: nerve cells called neurons, many different types and are extremely complex around 10 11 neurons in the brain (depending on counting technique) each with 10 3 connections INTERACTIONs: signal is conveyed by action potentials, interactions could be chemical (release or receive neurotransmitters) or electrical at the synapse STRUCTUREs: feedforward and feedback and self-activation recurrent
The nerve fibre is clearly a signalling mechanism of limited scope. It can only transmit a succession of brief explosive waves, and the message can only be varied by changes in the frequency and in the total number of these waves. … But this limitation is really a small matter, for in the body the nervous units do not act in isolation as they do in our experiments. A sensory stimulus will usually affect a number of receptor organs, and its result will depend on the composite message in many nerve fibres. Lord Adrian, Nobel Acceptance Speech, 1932.
We now know its not quite that simple Single neurons are highly complex electrochemical devices Synaptically connected networks are only part of the story Many forms of interneuron communication now known – acting over many different spatial and temporal scales
The complexity of a neuronal system can be partly seen from a picture in a book on computational neuroscience edited by Jianfeng that I am writing a chapter for
How do we go from real neurons to artificial ones? Hillock input output
Single neuron activity Membrane potential is the voltage difference between a neuron and its surroundings (0 mV) Cell 0 Mv Membrane potential
Single neuron activity If you measure the membrane potential of a neuron and print it out on the screen, it looks like: spike
Single neuron activity A spike is generated when the membrane potential is greater than its threshold
Abstraction So we can forget all sub-threshold activity and concentrate on spikes (action potentials), which are the signals sent to other neurons Spikes
Only spikes are important since other neurons receive them (signals) Neurons communicate with spikes Information is coded by spikes So if we can manage to measure the spiking time, we decipher how the brain works ….
Again its not quite that simple spiking time in the cortex is random
With identical input for the identical neuron spike patterns are similar, but not identical
Recording from a real neuron: membrane potential
Single spiking time is meaningless To extract useful information, we have to average to obtain the firing rate r for a group of neurons in a local circuit where neuron codes the same information over a time window Local circuit = Time window = 1 sec r = Hz
So we can have a network of these local groups w 1: synaptic strength wnwn r1r1 rnrn Hence we have firing rate of a group of neurons
r i is the firing rate of input local circuit The neurons at output local circuits receives signals in the form The output firing rate of the output local circuit is then given by R where f is the activation function, generally a Sigmoidal function of some sort w i weight, (synaptic strength) measuring the strength of the interaction between neurons.
Artificial Neural networks Local circuits (average to get firing rates) Single neuron (send out spikes)
Artificial Neural Networks (ANNs) A network with interactions, an attempt to mimic the brain UNITs: artificial neuron (linear or nonlinear input- output unit), small numbers, typically less than a few hundred INTERACTIONs: encoded by weights, how strong a neuron affects others STRUCTUREs: can be feedforward, feedback or recurrent It is still far too naïve as a brain model and an information processing device and the development of the field relies on all of us
The general artificial neuron model has five components, shown in the following list. (The subscript i indicates the i-th input or weight.) 1. A set of inputs, x i.inputs 2. A set of weights, w i.weights 3. A bias, u. 4. An activation function, f.activation function 5. Neuron output, y
Thus the key to understanding ANNs is to understand/generate the local input-output relationship