1. Artificial Neural Networks and applications
Dr. L. Iliadis
Assis. Professor Democritus University of Thrace, Greece

2. Overview Definition of a Neural Network The Human Brain Neuron Models
Artificial neural networks ANN
Historical Notes
ANN Architecture
Learning processes – Training and Testing ANN
7.1. Backpropagation Learning
8. Well Known Applications of ANN

3. What is a Neural Network?
A Neural Network is a collection of units connected in some pattern to allow communication between them and it acts as a massively distributed processor.
These units are also referred to as neurons or nodes. The Neural Network has a natural propensity for storing experiential knowledge and making it available for use.
It has two main characteristics:
Knowledge is acquired by the network from its environment through a learning process.
Interneuron connection strengths, known as synaptic weights, are used to store the acquired knowledge.
Animals are able to react adaptively to changes in their external and internal environment, and they use their nervous system to perform these behaviours. This is called Plasticity.

4. Zoom on Human Brain Neurons
Axon
Soma (Cell Body)
Dendrites

5. Biological Analogy of a Neuron
Dendrites
Soma (cell body)
Axon

6. The Role of the synapses
dendrites
axon
synapses
The synapses are responsible for the information transmission between two connected Neurons

7. Structural Organization of Levels in the Brain

9. Functional Areas of the Brain
Primary motor: voluntary movement
Primary somatosensory: tactile, pain, pressure, position, temp., mvt.
Motor association: coordination of complex movements
Sensory association: processing of multisensorial information
Prefrontal: planning, emotion, judgement
Speech center (Broca’s area): speech production and articulation
Wernicke’s area: comprehension
of speech
Auditory: hearing
Auditory association: complex auditory processing
Visual: low-level vision
Visual association: higher-level
vision

10. HUMAN BRAIN VERSUS SILLICON
The Human cortex has approximately 10 billion neurons and 60 trillion synapses. The net results is that the Brain is an enormously efficient structure
The Energetic Efficiency of the Brain is approximately Joules per Operation per second whereas the corresponding value for the best computers in use today is about 10-6 Joules per Operation per second
Human Brain Neurons (where events happen in the millisecond range 10-3 sec) are 5 or 6 orders of magnitude slower than silicon logic gates (where events happen in the nanosecond range 10-9 sec).

11. ARTIFICIAL NEURON MODEL
A signal Xi at the input of synapse i, connected to neuron k is multiplied by the synaptic weight wi An Adder is Summing the input signals weighted by the respective synapses An activation function for limiting the amplitude of the output of a neuron. It squashes the permissible amplitude range of output signal to a finit value in the interval [0,1] or alternatively in [-1,1].
ARTIFICIAL NEURON MODEL
Bias bk x1 x2 x3 …
xn-1 xn w1
Output w2
Inputs y w3 . .
Activation function .
wn-1
Summing function wn
Synaptic Weights

12. Artificial neural Networks
Artificial Neural Networks consist of interconnected elements which were inspired from studies of biological nervous Systems. They are an attempt to create machines that work in a similar way to the human brain, using components that behave like biological neurons.
Synaptic strengths are translated as synaptic weights;
Excitation means positive product between the incoming spike rate and the corresponding synaptic weight;
Inhibition means negative product between the incoming spike rate and the corresponding synaptic weight;

13. Neuron’s Output Nonlinear generalization of the neuron:
Sigmoidal or Gaussian functions may be used
Where y is the neuron’s output, x is the vector of inputs, and w is the vector of synaptic weights.
Sigmoidal neuron
Gaussian neuron

14. Software or Hardware?
Although ANN can be implemented as fast Hardware devices, much research has been performed using conventional computer running software simulations.
Software Simulations provide a somewhat cheap and flexible environment in which to research ideas for many real-world applications as there exists an adequate performance. e.g. An ANN Software package might be used to develop a System from Credit Scoring of an individual who applies for a Bank Loan.

15. Historical Notes
Ramon y Cajal in 1911 introduced the idea of neurons as structural constituents of the brain
The origins of ANN go way back to the 1940s, when McCulloch and Pitts published the first mathematical model of a biological Neuron
Research on ANN stopped for more than 20 years.
In the mid 1980s emerged a huge interest in ANN due to the publication of the book “Parallel Distributed Processors” by Rumelhart and McClelland
ANN made a grate come-back in the 1990’s and they are now widely accepted as a tool in the development of Intelligent Systems

16. Architecture of Artificial Neural Networks
Hidden Layer
Input Layer
Output Layer
The way that the artificial neurons are linked together to compose an ANN may vary according to its Architecture. The above architectural graph illustrates the layout of a Multilayer Feedforward (data flow only to one direction) ANN in the case of a single Hidden Layer. The Hidden Layer is where the process takes place.

17. Supervised Learning Learning = learning by adaptation
For example: Animals learn that the green fruits are sour and the yellowish/reddish ones are sweet. The learning happens by adapting the fruit picking behavior.
Learning can be perceived as an optimisation process. When an ANN is in its SUPERVISED training or learning phase, there are three factors to be considered:
The Inputs applied are chosen from a Training Set, where the desired response of the System to these Inputs is Known
The Actual Output Produced when an Input Pattern is applied, is compared to the desired Output and an Error is estimated
In ANN the learning occurs by changing the synaptic strengths (change of the weights) eliminating some synapses, and building new ones.

18. PERCEPTRON
The Perceptron is one of the early ANN which is built around a nonlinear neuron, namely the McCulloch-Pitts neuron model.
It produces an output equal to +1 if the hard limiter input is positive, and -1 if its is negative.

19. Learning with a Perceptron
The synapse strength modification rules for artificial neural networks
can be derived by applying mathematical optimisation methods
Perceptron:
Input Data:
Error:
Learning (Weight Adjustment):

20. Calculation of the weight changes is too complicated
Learning with MLP ANN
MLP Multi Layer Process ANN:
with p layers
yout x
… p-1 p
Data:
Error:
Calculation of the weight changes is too complicated

21. Backpropagation Learning
It was developed by Werbos but Rumelhart et. al. in 1986 gave a new lease of life to ANN. The Weight adaption rule is known as Backpropagation.
It defines 2 sweeps of the ANN. First it performs a forward sweep from the input layer. Thus it calculates first the changes for the synaptic weights of the output neuron;
Then is performs a backward sweep from the output layer to the Input. In this way it calculates the changes backward starting from layer p-1, and propagates backward the local error terms.
The bakcward sweep is similar to the forward, except that error values are propagated back through the ANN to determine how the weights are to be changed during training.

22. EXTDBD Learning Rule
The Extended Delta Bar Delta is a Heuristic technique that has been used successfully in a wide range of applications and its main characteristic is that it uses a termed momentum. More specifically, a term is added to the standard weight change, which is proportional to the previous weight change. In this way good general trends are reinforced and oscillations are damped.

23. EVALUATION INSTRUMENTS
The RMS Error adds up the squares of the errors for each PE in the output layer, divides by the number of PEs in the output layer to obtain an average and then takes the square root of that average hence the name “root square”.
Also another instrument is the Common Mean Correlation (CMC) coefficient of the desired (d) and the actual (predicted) output (y) across the Epoch. The CMC is calculated by
where and
It should be clarified that d stands for the desired values, y for the predicted values and i ranges from 1 to n (the number of cases in the data training set) and E is the Epoch size which is the number of sets of training data presented to the ANN learning cycles between weight updates.

24. TESTING AND OVERTRAINING
Over-Training is a very serious problem!!!
Testing is the process that actually determines the strength of the ANN
and its ability to generalize.
The performance of an ANN is critically dependent on the training data
that must be representative of the task to learn (Callan, 1999).
For this purpose in the Testing phase we chose randomly
A lomg set of actual cases (records) that were not applied in the
training phase.

25. New methods for learning with neural networks
Bayesian learning:
the distribution of the neural network parameters is learnt
Support vector learning:
the minimal representative subset of the available data is used to calculate the synaptic weights of the neurons

26. Tasks Performed by Artificial neural networks
The following tasks are usually performed by ANN
Controlling the movements of a robot based on self-perception and other information (e.g., visual information)
Decision making
Pattern Recognition (e.g. recognizing a visual object, a familiar face)

27. ANN tasks These can be reformulated in general as
FUNCTION APPROXIMATION
tasks.
Control
Classification
Prediction
Approximation
With the term Approximation we mean: given a set of values of a function g(x) build a neural network that approximates the g(x) values for any input x.

28. Learning to approximate
Error measure:
Rule for changing the synaptic weights:
Where c is the learning parameter (usually a constant)

29. Summary
Artificial neural networks are inspired by the learning processes that take place in biological systems.
Artificial neurons and neural networks try to imitate the working mechanisms of their biological counterparts.
Learning can be perceived as an optimisation process.
Biological neural learning happens by the modification of the synaptic strength. Artificial neural networks learn in the same way.
The synapse strength modification rules for artificial neural networks can be derived by applying mathematical optimisation methods.

30. Summary
Learning tasks of artificial neural networks can be reformulated as function approximation tasks.
Neural networks can be considered as nonlinear function approximating tools (i.e., linear combinations of nonlinear basis functions), where the parameters of the networks should be found by applying optimisation methods.
The optimisation is done with respect to the approximation error measure.
In general it is enough to have a single hidden layer neural network (MLP, RBF or other) to learn the approximation of a nonlinear function. In such cases general optimisation can be applied to find the change rules for the synaptic weights.

31. DEVELOPING ANN

32. DETERMINING ANN’S TOPOLOGY