1. NEURAL NETWORKS Biological analogy
Introduction to Artificial Neural Networks
Typical architectures

2. Biological neuron Soma: body of the neuron.
Dendrites: receptors (inputs) of the neuron.
Axon: output of neuron; connected to dendrites of other neurons via synapses.
Synapses: transfer of information between neurons (electrical-chemical-electrical).
João Sousa, José Borges XI.2

3. Neural networks Biological neural networks Artificial neural networks
Neuron switching time: second
Number of neurons: 1010
Connections per neuron (synapses): 104,5
Recognition time: 0.1 s
parallel computation
Artificial neural networks
Weighted connections amongst units
Highly parallel, distributed process
Emphasis on tuning weights automatically
João Sousa, José Borges XI.3

4. Artificial Neural Networks
Artificial Neuron
Threshold function
Piece-wise Linear
Sigmoidal function
João Sousa, José Borges XI.4

5. Use of Artificial Neural Networks
Input is high-dimensional
Output is multidimensional
Mathematical form of system is unknown
Interpretability of identified model is unimportant
Applications
Pattern recognition
Classification
Prediction
Modeling
Biological neural network
Artificial neural network
Soma
Neuron
Dendrite
Input
Axon
Output
Synapse
Weight
João Sousa, José Borges XI.5

6. Architectures of typical ANN
Feedforward ANN
João Sousa, José Borges XI.6

7. Architectures of typical ANN
Recurrent ANN
João Sousa, José Borges XI.7

8. ADAPTIVE NETWORKS
Adaptive ANN
Network Classification
Backpropagation

9. Adaptive (neural) networks
Massively connected computational units inspired by the working of the human brain
Provide a mathematical model for biological neural networks (brains)
Characteristics:
learning from examples
adaptive and fault tolerant
robust for fulfilling complex tasks
João Sousa, José Borges XI.9

10. Network classification
Learning methods: supervised, unsupervised
Architectures: feedforward, recurrent
Output types: binary, continuous
Node types: uniform, hybrid
Implementations: software, hardware
Connection weights: adjustable, hard-wired
Inspirations: biological, psychological
João Sousa, José Borges XI.10

11. Adaptive network Nodes can be static or parametric
Network can consist of heterogeneous nodes
Links do not have parameters associated
Node functions are differentiable except at a finite number of points
fixed nodes 3 x1 6 8 x8 4 7 9 x9 x2 5
adaptive nodes
Input layer
Layer 1
Layer 2
Output layer
João Sousa, José Borges XI.11

12. Calculating with a networkx g u y h v a x f y x a f y a y h v x g u
João Sousa, José Borges XI.12

13. Backpropagation learning rule
Simple gradient descent applied to layered networks
An overall error measure is minimized for P data points and L layers
Derivative information propagated by the use of chain rule,
change in parameter a
change in outputs of nodes containing a
change in network's outputs
change in error measure
João Sousa, José Borges XI.13

14. Ordered vs. partial derivativesy x f g z
partial derivative
ordered derivative
João Sousa, José Borges XI.14

15. BP for feedforward networks
Define an error signal at each node
output node
hidden layer node
João Sousa, José Borges XI.15

16. Error propagation networkx1 x2 4 5 3 6 7 9 8 x8 x9 e1 e2 4 5 3 6 7 9 8 e8 e9
w83
w97
w52
w75
w31
João Sousa, José Borges XI.16