1. Artificial Intelligence CSC 361
Dr. Yousef Al-Ohali
Computer Science Depart.
CCIS – King Saud University
Saudi Arabia

2. Intelligent Systems Part II: Neural Nets

3. Developing Intelligent Program Systems
Machine Learning : Neural Nets
Artificial Neural Networks: Artificial Neural Networks are crude attempts to model the highly massive parallel and distributed processing we believe takes place in the brain.
Two main areas of activity:
Biological: Try to model biological neural systems.
Computational: develop powerful applications.

4. Developing Intelligent Program Systems
Machine Learning : Neural Nets
Neural nets can be used to answer the following:
Pattern recognition: Does that image contain a face?
Classification problems: Is this cell defective?
Prediction: Given these symptoms, the patient has disease X
Forecasting: predicting behavior of stock market
Handwriting: is character recognized?
Optimization: Find the shortest path for the TSP.

5. Developing Intelligent Program Systems
Machine Learning : Neural Nets
Strength and Weaknesses of ANN
Examples may be described by a large number of attributes (e.g., pixels in an image).
Data may contain errors.
The time for training may be extremely long.
Evaluating the network for a new example is relatively fast.
Interpretability of the final hypothesis is not relevant (the NN is treated as a black box).

6. Artificial Neural Networks
Biological Neuron

7. The Neuron
The neuron receives nerve impulses through its dendrites. It then sends the nerve impulses through its axon to the terminal buttons where neurotransmitters are released to simulate other neurons.

8. The neuron The unique components are:
Cell body or soma which contains the nucleus
The dendrites
The axon
The synapses

9. The neuron - dendrites
The dendrites are short fibers (surrounding the cell body) that receive messages
The dendrites are very receptive to connections from other neurons.
The dendrites carry signals from the synapses to the soma.

10. The neuron - axon
The axon is a long extension from the soma that transmits messages
Each neuron has only one axon.
The axon carries action potentials from the soma to the synapses.

11. The neuron - synapses
The synapses are the connections made by an axon to another neuron. They are tiny gaps between axons and dendrites (with chemical bridges) that transmit messages
A synapse is called excitatory if it raises the local membrane potential of the post synaptic cell.
Inhibitory if the potential is lowered.

12. Artificial Neural Networks
History of ANNs

13. History of Artificial Neural Networks
1943: McCulloch and Pitts proposed a model of a neuron --> Perceptron
1960s: Widrow and Hoff explored Perceptron networks (which they called “Adalines”) and the delta rule.
1962: Rosenblatt proved the convergence of the perceptron training rule.
1969: Minsky and Papert showed that the Perceptron cannot deal with nonlinearly-separable data sets---even those that represent simple function such as X-OR.
: Very little research on Neural Nets
1986: Invention of Backpropagation [Rumelhart and McClelland, but also Parker and earlier on: Werbos] which can learn from nonlinearly-separable data sets.
Since 1985: A lot of research in Neural Nets

14. Artificial Neural Networks
artificial Neurons

15. Artificial Neuron
Incoming signals to a unit are combined by summing their weighted values
Output function: Activation functions include Step function, Linear function, Sigmoid function, … 1
f()
Inputs
Output=f()
xiwi x1 xp w1 w0 wp

16. Activation functions Linear function Step function Sign function
Sigmoid
(logistic) function
sign(x) = +1, if x >= 0
-1, if x < 0
step(x) = 1, if x >= threshold
0, if x < threshold
(in picture above, threshold = 0)
pl(x) =x
sigmoid(x) = 1/(1+e-x)
Adding an extra input with activation a0 = -1 and weight
W0,j = t (called the bias weight) is equivalent to having a
threshold at t. This way we can always assume a 0 threshold.

17. Real vs. Artificial Neurons
axon
dendrites
synapse
cell x0 xn w0 wn o
Threshold units

18. Neurons as Universal computing machine
In 1943, McCulloch and Pitts showed that a synchronous assembly of such neurons is a universal computing machine. That is, any Boolean function can be implemented with threshold (step function) units.

19. Implementing AND x1 x2
o(x1,x2) 1 -1
W=1.5

20. Implementing OR x1 x2 o(x1,x2) 1 -1 W=0.5
o(x1,x2) = 1 if –0.5 + x1 + x2 > 0
= 0 otherwise

21. Implementing NOT x1
o(x1) -1
W=-0.5

22. Implementing more complex Boolean functions1
0.5 -1
x1 or x2 x3
1.5
(x1 or x2) and x3

23. Artificial Neural Networks
When using ANN, we have to define:
Artificial Neuron Model
ANN Architecture
Learning mode

24. Artificial Neural Networks
ANN Architecture

25. ANN Architecture
Feedforward: Links are unidirectional, and there are no cycles, i.e., the network is a directed acyclic graph (DAG). Units are arranged in layers, and each unit is linked only to units in the next layer. There is no internal state other than the weights.
Recurrent: Links can form arbitrary topologies, which can implement memory. Behavior can become unstable, oscillatory, or chaotic.

26. Artificial Neural Network Feedforward Network
Output layer
Input layer
Hidden layers
fully connected
sparsely connected

27. Artificial Neural Network FeedForward Architecture
Information flow unidirectional
Multi-Layer Perceptron (MLP)
Radial Basis Function (RBF)
Kohonen Self-Organising Map (SOM)

28. Artificial Neural Network Recurrent Architecture
Feedback connections
Hopfield Neural Networks: Associative memory
Adaptive Resonance Theory (ART)

29. Artificial Neural Network Learning paradigms
Supervised learning:
Teacher presents ANN input-output pairs,
ANN weights adjusted according to error
Classification
Control
Function approximation
Associative memory
Unsupervised learning:
no teacher
Clustering

30. ANN capabilities Learning Approximate reasoning
Generalisation capability
Noise filtering
Parallel processing
Distributed knowledge base
Fault tolerance

31. Main Problems with ANN
Contrary to Expert sytems, with ANN the Knowledge base is not transparent (black box)
Learning sometimes difficult/slow
Limited storage capability

32. Some applications of ANNs
Pronunciation: NETtalk program (Sejnowski & Rosenberg 1987) is a neural network that learns to pronounce written text: maps characters strings into phonemes (basic sound elements) for learning speech from text
Speech recognition
Handwritten character recognition:a network designed to read zip codes on hand-addressed envelops
ALVINN (Pomerleau) is a neural network used to control vehicles steering direction so as to follow road by staying in the middle of its lane
Face recognition
Backgammon learning program
Forecasting e.g., predicting behavior of stock market

33. When to use ANNs?
Input is high-dimensional discrete or real-valued (e.g. raw sensor input).
Inputs can be highly correlated or independent.
Output is discrete or real valued
Output is a vector of values
Possibly noisy data. Data may contain errors
Form of target function is unknown
Long training time are acceptable
Fast evaluation of target function is required
Human readability of learned target function is unimportant
⇒ ANN is much like a black-box