1. Artificial Neural Networks

2. Outline Biological Motivation Perceptron Gradient Descent
Least Mean Square Error
Multi-layer networks
Sigmoid node
Backpropagation

3. Biological Neural Systems
Neuron switching time : > 10-3 secs
Number of neurons in the human brain: ~1010
Connections (synapses) per neuron : ~104–105
Face recognition : 0.1 secs
High degree of parallel computation
Distributed representations

4. Artificial Neural Networks
Many simple neuron-like threshold units
Many weighted interconnections
Multiple outputs
Highly parallel and distributed processing
Learning by tuning the connection weights

5. Perceptron: Linear threshold unit
x0=1 x1 w1 w0 w2 x2 S o .
i=0n wi xi wn xn
1 if i=0n wi xi >0
o(xi)=
-1 otherwise {

6. Decision Surface of a Perceptron+ - x1 x2
Xor x2 + + + - - x1 + - -
Linearly Separable
Theorem: VC-dim = n+1

7. Perceptron Learning Rule
S sample
xi input vector
t=c(x) is the target value
o is the perceptron output
 learning rate (a small constant ), assume =1
wi = wi + wi
wi =  (t - o) xi

8. Perceptron Algo. Correct Output (t=o) False Negative (t=-1 and o=1)
Weights are unchanged
Incorrect Output (to)
Change weights !
False Positive (t=1 and o=-1)
Add x to w
False Negative (t=-1 and o=1)
Subtract x from w

9. Perceptron Learning Rule
(x,t)=([2,1],-1)
o=sgn( ) =1
w=[0.25 – ]
x2 = 0.2 x1 – 0.5
o=-1
w=[0.2 –0.2 –0.2]
(x,t)=([-1,-1],1)
o=sgn( )
=-1
(x,t)=([1,1],1)
o=sgn( )
=-1
w=[ ]
w=[-0.2 –0.4 –0.2]

10. Perceptron Algorithm: Analysis
Theorem: The number of errors of the Perceptron Algorithm is bounded
Proof:
Make all examples positive
change <xi,bi> to <bixi, +1>
Margin of hyperplan w

11. Perceptron Algorithm: Analysis II
Let mi be the number of errors of xi
M=  mi
From the algorithm: w=  mixi
Let w* be a separating hyperplane

12. Perceptron Algorithm: Analysis III
Change in weights:
Since w errs on xi , we have wxi <0
Total weight:

13. Perceptron Algorithm: Analysis IV
Consider the angle between w and w*
Putting it all together

14. Gradient Descent Learning Rule
Consider linear unit without threshold and continuous output o (not just –1,1)
o=w0 + w1 x1 + … + wn xn
Train the wi’s such that they minimize the squared error
E[w1,…,wn] = ½ dS (td-od)2
where S is the set of training examples

15. Gradient Descent S={<(1,1),1>,<(-1,-1),1>,
<(1,-1),-1>,<(-1,1),-1>}
(w1,w2)
Gradient:
E[w]=[E/w0,… E/wn]
w=- E[w]
(w1+w1,w2 +w2)
wi=- E/wi
=/wi 1/2d(td-od)2
= /wi 1/2d(td-i wi xi)2
= d(td- od)(-xi)

16. Gradient Descent Gradient-Descent(S:training_examples, )
Until TERMINATION Do
Initialize each wi to zero
For each <x,t> in S Do
Compute o=<x,w>
For each weight wi Do
wi= wi +  (t-o) xi
For each weight wi Do1
wi=wi+wi

17. Incremental Stochastic Gradient Descent
Batch mode : Gradient Descent
w=w -  ES[w] over the entire data S
ES[w]=1/2d(td-od)2
Incremental mode: gradient descent
w=w -  Ed[w] over individual training examples d
Ed[w]=1/2 (td-od)2
Incremental Gradient Descent can approximate Batch Gradient Descent arbitrarily closely if  is small enough

18. Comparison Perceptron and Gradient Descent Rule
Perceptron learning rule guaranteed to succeed if
Training examples are linearly separable
No guarantee otherwise
Linear unit using Gradient Descent
Converges to hypothesis with minimum squared error.
Given sufficiently small learning rate 
Even when training data contains noise
Even when training data not linearly separable

19. Multi-Layer Networks
output layer
hidden layer(s)
input layer

20. Sigmoid Unit S . x0=1 x1 w1 w0 w2 x2 o wn xn z=i=0n wi xi
o=(z)=1/(1+e-z) w2 x2 S o . wn
(z) =1/(1+e-z)
sigmoid function. xn

21. Sigmoid Function (z) =1/(1+e-z) d(z)/dz= (z) (1- (z))
Gradient Decent Rule:
one sigmoid function
E/wi = -d(td-od) od (1-od) xi
Multilayer networks of sigmoid units:
backpropagation

22. Backpropagation: overview
Make threshold units differentiable
Use sigmoid functions
Given a sample compute:
The error
The Gradient
Use the chain rule to compute the Gradient

23. Backpropagation Motivation
Consider the square error
ES[w]=1/2d  S k  output (td,k-od,k)2
Gradient: ES[w]
Update: w=w -  ES[w]
How do we compute the Gradient?

24. Backpropagation: Algorithm
Forward phase:
Given input x, compute the output of each unit
Backward phase:
For each output k compute

25. Backpropagation: Algorithm
Backward phase
For each hidden unit h compute:
Update weights:
wi,j=wi,j+wi,j where wi,j=  j xi

26. Backpropagation: output node

27. Backpropagation: output node

28. Backpropagation: inner node

29. Backpropagation: inner node

30. Backpropagation: Summary
Gradient descent over entire network weight vector
Easily generalized to arbitrary directed graphs
Finds a local, not necessarily global error minimum
in practice often works well
requires multiple invocations with different initial weights
A variation is to include momentum term
wi,j(n)=  j xi +  wi,j (n-1)
Minimizes error training examples
Training is fairly slow, yet prediction is fast

31. Expressive Capabilities of ANN
Boolean functions
Every boolean function can be represented by network with single hidden layer
But might require exponential (in number of inputs) hidden units
Continuous functions
Every bounded continuous function can be approximated with arbitrarily small error, by network with one hidden layer [Cybenko 1989, Hornik 1989]
Any function can be approximated to arbitrary accuracy by a network with two hidden layers [Cybenko 1988]

32. VC-dim of ANN A more general bound. Concept class F(C,G):
G : Directed acyclic graph
C: concept class, d=VC-dim(C)
n: input nodes
s : inner nodes (of degree r)
Theorem: VC-dim(F(C,G)) < 2ds log (es)

33. Proof: Bound |F(C,G)(m)| Find smallest d s.t. |F(C,G)(m)| <2m
Let S={x1, … , xm}
For each fixed G we define a matrix U
U[i,j]= ci(xj), where ci is a specific i-th concept
U describes the computations of S in G
TF(C,G) = number of different matrices.

34. Proof (continue) Clearly |F(C,G)(m)|  TF(C,G)
Let G’ be G without the root.
|F(C,G)(m)|  TF(C,G)  TF(C,G’) |C(m)|
Inductively, |F(C,G)(m)|  |C(m)|s
Recall VC Bound: |C(m)|  (em/d)d
Combined bound |F(C,G)(m)| (em/d)ds

35. Proof (cont.) Solve for: (em/d)ds2m Holds for m  2ds log(es) QED
Back to ANN:
VC-dim(C)=n+1
VC(ANN)  2(n+1) log (es)