1. Presentation By Utkarsh Trivedi Y8544
Hopfield Networks
Presentation By
Utkarsh Trivedi
Y8544

2. Topics Covered What is Hopfield Network Some interesting facts
Major Applications
Mathematical model of HN’s
Learning HNs through examples

3. What is Hopfield Network ??
According to Wikipedia, Hopfield net is a form of recurrent artificial neural network invented by John Hopfield. Hopfield nets serve as content-addressable memory systems with binary threshold units. They are guaranteed to converge to a local minimum, but convergence to one of the stored patterns is not guaranteed.

4. What are HN (informally)
Hopfield Network
These are single layered recurrent networks
All the neurons in the network are fedback from all other neurons in the network
The states of neuron is either +1 or -1 instead of (1 and 0) in order to work correctly.
No of the input nodes should always be equal to no of output nodes
Following figure shows a Hopfield network with four nodes

5. Some Interesting Facts ….
Hopfield Network
Some Interesting Facts ….
The recall pattern of Hopfield network is similar to our recall mechanism.
Both are based on content addressable memory
If some of the neurons of network are destroyed the performance is degraded but some network capabilities may be retained even with major network damage. Just like our brains
Did you know that we are similar

6. Major Applications Recalling or Reconstructing corrupted patterns
Hopfield Network
Major Applications
Recalling or Reconstructing corrupted patterns
Large-scale computational intelligence systems
Handwriting Recognition Software
Practical applications of HNs are limited because number of training patterns can be at most about 14% the number of nodes in the network.
If the network is overloaded -- trained with more than the maximum acceptable number of attractors -- then it won't converge to clearly defined attractors.

7. Mathematical Modeling of HN’s
Hopfield Network
Mathematical Modeling of HN’s

8. Mathematical Modeling of HN’s
Hopfield Network
Mathematical Modeling of HN’s

9. Mathematical Modeling of HN’s
Hopfield Network
Mathematical Modeling of HN’s
Consider signum function to be neuron’s activation function.
i.e.
vi = +1 if hi>0
vi = -1 if hi<0

10. Mathematical Modeling of HN’s
Hopfield Network
Mathematical Modeling of HN’s
Liapunov Energy function :-

11. Power Of Hopfield Networks
We want to understand how to achieve this kind of performance form simple Hopfield networks

12. Learning HNs through simple example
Hopfield Network
Learning HNs through simple example
There are various ways to train these kinds of networks like back propagation algorithm , recurrent learning algorithm, genetic algorithm but there is one very simple algorithm to train these simple networks called ‘One shot method’.
I will be using this algorithm in order to train the network. Oa Ob Oc
W3,2
W1,2
W2,1
W3,1
W1,3
W3,3
W2,2
W2,3
W1,1

13. Learning HNs through simple example
Hopfield Network
Learning HNs through simple example
Lets train this network for following patterns
Pattern 1: ie Oa(1)=-1,Ob(1)=-1,Oc(1)=1
Pattern 2: ie Oa(2)=1,Ob(2)=-1,Oc(3)=-1
Pattern 3: ie Oa(3)=-1,Ob(3)=1,Oc(3)=1
w1,1 = 0
w1,2 = OA(1) × OB(1) + OA(2) × OB(2) + OA(3) × OB(3) = (-1) × (-1) + 1 × (-1) + (-1) × 1 = 1
w1,3 = OA(1) × OC(1) + OA(2) × OC(2) + OA(3) × OC(3) = (-1) × × (-1) + (-1) × 1 = -3
w2,2 = 0
w2,1 = OB(1) × OA(1) + OB(2) × OA(2) + OB(3) × OA(3) = (-1) × (-1) + (-1) × × (-1) = -1
w2,3 = OB(1) × OC(1) + OB(2) × OC(2) + OB(3) × OC(3) = (-1) × 1 + (-1) × (-1) + 1 × 1 = 1
w3,3 = 0
w3,1 = OC(1) × OA(1) + OC(2) × OA(2) + OC(3) × OA(3) = 1 × (-1) + (-1) × × (-1) = -3
w3,2 = OC(1) × OB(1) + OC(2) × OB(2) + OC(3) × OB(3) = 1 × (-1) + (-1) × (-1) + 1 × 1 = 1

14. Learning HNs through example
Hopfield Network
Learning HNs through example
Moving onto little more complex problem described in Haykin’s Neural Network Book
They book used N=120 neuron and trained network with 120 pixel images where each pixel was represented by one neuron.
Following 8 patterns were used to train neural network.

15. Learning HNs through example
Hopfield Network
Learning HNs through example
In order to recognizing power of HNs
For this they need corrupted image. They flipped the value of each pixel with p=0.25.
Using these corrupted images trained HN was run. And after certain number of iteration the output images converged to one of the learned pattern.
Next slides shows the results that they obtained

16. Learning HNs through example
Hopfield Network
Learning HNs through example

17. Learning HNs through example
Hopfield Network
Learning HNs through example

18. Flow Chart summarizing overall process
Hopfield Network
Flow Chart summarizing overall process
Train HN using Standard patterns
Update weight vectors of Network
Run the trained network with corrupted pattern
Network returns the decrypted pattern

19. Hopfield Network
Shortcomings of HNs
Training patterns can be at most about 14% the number of nodes in the network.
If more patterns are used then
the stored patterns become unstable;
spurious stable states appear (i.e., stable states which do not correspond with stored patterns).
Can sometimes misinterpret the corrupted pattern.

20. Hopfield Network
Shortcomings of HNs

21. References Zurada :- Introduction to Artificial Neural Systems
Hopfield Network
References
Zurada :- Introduction to Artificial Neural Systems
Haykins :- Neural Networks, A Comprehensive Foundation
J. J. Hopfield, "Neural networks and physical systems with emergent collective computational abilities",1982
R. Rojas: Neural Networks, Springer-Verlag, Berlin, 1996
Wikipedia :-

22. Hopfield Network
Thank you …..

23. Hopfield Network
Questions…
What is the major difference between HN and fully recurrent networks?
What is content addressable memory and how is it different from RAM?
What will happen if we train HN for only one pattern?
If we train a HN with a pattern will it be automatically trained for its inverse ?
Why can’t we increase number of nodes in network in order to overcome its shortcomings?
(ignore the increased computation complexity or time)