1. Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden

2. Hopfield Networks Network of McCulloch-Pitts neurons Output is 1 iff and is -1 otherwise

3. Hopfield Networks

6. Associative Memory Problem: Store a set of patterns in such a way that when presented with a new pattern, the network responds by producing whichever of the stored patterns most closely resembles the new pattern.

7. Hopfield Networks Resembles = Hamming distance Configuration space = all possible states of the network Stored patterns should be attractors Basins of attractors

8. Hopfield Networks N neurons Two states: -1 (silent) and 1 (firing) Fully connected Symmetric Weights Thresholds

9. Hopfield Networks w 13 w 16 w 57 +1

10. Hopfield Networks State: Weights: Dynamics:

11. Hopfield Networks Hebb’s learning rule: –Make connection stronger if neurons have the same state –Make connection weaker if the neurons have a different state

12. Hopfield Networks neuron 1synapseneuron 2

13. Hopfield Networks Weight between neuron i and neuron j is given by

14. Hopfield Networks Opposite patterns give the same weights This implies that they are also stable points of the network Capacity of Hopfield Networks is limited: 0.14 N

15. Hopfield Networks Hopfield defines the energy of a network: E = - ½  ij S i S j w ij +  i S i  i If we pick unit i and the firing rule does not change its S i, it will not change E. If we pick unit i and the firing rule does change its S i, it will decrease E.

17. Hopfield Networks Energy function: Alternative Form: Updates:

18. Hopfield Networks

19. Extension: use stochastic fire rule – S i := +1 with probability g(h i ) – S i := -1 with probability 1-g(h i )

20. Hopfield Networks Nonlinear function: x g(x)g(x) g(x) = 1 + e – x  1        0

21. Hopfield Networks Replace the binary threshold units by binary stochastic units. Define  = 1/T Use “temperature” T to make it easier to cross energy barriers. –Start at high temperature where its easy to cross energy barriers. –Reduce slowly to low temperature where good states are much more probable than bad ones. A B C

22. Hopfield Networks Kick the network our of spurious local minima Equilibrium: becomes time independent