1 Neural networks 3

2 Hopfield network (HN) model A Hopfield network is a form of recurrent artificial neural network invented by John Hopfield in 1982 The network is entirely interconnected –All neurons are both input and output with binary threshold units 1, if x > 0 F(x) = -1, otherwise These are single layered recurrent networks

3 Hopfield network model Applications Recalling or Reconstructing corrupted patterns Large-scale computational intelligence systems Handwriting Recognition Software Hopfield is model used to solve optimization problems such as traveler salesman, task scheduling, etc. Practical applications of HNs are limited because number of training patterns can be at most about 14% the number of nodes in the network. If the network is overloaded -- trained with more than the maximum acceptable number of attractors -- then it won't converge to clearly defined attractors. These are single layered recurrent networks

4 Hopfield network model Hopfield nets serve as content-addressable memory systems with binary threshold nodes (associative memory) –The information is distributed and stored in the network connection. If one element disappear, the information is not lost We access to information using memory content ( CAM : Content addressable memory) The network build progressively its attractors –In the learning phase neurones adjust there synapses based on there activities and predefined rules –Hebb rule – Widrow-Hoff rule (delta rule)

5 Hopfield network model Associative memory Learning

6 Hopfield network model Learning (Hebb rule): Neurone are symetric: |P|: number of training patterns (input combinations) is ith bit in the pattern P

7 Lets train this network for following patterns Pattern 1:- ie O a(1) =-1,O b(1) =-1,O c(1) =1 Pattern 2:- ie O a(2) =1,O b(2) =-1,O c(3) =-1 Pattern 3:- ie O a(3) =-1,O b(3) =1,O c(3) =1 w1,1 = 0 w1,2 = 1/3( OA(1) × OB(1) + OA(2) × OB(2) + OA(3) × OB(3)) =1/3( (-1) × (-1) + 1 × (-1) + (- 1) × 1) = 1/3 w1,3 = 1/3( OA(1) × OC(1) + OA(2) × OC(2) + OA(3) × OC(3)) = 1/3((-1) × × (-1) + (-1) × 1) = 1/3( -3) w2,2 = 0 w2,1 = 1/3( OB(1) × OA(1) + OB(2) × OA(2) + OB(3) × OA(3)) = 1/3((-1) × (-1) + (-1) × × (-1) = 1/3( -1) w2,3 = 1/3( OB(1) × OC(1) + OB(2) × OC(2) + OB(3) × OC(3)) = 1/3((-1) × 1 + (-1) × (-1) + 1 × 1) = 1/3(1) w3,3 = 0 w3,1 = 1/3( OC(1) × OA(1) + OC(2) × OA(2) + OC(3) × OA(3)) = 1/3( 1 × (-1) + (-1) × × (-1)) = 1/3( -3) w3,2 = 1/3( OC(1) × OB(1) + OC(2) × OB(2) + OC(3) × OB(3)) = 1/3( 1 × (-1) + (-1) × (-1) + 1 × 1) = 1/3( 1) Hopfield network model

8 Utilization : –A pattern is entered in the network by setting all nodes (neurones) to a specific value, or by setting only part of the nodes. –The network is then subject to a number of iterations using asynchronous or synchronous updating. –This is stopped after a while. The network neurons are then read out to see which pattern is in the network.

9 Hopfield network model There are two ways of updating them: Asynchronous: one picks one neuron, calculates the weighted input sum and updates immediately. This can be done in a fixed order, or neurons can be picked at random, which is called asynchronous random updating. Synchronous: the weighted input sums of all neurons are calculated without updating the neurons. Then all neurons are set to their new value, according to the value of their weighted input sum.

10 Hopfield network model Neurone state modification:

11 Hopfield network model Network energy : a stable states have a low energy This value is called the "energy" because the definition ensures that when units are randomly chosen to update, the energy E will either lower in value or stay the same.

12 Hopfield network model Shortcomings of HNs Training patterns can be at most about 14% the number of nodes in the network. If more patterns are used then  the stored patterns become unstable;  spurious stable states appear (i.e., stable states which do not correspond with stored patterns). Can sometimes misinterpret the corrupted pattern.

13 Learning HNs through example Moving onto little more complex problem described in Haykin’s Neural Network Book They book used N=120 neuron and trained network with 120 pixel images where each pixel was represented by one neuron. Following 8 patterns were used to train neural network. Hopfield network model

14 Flow Chart summarizing overall process Network returns the decrypted pattern Run the trained network with corrupted pattern Update weight vectors of Network Train HN using Standard patterns Hopfield network model

15 Learning HNs through example Hopfield network model

16 Learning HNs through example 16 Hopfield network model

17 Shortcomings of HNs Hopfield network model

18 Project (exam)