CS623: Introduction to Computing with Neural Nets (lecture-11) Pushpak Bhattacharyya Computer Science and Engineering Department IIT Bombay

Hopfield net (recap of main points) Inspired by associative memory which means memory retrieval is not by address, but by part of the data. Consists of N neurons fully connected with symmetric weight strength wij = wji No self connection. So the weight matrix is 0-diagonal and symmetric. Each computing element or neuron is a linear threshold element with threshold = 0.

Connection matrix of the network, 0-diagonal and symmetric j n1 n2 n3 . . . nk n1 n2 n3 . . . nk i wij 0 – diagonal

Example w12 = w21 = 5 w13 = w31 = 3 w23 = w32 = 2 s1(t) = 1 s2(t) = -1 At time t=0 s1(t) = 1 s2(t) = -1 s3(t) = 1 Unstable state: Neuron 1 will flip. A stable pattern is called an attractor for the net.

Concept of Energy Energy at state s is given by the equation:

Relation between weight vector W and state vector X Transpose of state vector For example, in figure 1, At time t = 0, state of the neural network is: s(0) = <1, -1, 1> and corresponding vectors are as shown. 3 2 1 5 -1 Fig. 1

W.XT gives the inputs to the neurons at the next time instant This shows that the n/w will change state

Theorem In the asynchronous mode of operation, the energy of the Hopfield net always decreases. Proof:

Proof Let neuron 1 change state by summing and comparing We get following equation for energy

Proof: note that only neuron 1 changes state Since only neuron 1 changes state, xj(t1)=xj(t2), j=2, 3, 4, …n, and hence

Proof (continued) Observations: When the state changes from -1 to 1, (S) has to be +ve and (D) is –ve; so ΔE becomes negative. When the state changes from 1 to -1, (S) has to be -ve and (D) is +ve; so ΔE becomes negative. Therefore, Energy for any state change always decreases. (D) (S)

The Hopfield net has to “converge” in the asynchronous mode of operation As the energy E goes on decreasing, it has to hit the bottom, since the weight and the state vector have finite values. That is, the Hopfield Net has to converge to an energy minimum. Hence the Hopfield Net reaches stability.

Training of Hopfield Net Early Training Rule proposed by Hopfield Rule inspired by the concept of electron spin Hebb’s rule of learning If two neurons i and j have activation xi and xj respectively, then the weight wij between the two neurons is directly proportional to the product xi ·xj i.e.

Hopfield Rule Training by Hopfield Rule Train the Hopfield net for a specific memory behavior Store memory elements How to store patterns?

Hopfield Rule To store a pattern <xn, xn-1, …., x3, x2, x1> make Storing pattern is equivalent to ‘Making that pattern the stable state’

Training of Hopfield Net Establish that <xn, xn-1, …., x3, x2, x1> is a stable state of the net To show the stability of impress at t=0 <xtn, xtn-1, …., xt3, xt2, xt1>

Training of Hopfield Net Consider neuron i at t=1

Establishing stability

Example We want <1, -1, 1> as stored memory C B A 1 -1 We want <1, -1, 1> as stored memory Calculate all the wij values wAB = 1/(3-1) * 1 * -1 = -1/2 Similarly wBC = -1/2 and wCA = ½ Is <1, -1, 1> stable? Initially C B A -0.5 0.5 1 -1 After calculating weight values

Observations How much deviation can the net tolerate? What if more than one pattern is to be stored?

Storing k patterns Let the patterns be: Generalized Hopfield Rule is: P1 : <xn, xn-1, …., x3, x2, x1>1 P2 : <xn, xn-1, …., x3, x2, x1>2 . Pk : <xn, xn-1, …., x3, x2, x1>k Generalized Hopfield Rule is: Pth pattern

Storing k patterns Study the stability of <xn, xn-1, …., x3, x2, x1> Impress the vector at t=0 and observer network dynamics Looking at neuron i at t=1, we have

Examining stability of the qth pattern

Examining stability of the qth pattern Small when k << n

Storing k patterns Condition for patterns to be stable on a Hopfield net with n neurons is: k << n The storage capacity of Hopfield net is very small Hence it is not a practical memory element