Ch7: Hopfield Neural Model HNM is a kind of BAM Two versions of Hopfield memory 1. Discrete: (a) sequential, (b) parallel 2. Continuous 7.1. Discrete Hopfield Memory Recall auto-BAM Mathematically, Training set: Weight matrix:

The HNM has no self-loop connection for each unit. HNM Architecture: The HNM has no self-loop connection for each unit. Each unit has an external input signal . 2 2

Weight matrix: 2. Force the diagonal elements of W to be zero ( no self-loop) Input: Output: Threshold to be defined is different from BAM (= 0).

Energy function: : external input (w = 1) : threshold viewed as a negative (inhibitive) input (w = 1) 7.1.1 Sequential (Asynchronous) Hopfield Model Given a set of M binary patterns of N components Weight matrix: Threshold vector:

If stored exemplars are orthogonal, every exemplar □ Energy Function ( with a minus sign) If stored exemplars are orthogonal, every exemplar corresponds to a local minimum of E. Feature space Energy space A particular exemplar is located by looking for its corresponding local minimum in the energy space using a descent approach.

8

where

Let i.e., the net input to unit i. Consider one-bit change, say To decrease energy, should be in consistence with in sign.

□ Algorithm (Sequential Hopfield Model) Input a i. Compute Sequential fashion: ii. Update iii. Repeat until none of elements changes state

□ Convergence proof From (A), on any bit-change. □ Local minimum and attractors Local minimum: a point that has an energy level ≦ any nearest neighbor Attraction: an equilibrium state ※ Local minimum must be attraction, while it is not necessarily true for the reverse

○ Example 1: Two training pattern vectors Weight matrix:

where nullifies the diagonal element Threshold vector: Suppose input vector By cyclic update ordering: i. First iteration (k = 0) Initial vector =

a. lst bit (i = 1) Compute the net input Update Obtain 1st bit updated

b. 2nd bit (i = 2) Compute the net input Update the state Obtain 2nd bit unchanged c. 3rd bit (i = 3) Unchanged d. 4th bit (i = 4) Unchanged

The above can simply be performed as 1. Compute 2. Update

ii. Second iteration 1. Compute 2. Update

iii. Terminate ※ Different ordering retrieves different output ○ Example 2: Convergent state depends on the order of update Two patterns: Weight matrix:

Threshold vector: ※ The output can be obtained by following the energy-descending directions in a hypercube.

i. Energy level for [0 1 1 0]

ii. Energy level for [0 0 1 1] There are more than one directions in which the energy level can descend. The selection of the path is determined by the order of updating bits.

。 Start with with energy -1 Two paths lead to lower energy (0 0 1 0)→(0 0 1 1)/-2, (0 0 1 0)→(1 0 1 0)/-2 depend on left- or right-most bit updated first

7.1.2 Parallel (Synchronous) Hopfield Model □ Weights: ※ The diagonal weights are not set to zero (i.e., having self-loop) Thresholds: 。 Algorithm: During the kth iteration: i. Compute the net input in parallel i = 1, …, N 25

ii. Update the states in parallel Repeat until none of the element changes □ Convergence: At the kth parallel iteration, energy function The energy-level change due to one iteration

∵ W is a nonnegative definite matrix (∵ W formed by outer product Symmetric, nonnegative definite (1), (2)

□ Remarks A local/global minimum must be an attractor. An attractor is not necessarily a local/global minimum. There are many more spurious attractors in the parallel model than sequential model. The parallel model does not get trapped to local minimum as easily as the sequential model (∵ Even if a state is one bit away from a local minimum, it may not be trapped by that attractor because more bits changed in one iteration)

The parallel model appears to outperform the sequential model in terms of percentage of correct retrieval 7.1.3. Capacities of Hopfield and Hamming Networks Capacity: the number of distinct patterns that can be stored in the network. □ If a neural network contain N neurons, the capacity C of the network is at most

Proof: Given p patterns Idea: (i) For a pattern, if it is of sufficiently low probability that any bit may change, then the pattern is considered to be a good attractor (ii) If all the p patterns are good attractors, then the network is said to have a capacity p; otherwise, lower than p.

。Work with bipolar representation Consider an input examplar Ignore and let

Multiply where When changes to When changes to 1 changes when and only when (i.e., or 。 Define bit-error-rate

Suppose If Np large, from central limit theory

Suppose the total error probability < ε criterion of stability discernibility The error probability for each pattern and each neuron (bit) This leads to Take the logarithm If N large, (N dominates)

□ Central Limit Theorem iid r.v. with mean and variance □ Change of variation formulas : differentiable strictly increasing or strictly decreasing function X : a continuous r.v.,

7.2. Continuous Hopfield memory □ Resemble actual neuron having continuous graded output □ An electronic circuit using amplifies, resistors, and capacitors is possibly built using VLSI A PE consists of an amplifier and an inverting amplifier used to simulate inhibitory signal. A resister , where is the weight matrix, is placed at the intersection connecting units i and j. 36

Total input current: external current, linking current leakage current where 37

Treat the circuit as a transient RC circuit, in which charging the capacitor is due to the net-input current, i.e., □ Energy function: c : capacity From (A), Show E is a Lyapunov function From (C), From (B),

Let the output function monotonically increasing functions

7.3.3 The Traveling-Salesperson Problem Constraints: 1. visit each city, 2. only once, 3. minimum distance Brute force: 2 directions fixed starting city

i. A set of n PEs representing n possible positions □ Hopfield solution i. A set of n PEs representing n possible positions for a city in the tour Example tour solution: BAECD A: 01000, B: 10000, C: 00010, D: 00001, E: 00100 ii. Entries of matrix x: city, i: position Matrix representation

( Distance between cities x and y) iii. Energy function Criteria: (a) Each city visited only once (b) Each position appearing on the tour only once (c) Include all cities, (d) Shortest total distance (1) (2) (3) (4) ( Distance between cities x and y)

When the network is stabilized, ideally Term 1: Each row of the matrix contains a single value 1 Term 2: Each column of the matrix contains a single 1 Term 3: Each row and each column contain at most one 1

Term 4: minimum when x, y are not in sequence on the tour 0 when x, y are in sequence on the tour

iv. Weight matrix Defined in terms of inhibitions between PEs (A) Inhibition term from criterion (a) (B) Inhibition term from criterion (b)

(C) Inhibition term from criterion (c) -C -C : constant (global inhibition) (D) Inhibition term from criterion (d) If j = i－1 or i+1, x and y are adjacent cities. two cities far apart should receive a large inhibition

Pattern of inhibitory connections Unit a illustrates the inhibition between units on a single row. Unit b illustrates the inhibition between units on a single column. Unit c shows the inhibition of units in adjacent columns. The global inhibition is not shown.

。 Evolution of the network where Discretize: (1-D) (2-D)

Substitute into

Update: Output: 。 Example: n = 10 (cities) Select A = B = 500, C = 200, D = 500 Initialize s.t.

1. Initialize s.t. 4. Update: 6. Repeat steps 1-5 until stopping criteria are satisfied.