1. Correlation Matrix Memory CS/CMPE 333 – Neural Networks

2. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS2 Memory – Biological and Physical What are the characteristics of biological and physical memory systems? In the neurobiological context, memory refers to the relatively enduring neural alterations induced by the interactions of the organism with its environment  Store and recall capabilities Our memory is physically distributed and operates by association (rather than addressing)  Content-addressable memory What about memory in a computer?

3. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS3 Associative Memory (1) Associative memory – content-addressable memory: memory that operates by association, mapping of an output to an input (stimulus to output)  Example: face recognition Characteristics of associative memory  Distributed  Both key (stimulus) and output (stored pattern) are data vectors (as opposed to address and data)  Storage pattern different from key and is spatially distributed  Stimulus is address as well as stored pattern (with imperfections)  Tolerant to noise and damage  Same elements share in storage of different patterns

4. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS4 Associative Memory (2) Auto-associative memory – a key vector is associated with itself in memory  Dimension of input and output are same Hetero-associative memory – a key vector is associated (paired) with another memorized vector  Dimension of input and output may or may not be the same Linear associative memory – linear mapping from input (stimulus) to output b = Ma Nonlinear associative memory – nonlinear mapping from input to output b = g(M, a)a

5. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS5 Distributed Memory Mapping (1) Distributed memory – simultaneous or near simultaneous activities of many different neurons that contain information about the external stimuli (keys) Distributed memory mapping – transform an activity pattern in the input space to an activity pattern in the output space

6. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS6 Distributed Memory Mapping (2)

7. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS7 Distributed Memory Mapping (3) a k = [a k1 …a kp ] T = input activity pattern b k = [b k1 …b kp ] T = output activity pattern  How do we learn from activity patterns a k and b k ? Associations mapping a k -> b k k = 1…q  a k is thus a key pattern and b k the memorized pattern Number of patterns stored by network = q which, in practice, is less than p

8. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS8 Distributed Memory Mapping (4) For linear memories b k = W(k)a k k = 1…q

9. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS9 Distributed Memory Mapping (5) Considering a single output neuron i and pattern k b ki = Σ j=1 p w ij (k)a kj i = 1…p Or in vector notation b ki = w i (k) T a k Considering the entire stored pattern b k (i.e. b ki where i = 1…p) b k = W(k)a k

10. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS10 Distributed Memory Mapping (6) W(k) is the memory matrix for a single pattern k. The memory matrix for the entire set of pattern q is defined as the summation M = Σ k=1 q W(k) M contains a piece of every input-output pattern presented to the memory. Recursive updating: M k = M k-1 + W(k)k = 1…q

11. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS11 Correlation Matrix Memory (1) Define M’ as an estimate of the memory matrix M M’ = Σ k=1 q b k a k T  Outer product b k a k T is an estimate of the weight matrix W(k) Key points of this equation  Outer product rule  Generalized Hebbian learning at the local level  M’ is the correlation memory matrix  This associative memory is called correlation matrix memory

12. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS12 Correlation Matrix Memory (2) Correlation matrix memory M’ can be written as M’ = BA T A = [a 1, a 2,…,a q ] = key matrix B = [b 1, b 2,…,b q ] = memorized matrix And, in recursive form M’ k = M’ k-1 + b k a k T k = 1,…,q  b k a k T is an estimate of weight matrix W(k)

13. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS13 Recall (1) Talking about memory, how is it addressed and how is specific information recalled? What is its capacity? Consider correlation memory matrix M’ present random key a j as stimulus, response is b = M’a j = Σ k=1 q b k a k T a j = Σ k=1 q b k (a k T a j ) rewriting b = Σ k=1;k ≠ j q b k (a k T a j ) + b j (a j T a j )

14. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS14 Recall (2) Assuming a k T a k = 1 for all k (i.e. key vectors are normalized to ‘energy’ 1), then b = Σ k=1;k ≠ j q b k (a k T a j ) + b j = v j + b j  b j = desired response or vector; v j = noise vector  Noise vector results from the crosstalk of a j with all other key vectors  This crosstalk limits the capacity of the memory in reliably storing and recalling information

15. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS15 Recall (3) Assume the key vectors a i are orthonormal, i.e. a j T a k = 1 if j = k and a j T a k = 0 if j ≠ k Then, what is the storage capacity?  It is equal to p What if the key vectors are not orthonormal?  It is equal to the rank of the memory matrix M’ (in general)  Since M’ has the dimensions p x p, its rank cannot be greater than p Thus, correlation matrix memory can reliably store a maximum of p patterns, where p is the input dimension

16. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS16 Correlation Matrix Memory in Practice Real-world patterns are not orthonormal. Hence, storage capacity is less than p (key vector dimension) If key vectors (patterns) are linearly independent, then they can be preprocessed to form an orthonormal set (using Gram-Schmidt procedure) Preprocessing to enhance key vector ‘separation’ (i.e. feature enhancement) helps improve storage capacity and recall performance Correlation matrix memory may recall patterns that it has never seen before (i.e. it can make errors if the key patterns are not orthonormal and/or greater than p)

17. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS17 Error-Correction in CMM (1) Correlation matrix memory has no mechanism for feedback since it is based on Hebbian learning  The consequence is there are errors in recall How do we continually update the memory to reduce the error? Impose error-correction learning and updating of the memory Remember, memory matrix M’ is made up of the weights of the neural network

18. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS18 Error-Correction in CMM (2) At iteration n, when presented with pattern k, memory error is e k (n) = b k – M’(n)a k Correction at iteration n (delta rule) ΔM’(n) = ηe(n)a k T Updated memory matrix M’(n+1) = M’(n) + ΔM’(n)