1. 2806 Neural Computation Self-Organizing Maps Lecture 9 2005 Ari Visa

2. Agenda n Some historical notes n Some theory n Self-Organizing Map n Learning Vector Quantization n Conclusions

3. Some Historical Notes Local ordering (von der Malsbyrg, 1973) (Amari, 1980) a matematical analysis elucidates the dynamic stability of a cortical map Self-organizing feature map (SOM), (Kohonen 1982) (Erwin, 1992) a convex neighbourhood function should be used (~ Gaussian) The relationship between the SOM and principal curves is dicussed (Ritter, 1992 & Cherkassky and Mulier, 1995)

4. Some Historical Notes n Vector quantization: Lloyd algorithm 1957 for scalar quantization (= Max quantizer ) n The generilized Lloyd algorithm for vector quantization (= k-means algorithm McQueen 1969 = LBG algorithm Linde et al 1980) n The idea of learning vector quantization (LVQ) (Kohonen, 1986) n Convergence properties of the LVQ algorithm using the ordinary differential equation (ODE) (Baras and LaVigna, 1990)

5. Some Theory n The spatial location of an output neuron in a topographic map corresponds to a particular domain or feature of data drawn from the input space. (Kohonen 1990) n Two approaches: Willshaw-von der Malsburg (explain neurobiological details) and Kohonen ( a more general than the first model in the sense that it is capable of performing data reduction).

6. Some Theory n The principle goal of the self-organizing map is to transform an incoming signal pattern of arbitary dimension into a one- or two dimensional discrete map, and to perform this transformation adaptively in a topologically ordered fashion.

7. SOM The formation of the self-organizing map 1. Competition 2. Cooperation 3. Synaptic Adaptation n Competitive Process n x = [x 1, x 2,...,x m ] T n w j = [w j1,w j2,...,w jm ] T n where j = 1,2,...,l (=total number of neurons in the network) n Select the neuron with largest w j T x n i(x) = arg min j ||x – w j || n A continuous input space of activation patterns is mapped onto a discrete output space of neurons by a process of competition among the neurons in the network.

8. SOM Cooperation n How to define a topological neigborhood that is neurobiologically correct? n Let h ij denote the topological neighborhood centered on winning neuron i, and encompassing a set of excited neurons denoted by j. n The topological neighborhood is symmetric about the maximum point n The amplitude of the topological neighborhood decreases monotonically with the increasing lateral distance  h j,i(x) (n) = exp(d² j,i /2  ²(n)) n  (n) =  0 exp(-n/  1 ), n =0,1,2,….

9. SOM Adaptation n w j (n+1) = w j (n) +  (n) h j,i(x) (n)[x-w j (n)], note Hebbian learning n The synaptic weight vector w j of winning neuron i move toward the input vector x. Upon repeated presentations of the training data, the synaptic weight vector tend to follow the distribution of the input vectors due to the neighborhood updating.  topological ordering n The learning-rate parameter  (n) should be time varying.  (n) =  0 exp(-n/  2 ), n =0,1,2,…

10. SOM n Ordering and Convergence n Self-organizing or ordering phase: (max 1000 iterations) n  (n) = [0.1, 0.01],  2 = 1000 n h j,i (n) = [”radius” of the lattice, the winning neuron and a couple neighboring neurons around],  1 = 1000/ log  0 n Convergence phase: (fine tune the feature map, 500*the number of neurons in the network)  (n) = 0.01, h j,i (n) = the winning neuron and one or zero neighboring neurons. (summary of SOM)

11. Some Theory n Property 1. Approximation of the Input Space. The feature map , represented by the set of synaptic weight vectors {w j } in the input space A, provides a good approximation to the input space H. n The theoretical basis of the idea is rooted in vector quantization theory.

12. Some Theory n c(x) acts as an encoder of the input vector x and x’(c) acts as a decoder of c(x). The vector x is selected at random from the training sample, subject to an underlying probability density function f x (x). The optimum encoding-decoding scheme is detemined by varying the functions c(x) and x’(c), so as to minimize the expected distortion defined by.

13. Some Theory n d(x,x’) = ||x-x’||² = (x-x’) T (x-x’) Generalized Lloyd algorithm Condition 1. Given the input vector x, choose the code c = c(x) to minimize the squared error distortion ||x-x’(c)||². Condition 2. Given the code c, compute the reconstuction vector x =x’(c) as the centroid of the input vectors x that satisfy condition 1. The Generalized Lloyd algorithm is closely related to the SOM.

14. Some Theory Condition 1. Given the input vector x, choose the code c = c(x) to minimize the distortion measure D 2. Condition 2. Given the code c, compute the reconstuction vector x =x’(c) to satisfy the conditions. n x’ new (c) ← x’ old (c) +  (c-c(x))[ x-x’ old (c)]

15. Some Theory n Property 2. Topological Ordering. The feature map  computed by the SOM algorithm is topologically ordered in the sense that the spatial location of a neuron in the lattice corresponds to a particular domain or feature of input patterns.

16. Some Theory n Property 3. Density Matching. The feature map  reflects variations in the statistics of the input distributions: regions in the input space H from which the sample vectors x are drawn with a high probability of occurrence are mapped onto larger domains of the output space A, and therefore with better resolution than regions in H from which sample vectors x are drawn with a low probability of occurrence. n Minimum-distortion encoding, according to which the curvature terms and all higher-order terms in the distortion measure due to noise model  ( ) are retained. m(x)  f x ⅓ (x) n Nearest-neighbor encoding, which emerges if the curvature terms are ignored, as in the standard form of the SOM algorithm. m(x)  f x ⅔ (x)

17. Some Theory. n Property 4. Feature Selection. Given data from an input space with an nonlinear distribution, the self-organizing map is able to select a set of best features for approximating the underlying distribution.

18. Learning Vector Quantizer n Vector quantization: an input space is divided into a number of distinct regions, and for each region a reconstruction vector is defined. n A vector quantizer with minimum encoding distortion is called a Voronoi or nearest- neighbor quantizer. n The collection of possible reproduction vectors is called the code book of the quantizer, and its members are called code vectors.

19. Learning Vector Quantizer The SOM algorithm provides an approximate method for computing the Voronoi vectors in unsupervised manner. Learning vector quantization (LVQ) is a supervised learning technique that uses class information to move the Voronoi vectors slightly, so as to improve the quality of the classifier decision regions.

20. Learning Vector Quantizer n An input vector x is picked at random from the input space. If the class labels of the input vector x and a Voronoi vector w agree, the Voronoi vector w is moved in the direction of the input vector x. If the class labels of the input vector x and the Voronoi vector w disagree, the Voronoi vector w is moved away from the input vector x. n Let {w i } 1 i=1 denote the set of Voronoi vectors, and the {x i } N i=1 denote the set of input vectors. LVQ: I. Suppose that the Voronoi vector w c is the closest to the input vector x i. Let L wc denote the class associated with the Voronoi vector w c and L xi denote the class label of the input vector x i. The Voronoi vector w c is adjusted as follows: If L wc = L xi,then w c (n+1) = w c (n) + α n [x i - w c (n)] where 0< α n <1. If L wc ≠ L xi,then w c (n+1) = w c (n) - α n [x i - w c (n)] where 0< α n <1. II. The other Voronoi vectors are not modified.

21. Learning Vector Quantizer Vector quantization is a form of lossy data compression. Rate distortion theory (Gray 1984): Better data compression performance can always be achieved by coding vectors instead of scalars, even if the source of data is memoryless, or if the data compression system has memory. A multistage hierarchical vector quantizer (Luttrell 1989)

22. Learning Vector Quantizer n First-order autoregressive (AR) model: x(n+1) =  x(n) + (n) where  is the AR coefficient and the (n) are independent and identical distributed (iid) Gaussian random variables of zero mean and unit variance.

23. Some Theory n Attribute code x a n Symbol code x s n x =[x s, 0] T + [0, x a ] T n Contextual map or semantic map

24. Summary n The SOM algorithm is neurobiologically inspired, incorporating all the mechanisms that are basic to self-organization: competition, cooperation, and self-amplification. n The Kohonen’s SOM algorithm is so simple to implement, yet matematically so difficult to analyze its properties in a general setting. n The self-organizing map may be viewed as a vector quantizer.