Presentation is loading. Please wait.

Presentation is loading. Please wait.

Computational Intelligence: Methods and Applications Lecture 10 SOM and Growing Cell Structures Włodzisław Duch Dept. of Informatics, UMK Google: W Duch.

Published byJeffry Preston Modified over 9 years ago

Similar presentations

Presentation on theme: "Computational Intelligence: Methods and Applications Lecture 10 SOM and Growing Cell Structures Włodzisław Duch Dept. of Informatics, UMK Google: W Duch."— Presentation transcript:

1 Computational Intelligence: Methods and Applications Lecture 10 SOM and Growing Cell Structures Włodzisław Duch Dept. of Informatics, UMK Google: W Duch

2 Italian olive oil 572 samples of olive oil were collected from 9 Italian provinces. Content of 8 fats was determine for each oil. SOM 20 x 20 network, Maps 8D => 2D. Classification accuracy was around 95-97%. An example of SOM application: Note that topographical relations are preserved, region 3 is most diverse.

3 SOM properties General properties of the algorithm: Hard to prove anything in d=2 or more dimensions about convergence properties of SOM maps. In d=1 analytical results for continuous time (infinitely many small step iterations) show that correct ordering of points is obtained by the SOM algorithm. Reducing the neighborhood too quickly leads to twisted configurations – forming quite wrong representations of data (“strange opinions”?). To avoid such effects convergence should be very slow, typically 10 4 -10 6 iterations. Most adjacent neurons point to adjacent feature space regions, but kinks are possible since d-dimensional topography cannot be represented in 2D – try to do this with a map of a globe.

4 More SOM properties Complexity of the algorithm is O(KndN it ) for K nodes (processors), d dimensions, N it iterations n vectors in d-dimensions: all distances have to be computed and compared, therefore maps are not too large K    . Parallelization: easy to compute distances with K physical processors, but N it iterations is still large. Classification quality is poor. Kohonen: SOM is mainly for visualization but... visualization is also poor, there is no estimation of map distortion. SOM algorithm does not optimize any cost function measuring distortion of visualization!

5 Quantization error How to evaluate the errors that SOM maps make? Data in feature space region where node c is the winner is replaced by average W c of winner node. For m nodes in the SOM network Map of local quantization errors gives an idea where this approximation is acceptable. The total quantization error is a sum over all SOM nodes: local quantization error is a sum over all X vectors for which W c is the winner (the algorithm does the opposite, finds a winner to update): Hierarchical SOM maps: rough description of all data, separate maps for subsets of data, and more precise maps for even smaller subsets.

6 Growing Cell Structures (GCS) Constructive SOM (Fritzke 1993), many versions exist. Initial topology: for k-D maps (k=1, 2, 3) (k+1)-dim. simplex, i.e. connect k+1 nodes with each other: line, triangle, pyramid. Idea: add new nodes whenever quantization error is large. Calculate average quantization error for each node for all {X}; find the worst node, find its least similar (in the data space) neighbor add new node between these two nodes, interpolating their parameters. SOM algorithm is used for training, but adaptation is always only for the winner and its nearest neighbors (linked in physical space). Many other competitive learning algorithms like that have been invented and are suitable for visualization. See the Growing Self-Organizing Networks demo and the report “Some Competitive Learning Methods”.Growing Self-Organizing Networks

7 GCS algorithm 1. Create initial network topology, initialize node parameters W (i), for i = 1.. k+1, with small values. 2. Repeat steps 2-5 for t = 1.. L (number of presentation epochs): Randomly select input vector X, and find the winner node c. 3. Move the winner node towards the input  W c =  c (X  W c ). 4. Increase frequency counter  c =  c (t+1)   c (t)=1 of the winner node (alternatively, count quantization errors). 5. Correct parameters of its direct neighbors  W n =  n (X  W n ). 6. After L epochs calculate renormalized frequency for each node, f i =  i /  j  j 7. Find node with highest frequency W 1 (alternatively – find node with the largest quantization error); find direct neighbor W 2 with largest distance ||W 1 -W 2 ||; add new node W n =(W 1 + W 2 )/2 in between, link W 1, W 2, W n. 8. Stop if total number of epochs is too large, or quantization error is sufficiently small; otherwise repeat steps 2-7.

8 GCS growth

9 GCS - 2 clusters In 3-D feature space there are 2 separate data clusters. GCS network growing in 2D finds appropriate topology of the network. Growing grid tries to preserve rectangular structure, adds rows of nodes instead of a single one. Neural gas adds new unit between first and second winner after steps.

10 Voronoi and Delaunay Data Voronoi Delaunay points decision borders triangulation Voronoi diagram, tessellation – node in the center (codebook vector) is the winner, lines (planes) show decision borders, forming a Voronoi cell. Voronoi set – all vectors inside the Voronoi cell. Connecting nodes that have Voronoi cells with common edge Delaunay triangulation is obtained.

11 Ball in 3D SOM: 216 nodes in 1D, 18x12 nodes in 2D and 6x6x6 nodes in 3D. GCS: 216 nodes in 1D, 2D and 3D.

12 Some examples of real-life applications Helsinki University of Technology web site http://www.cis.hut.fi/research/refs/ has a list (2010) of > 7700 papers on SOM and its applications ! Brain research: modeling of formation of various topographical maps in motor, auditory, visual and somatotopic areas. AI and robotics: analysis of data from sensors, control of robot’s movement (motor maps), spatial orientation maps. Information retrieval and text categorization. Clusterization of genes, protein properties, chemical compounds, speech phonemes, sounds of birds and insects, astronomical objects, economical data, business and financial data.... Data compression (images and audio), information filtering. Medical and technical diagnostics.

13 More examples Natural language processing: linguistic analysis, parsing, learning languages, hyphenation patterns. Optimization: configuration of telephone connections, VLSI design, time series prediction, scheduling algorithms. Signal processing: adaptive filters, real-time signal analysis, radar, sonar seismic, USG, EKG, EEG and other medical signals... Image recognition and processing: segmentation, object recognition, texture recognition... Content-based retrieval: examples of WebSOM, Visier,WebSOMVisier PicSom – similarity based image retrieval.PicSom Check the links to maps for astronomical catalogs and journals! http://www.is.umk.pl/~duch/CI.html#SOM

14 Quality of life data WorldBank data 1992, 39 quality of life indicators. SOM map and the same colors on the world map. More examples of business applications from http://www.eudaptics.com/http://www.eudaptics.com/

15 SOM software A number of free programs for SOM were written. Best visualization is offered by Viscovery free viewer http://www.eudaptics.com It can be used with free SOM_pack software from http://www.cis.hut.fi/research/som_lvq_pak.shtml http://www.eudaptics.com http://www.cis.hut.fi/research/som_lvq_pak.shtml Growing Self-Organizing Networks demo (demoGNG) Growing Self-Organizing Networks demo and review paper on competitive learning. Book: T. Kohonen, Self-organizing Maps (3 rd ed, Springer 2001)

Download ppt "Computational Intelligence: Methods and Applications Lecture 10 SOM and Growing Cell Structures Włodzisław Duch Dept. of Informatics, UMK Google: W Duch."

Similar presentations

Unsupervised Learning Clustering K-Means. Recall: Key Components of Intelligent Agents Representation Language: Graph, Bayes Nets, Linear functions Inference.

Unsupervised Learning Clustering K-Means. Recall: Key Components of Intelligent Agents Representation Language: Graph, Bayes Nets, Linear functions Inference.
2806 Neural Computation Self-Organizing Maps Lecture 9 2005 Ari Visa.

2806 Neural Computation Self-Organizing Maps Lecture Ari Visa.
Efficient access to TIN Regular square grid TIN Efficient access to TIN Let q := (x, y) be a point. We want to estimate an elevation at a point q: 1. should.

Efficient access to TIN Regular square grid TIN Efficient access to TIN Let q := (x, y) be a point. We want to estimate an elevation at a point q: 1. should.
Clustering & image segmentation Goal::Identify groups of pixels that go together Segmentation.

Clustering & image segmentation Goal::Identify groups of pixels that go together Segmentation.
Self Organization of a Massive Document Collection

Self Organization of a Massive Document Collection
Un Supervised Learning & Self Organizing Maps. Un Supervised Competitive Learning In Hebbian networks, all neurons can fire at the same time Competitive.

Un Supervised Learning & Self Organizing Maps. Un Supervised Competitive Learning In Hebbian networks, all neurons can fire at the same time Competitive.
Neural Networks Chapter 9 Joost N. Kok Universiteit Leiden.

Neural Networks Chapter 9 Joost N. Kok Universiteit Leiden.
Unsupervised learning. Summary from last week We explained what local minima are, and described ways of escaping them. We investigated how the backpropagation.

Unsupervised learning. Summary from last week We explained what local minima are, and described ways of escaping them. We investigated how the backpropagation.
Self Organization: Competitive Learning

Self Organization: Competitive Learning
DATA-MINING Artificial Neural Networks Alexey Minin, Jass 2006.

DATA-MINING Artificial Neural Networks Alexey Minin, Jass 2006.
Kohonen Self Organising Maps Michael J. Watts

Kohonen Self Organising Maps Michael J. Watts
Self-Organizing Map (SOM). Unsupervised neural networks, equivalent to clustering. Two layers – input and output – The input layer represents the input.

Self-Organizing Map (SOM). Unsupervised neural networks, equivalent to clustering. Two layers – input and output – The input layer represents the input.
Machine Learning: Connectionist McCulloch-Pitts Neuron Perceptrons Multilayer Networks Support Vector Machines Feedback Networks Hopfield Networks.

Machine Learning: Connectionist McCulloch-Pitts Neuron Perceptrons Multilayer Networks Support Vector Machines Feedback Networks Hopfield Networks.
X0 xn w0 wn o Threshold units SOM.

X0 xn w0 wn o Threshold units SOM.
Self Organizing Maps. This presentation is based on: SOM’s are invented by Teuvo Kohonen. They represent multidimensional.

Self Organizing Maps. This presentation is based on: SOM’s are invented by Teuvo Kohonen. They represent multidimensional.
Self-Organizing Hierarchical Neural Network

Self-Organizing Hierarchical Neural Network
KNN, LVQ, SOM. Instance Based Learning K-Nearest Neighbor Algorithm (LVQ) Learning Vector Quantization (SOM) Self Organizing Maps.

KNN, LVQ, SOM. Instance Based Learning K-Nearest Neighbor Algorithm (LVQ) Learning Vector Quantization (SOM) Self Organizing Maps.
November 24, 2009Introduction to Cognitive Science Lecture 21: Self-Organizing Maps 1 Self-Organizing Maps (Kohonen Maps) In the BPN, we used supervised.

November 24, 2009Introduction to Cognitive Science Lecture 21: Self-Organizing Maps 1 Self-Organizing Maps (Kohonen Maps) In the BPN, we used supervised.
Neural Networks Lecture 17: Self-Organizing Maps

Neural Networks Lecture 17: Self-Organizing Maps
A Hybrid Self-Organizing Neural Gas Network James Graham and Janusz Starzyk School of EECS, Ohio University Stocker Center, Athens, OH 45701 USA IEEE World.

A Hybrid Self-Organizing Neural Gas Network James Graham and Janusz Starzyk School of EECS, Ohio University Stocker Center, Athens, OH USA IEEE World.

Similar presentations

Ads by Google