Presentation is loading. Please wait.

Presentation is loading. Please wait.

Lecture 09 Clustering-based Learning

Published by Modified over 8 years ago

Similar presentations

Presentation on theme: "Lecture 09 Clustering-based Learning"— Presentation transcript:

1 Lecture 09 Clustering-based Learning
Topics Basics K-Means Self-Organizing Maps Applications Discussions

2 Basics Clustering Grouping a collection of objects (examples) into clusters, such that objects are most similar inside each cluster and least similar between clusters. Core problem: similarity definition Intra cluster similarity Inter cluster similarity Inductive learning Unsupervised learning

3 Basics Minimizing intra cluster dissimilarity is equivalent to maximizing inter cluster dissimilarity Clustering performance in terms of Intra cluster dissimilarity: K for K clusters and d(xi,xi’) for dissimilarity measure

4 Basics Dissimilarity measure depends on value types and value coding systems Some examples Quantitative variables: Ordinal variables: Categorical variables:

5 Basics Clustering algorithms Combinatorial Algorithms
Work directly on the observed data K-Means Self-Organizing Maps

6 K-Means A statistical learning mechanism
A given object is assigned to a cluster if it has least dissimilarity to the mean value of the cluster. Euclidean or Manhattan distance is commonly used to measure dissimilarity The mean value of each cluster is recalculated in each iteration

7 K-Means Step 1: Selecting Centers
Selects k objects randomly, each becoming the center (mean) of an initial cluster. Step 2: Clustering Assign each of the remaining objects to the cluster with the nearest distance. The most popular method for calculating distance is Euclidean distance. Given two points p = ( p1, p2, …, pk ) and q = ( q1, q2, …, qk ), their Euclidean distance is defined as:

8 K-Means Step 3: Computing New Centers
Compute new cluster centers. Let xi be one of the elements assigned to the kth cluster, and Nk be the number of elements in the cluster. The new center of cluster k, Ck, is calculated as: Step 4: Iteration Repeat steps 2 and 3 until no members change their clusters.

9 K-Means Example Assign each object to most similar center
1 2 3 4 5 6 7 8 9 10 10 9 8 7 6 5 Assign each object to most similar center Update the cluster means 4 3 2 1 1 2 3 4 5 6 7 8 9 10 reassign reassign K=2 Arbitrarily choose K object as initial cluster center Update the cluster means

10 K-Means Usually, the problem itself has the setting of K
If K is not given, in order to find the best K, we examine the intra cluster dissimilarity Wk, which is a function of K Usually Wk decreases with increasing K

11 K-Means Decide K A sharp drop of Wk is observed

12 K-Means Hierarchical Clustering

13 K-Means Agglomerative Hierarchical Clustering

14 K-Means Divisive Hierarchical Clustering

15 Self-Organizing Maps Brain self-organizing structure
Our brain is dominated by the cerebral cortex, a very complex structure of billions of neurons and hundreds of billions of synapses. The cortex includes areas that are responsible for different human activities (motor, visual, auditory, etc.), and associated with different sensory inputs. We can say that each sensory input is mapped into a corresponding area of the cerebral cortex. The cortex is a self-organising computational map in the human brain.

16 Self-Organizing Maps The self-organising map (SOM) provides a topological mapping emulating the cortex structure. It places a fixed number of input patterns from the input layer into a higher-dimensional output or Kohonen layer. SOM is a subsymbolic learning algorithm; data input need to be numerically coded.

17 Self-Organizing Maps

18 Self-Organizing Maps Training of SOM is based on competitive learning: Neurons compete among themselves to be activated, but only a single output neuron can be active at any time. The output neuron that wins the “competition” is called the winner-takes-all neuron Training in SOM begins with the winner’s neighborhood of a fairly large size. Then, as training proceeds, the neighborhood size gradually decreases.

19 Self-Organizing Maps Conceptual architecture

20 Self-Organizing Maps The lateral connections are used to create a competition between neurons. The neuron with the largest activation level among all neurons in the output layer becomes the winner. This neuron is the only neuron that produces an output signal. The activity of all other neurons is suppressed in the competition. The lateral feedback connections produce excitatory or inhibitory effects, depending on the distance from the winning neuron. This can be achieved by the use of a Mexican hat function which describes synaptic weights between neurons in the Kohonen layer.

21 Self-Organizing Maps Mexican hat function of lateral connection

22 SOM – Competitive Learning Algorithm
Step 1: Initialization Set initial weights to small random values, say in an interval [0, 1], and assign a small positive value, e.g., 0.2 to 0.5, to the learning rate parameter 0.

23 SOM – Competitive Learning Algorithm
Step 2: Activation and Similarity Matching Activate the SOM by applying the input vector X, and find the best matching neuron JX at iteration p, using the minimum Euclidean distance criterion where n is the number of neurons in the input layer, m is the number of neurons in the Kohonen layer, and j = 1, 2, …m.

24 SOM – Competitive Learning Algorithm
Step 3: Learning (a) Calculate weight corrections according to the competitive learning rule: where ΛJ: neighborhood of neuron J, d0: initial neighborhood size and T: total repetitions.

25 SOM – Competitive Learning Algorithm
Step 3: Learning (Continued) (b) Update the weights where wij(p) is the weight correction at iteration p. Step 4: Iteration Increase iteration p by one, go back to Step 2 and continue until the minimum-distance Euclidean criterion is satisfied, or no noticeable changes occur in the feature map.

26 Self-Organizing Maps SOM is online K-Means New Object

27 Self-Organizing Maps Example: A SOM with 100 neurons arranged in the form of a two-dimensional lattice with 10 rows and 10 columns. It is required to classify two-dimensional input vectors  each neuron in the network should respond only to the input vectors occurring in its region. The network is trained with 1000 two-dimensional input vectors generated randomly in a square region in the interval between –1 and +1. The learning rate parameter  is fixed, equal to 0.1.

28 Self-Organizing Maps Initial random weights

29 Self-Organizing Maps 100 repetitions

30 Self-Organizing Maps 1,000 repetitions

31 Self-Organizing Maps 10,000 repetitions

32 Applications K-Means Self-Organizing Maps
Cluster ECG signals according to Correlation Dimensions Self-Organizing Maps Find churner groups Speech recognition

33 Discussions Clustering events with attribute-based representation
Attribute-based similarity measure for clusters Hierarchical clustering of event sequences Generalization, e.g., “A ∧ B ∧C” generalized to “A ∧ B” “A ∨ B” generalized to “A ∨ B ∨ C” ontology Specialization

Download ppt "Lecture 09 Clustering-based Learning"

Similar presentations

© Negnevitsky, Pearson Education, 2005 1 Introduction Introduction Hebbian learning Hebbian learning Generalised Hebbian learning algorithm Generalised.

© Negnevitsky, Pearson Education, Introduction Introduction Hebbian learning Hebbian learning Generalised Hebbian learning algorithm Generalised.
2806 Neural Computation Self-Organizing Maps Lecture 9 2005 Ari Visa.

2806 Neural Computation Self-Organizing Maps Lecture Ari Visa.
Neural Networks Dr. Peter Phillips. The Human Brain (Recap of week 1)

Neural Networks Dr. Peter Phillips. The Human Brain (Recap of week 1)
Un Supervised Learning & Self Organizing Maps. Un Supervised Competitive Learning In Hebbian networks, all neurons can fire at the same time Competitive.

Un Supervised Learning & Self Organizing Maps. Un Supervised Competitive Learning In Hebbian networks, all neurons can fire at the same time Competitive.
Unsupervised learning. Summary from last week We explained what local minima are, and described ways of escaping them. We investigated how the backpropagation.

Unsupervised learning. Summary from last week We explained what local minima are, and described ways of escaping them. We investigated how the backpropagation.
Self Organization: Competitive Learning

Self Organization: Competitive Learning
5/16/2015Intelligent Systems and Soft Computing1 Introduction Introduction Hebbian learning Hebbian learning Generalised Hebbian learning algorithm Generalised.

5/16/2015Intelligent Systems and Soft Computing1 Introduction Introduction Hebbian learning Hebbian learning Generalised Hebbian learning algorithm Generalised.
Kohonen Self Organising Maps Michael J. Watts

Kohonen Self Organising Maps Michael J. Watts
Artificial neural networks:

Artificial neural networks:
Adaptive Resonance Theory (ART) networks perform completely unsupervised learning. Their competitive learning algorithm is similar to the first (unsupervised)

Adaptive Resonance Theory (ART) networks perform completely unsupervised learning. Their competitive learning algorithm is similar to the first (unsupervised)
Competitive learning College voor cursus Connectionistische modellen M Meeter.

Competitive learning College voor cursus Connectionistische modellen M Meeter.
Unsupervised Networks Closely related to clustering Do not require target outputs for each input vector in the training data Inputs are connected to a.

Unsupervised Networks Closely related to clustering Do not require target outputs for each input vector in the training data Inputs are connected to a.
Self-Organizing Map (SOM). Unsupervised neural networks, equivalent to clustering. Two layers – input and output – The input layer represents the input.

Self-Organizing Map (SOM). Unsupervised neural networks, equivalent to clustering. Two layers – input and output – The input layer represents the input.
X0 xn w0 wn o Threshold units SOM.

X0 xn w0 wn o Threshold units SOM.
Self Organizing Maps. This presentation is based on: SOM’s are invented by Teuvo Kohonen. They represent multidimensional.

Self Organizing Maps. This presentation is based on: SOM’s are invented by Teuvo Kohonen. They represent multidimensional.
Slides are based on Negnevitsky, Pearson Education, 2005 1 Lecture 8 Artificial neural networks: Unsupervised learning n Introduction n Hebbian learning.

Slides are based on Negnevitsky, Pearson Education, Lecture 8 Artificial neural networks: Unsupervised learning n Introduction n Hebbian learning.
November 9, 2010Neural Networks Lecture 16: Counterpropagation 1 Unsupervised Learning So far, we have only looked at supervised learning, in which an.

November 9, 2010Neural Networks Lecture 16: Counterpropagation 1 Unsupervised Learning So far, we have only looked at supervised learning, in which an.
KNN, LVQ, SOM. Instance Based Learning K-Nearest Neighbor Algorithm (LVQ) Learning Vector Quantization (SOM) Self Organizing Maps.

KNN, LVQ, SOM. Instance Based Learning K-Nearest Neighbor Algorithm (LVQ) Learning Vector Quantization (SOM) Self Organizing Maps.
1 Kunstmatige Intelligentie / RuG KI2 - 7 Clustering Algorithms Johan Everts.

1 Kunstmatige Intelligentie / RuG KI2 - 7 Clustering Algorithms Johan Everts.
November 24, 2009Introduction to Cognitive Science Lecture 21: Self-Organizing Maps 1 Self-Organizing Maps (Kohonen Maps) In the BPN, we used supervised.

November 24, 2009Introduction to Cognitive Science Lecture 21: Self-Organizing Maps 1 Self-Organizing Maps (Kohonen Maps) In the BPN, we used supervised.

Similar presentations

Ads by Google