EE3J2 Data Mining Lecture 18 K-means and Agglomerative Algorithms Ali Al-Shahib

Today Unsupervised Learning Clustering K-means

Distortion The distortion for the centroid set C = c1,…,cM is defined by: In other words, the distortion is the sum of distances between each data point and its nearest centroid The task of clustering is to find a centroid set C such that the distortion Dist(C) is minimised EE3J2 Data Mining

The K-Means Clustering Method Given k, the k-means algorithm is implemented in 4 steps: Initialisation Define the number of clusters (k). Designate a cluster centre (a vector quantity that is of the same dimensionality of the data) for each cluster. Assign each data point to the closest cluster centre (centroid). That data point is now a member of that cluster. Calculate the new cluster centre (the geometric average of all the members of a certain cluster). Calculate the sum of within-cluster sum-of-squares. If this value has not significantly changed over a certain number of iterations, exit the algorithm. If it has, or the change is insignificant but has not been seen to persist over a certain number of iterations, go back to Step 2. Remember you converge when you have found the minimum overall distance between the centroid and the objects.

The K-Means Clustering Method Example

Lets watch an animation! http://r.yihui.name/stat/multivariate_stat/kmeans/index.htm

K-means Clustering Suppose that we have decided how many centroids we need - denote this number by K Suppose that we have an initial estimate of suitable positions for our K centroids K-means clustering is an iterative procedure for moving these centroids to reduce distortion

K-means clustering - notation Suppose there are T data points, denoted by: Suppose that the initial K clusters are denoted by: One iteration of K-means clustering will produce a new set of clusters Such that

K-means clustering (1) For each data point yt let ci(t) be the closest centroid In other words: d(yt, ci(t)) = minmd(yt,cm) Now, for each centroid c0k define: In other words, Y0k is the set of data points which are closer to c0k than any other cluster

K-means clustering (2) Now define a new kth centroid c1k by: where |Yk0| is the number of samples in Yk0 In other words, c1k is the average value of the samples which were closest to c0k

K-means clustering (3) Now repeat the same process starting with the new centroids: to create a new set of centroids: … and so on until the process converges Each new set of centroids has smaller distortion than the previous set

So….Basically Start with randomly k data points (objects). Find the set of data points that are closer to C0k (Y0k). Compute average of these points, notate C1k -> new centroid. Now repeat again this process and find the closest objects to C1k Compute the average to get C2k -> new centroid, and so on…. Until convergence.

Comments on the K-Means Method Strength Relatively efficient: O(tkn), where n is # objects, k is # clusters, and t is # iterations. Normally, k, t << n. Often terminates at a local optimum. The global optimum may be found using techniques such as: deterministic annealing and genetic algorithms Weakness Applicable only when mean is defined, then what about categorical data? Need to specify k, the number of clusters, in advance Unable to handle noisy data and outliers Not suitable to discover clusters with non-convex shapes

Hierarchical Clustering Grouping data objects into a tree of clusters. Agglomerative clustering Begin by assuming that every data point is a separate centroid Combine closest centroids until the desired number of clusters is reached Divisive clustering Begin by assuming that there is just one centroid/cluster Split clusters until the desired number of clusters is reached

Agglomerative Clustering - Example Students Exam1 Exam2 Exam3 Mike 9 3 7 Tom 10 2 Bill 1 4 T Ren 6 5 Ali

Distances between objects Using Euclidean Distance measure, what's the difference between Mike and Tom? Mike:9,3,7 Tom: 10,2,9 S E1 E2 E3 Mike 9 3 7 Tom 10 2 Bill 1 4 T 6 5 Ali

Distance Matrix Mike Tom Bill T Ren Ali 2.5 10.44 4.12 11.75 - 12.5   Mike Tom Bill T Ren Ali 2.5 10.44 4.12 11.75 - 12.5 6.4 13.93 6.48 1.41 7.35

The Algorithm Step 1 Identify the entities which are most similar - this can be easily discerned from the distance table constructed. In this example, Bill and Ali are most similar, with a distance value of 1.41. They are therefore the most 'related' Bill Ali

The Algorithm – Step 2 The two entities that are most similar can now be merged so that they represent a single cluster (or new entity). So Bill and Ali can now be considered to be a single entity. How do we compare this entity with others? We use the Average linkage between the two. So the new average vector is [1, 9.5, 3.5] – see first table and average the marks for Bill and Ali. We now need to redraw the distance table, including the merger of the two entities, and new distance calculations.

The Algorithm – Step 3 Mike Tom T Ren {Bill & Ali} - 2.5 4.12 10.9 6.4   Mike Tom T Ren {Bill & Ali} - 2.5 4.12 10.9 6.4 9.1 6.9

Next closest students Mike and Tom with 2.5! So, now we have 2 clusters! Bill Ali Mike Tom

The distance matrix now   {Mike & Tom} T Ren {Bill & Ali} - 3.7 9.2 6.9 Now, T Ren is closest to Bill and Ali so T Ren joines them In the cluster.

The final dendogram Bill Ali MANY ‘SUB-CLUSTERS’ WITHIN ONE CLUSTER Mike Tom T Ren

Conclusions K- Means Algorithm – memorize equations and algorithm. Hierarchical Clustering: Agglomerative Clustering

On Tuesday Sequence Analysis: BLAST Algorithm

Some References and Acknowledgments University College London: http://www.ucl.ac.uk/oncology/MicroCore/HTML_resource/Hier_Example.htm