Clustering (2) Center-based algorithms Fuzzy k-means Density-based algorithms ( DBSCAN as an example ) Evaluation of clustering results Figures and equations from Data Clustering by Gan et al.

Center-based clustering  Have objective functions which define how good a solution is;  The goal is to minimize the objective function;  Efficient for large/high dimensional datasets;  The clusters are assumed to be convex shaped; The cluster center is representative of the cluster;  Some model-based clustering, e.g. Gaussian mixtures, are center-based clustering.

Center-based clustering K-means clustering. Let C 1, C 2,…,C k be k disjoint clusters. Error is defined as the sum of the distance from the cluster center

The k-means algorithm: Center-based clustering

Understanding k-means as an optimization procedure: The objective function is: Minimize the P(W,Q) subject to:

The solution is iteratively solving two sub-problems: Center-based clustering

 In terms of optimization, the k-means procedure is greedy.  Every iteration decreases the value of the objective function; The algorithm converges to a local minimum after a finite number of iterations.  Results depend on initiation values.  The computational complexity is proportional to the size of the dataset  efficient on large data.  The clusters identified are mostly ball-shaped.  Works only on numerical data.

Center-based clustering A variant of k-means to save computing time: the compare-means algorithm. (There are many.) Based on triangle inequality, d(x, m i )+d(x, m j )≥d(m i, m j ) d(x, m j )≥d(m i, m j )-d(x, m i ) If d(m i, m j )≥2d(x, m i ), then d(x, m j )≥d(x, m i ) In every iteration, the small number of between-mean distances are first computed. Then for every x, first compare its distance to the closest known mean with the between-mean distances, to find which of the d(x, m j ) really need to be compute.

Center-based clustering Automated selection of k? The x-means algorithm based on AIC/BIC. A family of models at different k: is the likelihood of the data given the j th model. p j is the number of parameters. We have to assume a model to get the likelihood. The convenient one is Gaussian.

Center-based clustering Under the assumption of identical spherical Gaussian assumption, (n is sample size; k is number of centroids) μ (i) is the centroid associated with x i. The likelihood is: The number of parameters is (d is dimension): (class probabilities + parameters for mean & variance)

Center-based clustering K-harmonic means --- insensitive to initiation. K-means error: K-harmonic means error:

Center-based clustering K-modes algorithm for categorical data. Let x be a d-vector with categorical attributes. For a group of x’s, the mode is defined as the vector q that minimizes Where The objective function is similar to the one for the original k-means.

Center-based clustering K-prototypes algorithm for mixed type data. Between any two points, the distance is defined: γ is a parameter to balance between continuous and categorical variables. Cost function to minimize:

Fuzzy k-means Soft clustering --- an observation can be assigned to multiple clusters. With n samples and c partitions, the fuzzy c-partition matrix (c × n): If take max for every sample we get back to hard partition:

Fuzzy k-means The objective function is: q>1, it controls the “fuzziness”. V i is the centroid of cluster i, u ij is the degree of membership of x j belonging to cluster i, k is number of clusters.

Fuzzy k-means

Density-based algorithms (DBSCAN as an example) Capable of finding arbitrarily shaped clusters. Clusters are defined as dense regions surrounded by low-density regions. Automatically select the number of clusters. Needs only one scan through the original data set.

Density-based algorithms (DBSCAN as an example) Define the ε-neighborhood of a point x as: A point x is “directly density-reachable” from point y if: This relationship is not symmetric.

Density-based algorithms (DBSCAN as an example) Point x is “density-reachable” from point y if there’s a sequence of points x, x 1, x 2, ……, x i, y where each point is directly density reachable from the next one. Points x and y are “density connected” if there exists a point z, such that both x and y are density- reachable from z. (* all the relationships are with respect to ε and N min )

Density-based algorithms (DBSCAN as an example) The definition of “cluster”: Let D be the dataset. A cluster C with respect to ε and N min satisfies: (1)If x is in C and y is density-reachable from x, then y is in C; (2)Every pair of x and y in C are density-connected. “Noise” is a set of points that don’t belong to any cluster.

Density-based algorithms (DBSCAN as an example) Core point Border point noise

Density-based algorithms (DBSCAN as an example) Algorithm: Start with an arbitrary point x, find all points that are density-reachable from x. (If x is a core point, then a cluster is found; if x is a border or noise point, no point is density reachable from it.) Visit the next unclassified point. Two clusters may be merged if close enough. Cluster distance is defined by single linkage:

Density-based algorithms (DBSCAN as an example)

How to choose ε and N min ? A heuristic method called the “sorted k-dist graph”. Sort the F k (D) (from the entire dataset) in descending order and plot. Find the k* where k>k* doesn’t bring much change to the graph. N min =k* Find the first point in the first valley z 0. Set ε=F k* (z 0 ). Density-based algorithms (DBSCAN as an example)

Evaluation of clustering results

Evaluation External criteria approach: Comparing clustering results (C ) with a pre-specified partition (P). For all pairs of samples, M=a+b+c+d In same cluster in P In different cluster in P In same cluster in C ab In different cluster in C cd

Evaluation Monte Carlo methods based on H 0 (random generation), or bootstrap are needed to find significance.

Evaluation External criteria: An alternative is to compare the proximity matrix Q with the given partition P. Define matrix Y based on P:

Evaluation Internal criteria: evaluate clustering structure by features of the dataset (mostly proximity matrix of the data). Example: For Hierarchical clustering, P c : cophenetic matrix, the ij th element represents proximity level at which two data points x i and x j are first joined into the same cluster. P: proximity matrix.

Evaluation Cophenetic correlation coefficient index: CPCC is in [-1,1]. Higher value indicates better agreement.

Evaluation Relative criteria: choose the best result out of a set according to predefined criterion. Example: Modified Hubert’s Γ statistic: P is the proximity matrix of the data. High value indicates compact clusters.