Assessment

Schedule graph may be of help for selecting the best solution Best solution corresponds to a plateau before a high jump Solutions with very small or even singletons clusters are rather suspicious

Standardization K-means Initial centroids Validation example Cluster merit Index Cluster validation, three approaches Relative criteria Validity Index Dunn Index Davies-Bouldin (DB) index Combination of different distances/diameter methods

Standardization Issue The need for any standardization must be questioned If the interesting clusters are based on the original features, than any standardization method may distort those clusters Only when there are grounds to search for clusters in transformed space that some standardization rule should new used

There is no methodological way except by “trail and error”

An easy standardization method that will often follow and frequently achieve good results is the simple division or multiplication by a simple scale factor A should be properly chosen so that all feature values occupy a suitable interval

k-means Clustering Cluster centers c 1,c 2,.,c k with clusters C 1,C 2,.,C k

Initial centroids Specify which patterns are used as initial centroids Random initialization Tree clustering in a reduced number of patterns may performed for this purpose Choose first k patterns as initial centroids Sort distances between all patterns and choose patterns at constant intervals of these distances as initial centroids Adaptive initialization (according to a chosen radius)

k-means example (Sá 2001) Cluster merit index R i (n patterns in k cluserts)

Cluster merit index measure the decrease in overall within-cluster distance when passing from a solution with k clusters to one with k+1 clusters High value of the merit indexes indicates a substantial decrease in overall within-cluster distance

Cluster merit index Factor 1 has the most important contribution The values k=3,5,8 are sensible choices k=3 attractive

Cluster validation The procedure of evaluating the results of a clustering algorithm is known under the term cluster validity In general terms, there are three approaches to investigate cluster validity

The first is based on external criteria This implies that we evaluate the results of a clustering algorithm based on a pre- specified structure, which is imposed on a data set and reflects our intuition about the clustering structure of the data set

Error Classification Rate smaller value, good representation Data partition according to known classes L i,

The second approach is based on internal criteria We may evaluate the results of a clustering algorithm in terms of quantities that involve the vectors of the data set themselves (e.g. proximity matrix)

Proximity matrix Dissimilarity matrix

The basis of the above described validation methods is often statistical testing Major drawback of techniques based on internal or external criteria and statistical testing is their high computational demands

The third approach of clustering validity is based on relative criteria Here the basic idea is the evaluation of a clustering structure by comparing it to other clustering schemes, resulting by the same algorithm but with different parameter values

There are two criteria proposed for clustering evaluation and selection of an optimal clustering scheme (Berry and Linoff, 1996) Compactness, the members of each cluster should be as close to each other as possible. A common measure of compactness is the variance, which should be minimized Separation, the clusters themselves should be widely spaced

Distance between two clusters There are three common approaches measuring the distance between two different clusters Single linkage: It measures the distance between the closest members of the clusters Complete linkage: It measures the distance between the most distant members Comparison of centroids: It measures the distance between the centers of the clusters

Relative criteria Based on relative criteria, does not involve statistical tests The fundamental idea of this approach is to choose the best clustering scheme of a set of defined schemes according to a pre-specified criterion

Among the clustering schemes C i,i=1,..., k defined by a specific algorithm, for different values of the parameters choose the one that best fits the data set The procedure of identifying the best clustering scheme is based on a validity index q

Selecting a suitable performance index q,we proceed with the following steps We run the clustering algorithm for all values of k between a minimum k min and a maximum k max The minimum and maximum values have been defined a- priori by user For each of the values of k, we run the algorithm r times, using different set of values for the other parameters of the algorithm (e.g. different initial conditions) We plot the best values of the index q obtained by each k as the function of k

Based on this plot we may identify the best clustering scheme There are two approaches for defining the best clustering depending on the behavior of q with respect to k If the validity index does not exhibit an increasing or decreasing trend as k increases we seek the maximum (minimum) of the plot

For indices that increase (or decrease) as the number of clusters increase we search for the values of k at which a significant local change in value of the index occurs This change appears as a “knee” (joelho) in the plot and it is an indication of the number of clusters underlying the data-set The absence of a knee may be an indication that the data set possesses no clustering structure

Validity index Dunn index, a cluster validity index for k- means clustering proposed in Dunn (1974) Attempts to identify “compact and well separated clusters”

Dunn index

If the dataset contains compact and well- separated clusters, the distance between the clusters is expected to be large and the diameter of the clusters is expected to be small Large values of the index indicate the presence of compact and well-separated clusters

The index D k does not exhibit any trend with respect to number of clusters Thus, the maximum in the plot of D k versus the number of clusters k can be an indication of the number of clusters that fits the data

The implications of the Dunn index are: Considerable amount of time required for its computation Sensitive to the presence of noise in datasets, since these are likely to increase the values of diam(c)

The Davies-Bouldin (DB) index (1979)

Small indexes correspond to good clusters, clusters are compact and their centers are far away The DB k index exhibits no trends with respect to the number of clusters and thus we seek the minimum value of DB k its plot versus the number of clusters

Different methods may be used to calculate distance between clusters Single linkage Complete linkage Comparison of centroids Average linkage

Differnet methods to calculate the diamater of a cluster Max Radius Average distance

A connected graph with s nodes has edges

Combination of different distances/diameter methods It has been shown that using different distances/diameter methods may produce indices of different scale range (Azuje and Bolshakova 2002)

Normalization i selects the different distance method i  (1,2,3,4) j selects the different diameter method j  (1,2,3)  (D ij ) or  (DB ij ) standart deviation of D k ij or DB k ij accross diferent values for k

Normalized indexes

Literature

See also J.P Marques de Sá, Pattern Recognition, Springer, TCD-CS pdf TCD-CS pdf

Standardization K-means Initial centroids Validation example Cluster merit Index Cluster validation, three approaches Relative criteria Validity Index Dunn Index Davies-Bouldin (DB) index Combination of different distances/diameter methods

Next lecture KNN LVQ SOM