1. Clustering Categorical Data
Pasi Fränti

2. K-means clustering

3. Definitions and data Set of N data points: Partition of the data:
X={x1, x2, …, xN}
Partition of the data:
P={p1, p2, …, pM},
Set of M cluster prototypes (centroids):
C={c1, c2, …, cM},

4. Distance and cost function
Euclidean distance of data vectors:
Mean square error:

5. Clustering result as partition
Partition of data
Cluster prototypes
Illustrated by Voronoi diagram
Illustrated by Convex hulls

6. Duality of partition and centroids
Partition of data
Cluster prototypes
Partition by nearest prototype mapping
Centroids as prototypes

7. Categorical data

8. Categorical clustering
Three attributes
director
actor
genre
t1 (Godfather II)
Coppola
De Niro
Crime
t2 (Good Fellas)
Scorsese
t3 (Vertigo)
Hitchcock
Stewart
Thriller
t4 (N by NW)
Grant
t5 (Bishop's Wife)
Koster
Comedy
t6 (Harvey)

9. Categorical clustering Sample 2-d data: color and shape
Model A
Model B
Model C

10. Hamming Distance (Binary and categorical data)
Number of different attribute values.
Distance of ( ) and ( ) is 2.
Distance ( ) and ( )
Distance between (toned) and (roses) is 3.
100->011 has distance 3 (red path) 010->111 has distance 2 (blue path)
3-bit binary cube

11. Histogram-based methods:
K-means variants
Methods:
Histogram-based methods:
k-modes
k-medoids
k-distributions
k-histograms
k-populations
k-representatives

12. Entropy-based cost functions
Category utility:
Entropy of data set:
Entropies of the clusters relative to the data:

13. Iterative algorithms

14. K-modes clustering Distance function

15. K-modes clustering Prototype of cluster

16. K-medoids clustering Prototype of cluster
Vector with minimal total distance to every other 3
Medoid: 2 2 A C E B C F B D G B C F
2+3=5
2+2=4
2+3=5

17. K-medoids Example

18. K-medoids Calculation

19. K-histograms
D 2/3
F 1/3

20. K-distributions Cost function with ε addition

21. Example of cluster allocation Change of entropy

22. Problem of non-convergence Non-convergence

23. Results with Census dataset

24. Literature
Modified k-modes + k-histograms: M. Ng, M.J. Li, J. Z. Huang and Z. He, On the Impact of Dissimilarity Measure in k-Modes Clustering Algorithm, IEEE Trans. on Pattern Analysis and Machine Intelligence, 29 (3), , March, 2007.
ACE: K. Chen and L. Liu, The “Best k'' for entropy-based categorical dataclustering, Int. Conf. on Scientific and Statistical Database Management (SSDBM'2005), pp , Berkeley, USA, 2005.
ROCK: S. Guha, R. Rastogi and K. Shim, “Rock: A robust clustering algorithm for categorical attributes”, Information Systems, Vol. 25, No. 5, pp , 200x.
K-medoids: L. Kaufman and P. J. Rousseeuw, Finding groups in data: an introduction to cluster analysis, John Wiley Sons, New York, 1990.
K-modes: Z. Huang, Extensions to k-means algorithm for clustering large data sets with categorical values, Data mining knowledge discovery, Vol. 2, No. 3, pp , 1998.
K-distributions: Z. Cai, D. Wang and L. Jiang, K-Distributions: A New Algorithm for Clustering Categorical Data, Int. Conf. on Intelligent Computing (ICIC 2007), pp , Qingdao, China, 2007.
K-histograms: Zengyou He, Xiaofei Xu, Shengchun Deng and Bin Dong, K-Histograms: An Efficient Clustering Algorithm for Categorical Dataset, CoRR, abs/cs/ ,