Iterative K-Means Algorithm Based on Fisher Discriminant UNIVERSITY OF JOENSUU DEPARTMENT OF COMPUTER SCIENCE JOENSUU, FINLAND Mantao Xu to be presented at: Infomation Fusion 2004

Problem Formulation Given N data samples X={x 1, x 2, …, x N }, construct the codebook C = {c 1, c 2, …, c M } such that mean-square-error is minimized. The class membership p (i) is

Traditional K-Means Algorithm Iterations of two steps: assignment of each data vector with a class label computation of cluster centroid by averaging all data vectors that are assigned to it Characteristics: Randomized initial partition or codebook Convergence to a local minimum Use of L 2, L 1 and L  distance Fast and easy implementation Extensions: Kernel Kmeans algorithm EM algorithm K-median algorithm

Motivation Investigation on a clustering algorithm that : iteratively performs the regular K-Means algorithm in searching a solution close to the global optima estimates the initial partition close to the optimal solution by at each iteration applies a dissimilarity function based on the current partition instead of L 2 distance

Selecting K-Means initial partiton Selection of initial partion based on Fisher discriminant and dynamic programming: A suboptimal partition is estimated by dynamic programming in some one- dimenaional subspace of feature space. The one-dimensional subspace is constructed through linear multi-class Fisher discriminant analysis The output class of K-Means in each iteration is selected as the input class of discriminant analysis for next iteration

The multi-class Fisher discriminant analysis The separation of input classes in the discriminant direction w can be measured by F-ratio validity index, F(w) The multi-class linear Fisher discriminant w is the minimization of F-ratio validity index

The dynamic programming in discriminant direction The optimal convex partition Q k ={(q j-1,q j ]| j=1, ,n} in the discriminant direction w can be estimated by dynamic promgramming in terms of MSE distortion on the discriminant subspace or in terms of MSE distortion on original feature space (2) (1)

Application of Delta-MSE Dissimilarity x 1 x 2 x 3 y 1 y 2 y 3 x 4 G 1 G 2 Delta-MSE(x 4,G,G 1 )=RemovalVariance Delta-MSE(x 4,G,G 2 )=AddVariance Move vector x from cluster i to cluster j, the change of the MSE function [10] caused by this move is:

Pseudocodes of Iterative K-Means algorithm

Four K-Means algorithms conducted in experimental tests K-D tree based K-Means: selects its initial cluster centroids from the k-bucket centers of a kd-tree structure that is recursively built by principal component analysis PCA based K-Means: an intuitive approach to estimate a sub-optimal initial partition by applying the dynamic programming in the principal component direction LFD-I: the proposed iterative K-Means algorithm based on the dynamic programming criterion (1) LFD-II: the proposed iterative K-Means algorithm based on the dynamic programming criterion (2)

Comparisons of the four K-Means algorithms Table 1: Performance comparisons (in F-ratio validity indices ) of the four K-Means algorithms on the practical numbers of clusters

F-ratios produced by the four K- Means clusterings on dataset glass

F-ratios produced by the four K- Means clusterings on dataset heart

F-ratios produced by the four K- Means clusterings on dataset image

Conclusions A new approach to the k-center clustering problem by iteratively incorporating the Fisher discriminant analysis and the dynamic programming technique The proposed approach in general outperforms the two other algorithms: the PCA based K-Means algorithm and the kd-tree based K-Means algorithm The classification performance gains of the proposed approach over the two others is increased with the number of clusters

Further Work Sovling the k-center clustering problem by iteratively incorporating the kernel Fisher discriminant analysis and the dynamic programming technique Sovling the k-center clustering problem by incorporating the kernel PCA technique and the dynamic programming technique