Optimal Dimensionality of Metric Space for kNN Classification Wei Zhang, Xiangyang Xue, Zichen Sun Yuefei Guo, and Hong Lu Dept. of Computer Science & Engineering FUDAN University, Shanghai, China

2 Outline Motivation  Related Work  Main Idea Proposed Algorithm  Discriminant Neighborhood Embedding  Dimensionality Selection Criterion Experimental Results  Toy Datasets  Real-world Datasets Conclusions

3 Related Work Many recent techniques have been proposed to learn a more appropriate metric space for better performance of many learning and data mining algorithms, for examples,  Relevant Component Analysis, Bar-Hillel, A., et al. ICML2003.  Locality Preserving Projections, He, X. et al., NIPS  Neighborhood Components Analysis, Goldberger, J., et al. NIPS  Marginal Fisher Analysis, Yan, S., et al., CVPR  Local Discriminant Embedding, Chen, H.-T., et al. CVPR  Local Fisher Discriminant Analysis, Sugiyama, M. ICML 2006  …… However, the target dimensionality of the new space is selected empirically in the above mentioned approaches

4 Main Idea Given finite labeled multi-class samples, what can we do for better performance of kNN classification?  Can we learn a low dimensional embedding for that kNN points in the same class have smaller distances to each other than to points in different classes?  Can we estimate the optimal dimensionality of the new metric space in the meantime ? Original Space (D=2) New Space (d=1)

5 Outline Motivation  Related Work  Main Idea Proposed Algorithm  Discriminant Neighborhood Embedding  Dimensionality Selection Criterion Experimental Results  Toy Datasets  Real-world Datasets Conclusions

6 Setup N labeled multi-class points: k nearest neighbors of in the same class: k nearest neighbors of in the other classes: Discriminant adjacent matrix F ：

7 Objective Function Intra-class compactness in the new space ： Inter-class separability in the new space ： (S is a diagonal matrix whose entries are column sums of F)

8 How to Compute P Note  The matrix X(S-F)X T is symmetric, but not positive definite. It might have negative, zero, or positive eigenvalues  The optimal transformation P can be obtained by the eigenvectors of X(S-F)X T corresponding to its all d negative eigenvalues P arg

9 What does the Positive/Negative Eigenvalue Mean? The i th eigenvector P i corresponding to the i th eigenvalue  : the total kNN pairwise distance in the same class  : the total kNN pairwise distance in different class

10 Choosing the Leading Negative Eigenvalues Among all the negative eigenvalues, some might have much larger absolute values, but the others with small absolute values could be ignored We can then choose t (t<d) negative eigenvalues with the largest absolute values such that

11 Learned Mahalanobis Distance In the original space, the distance between any pair of points can be obtained by

12 Outline Motivation  Related Work  Main Idea Proposed Algorithm  Discriminant Neighborhood Embedding  Dimensionality Selection Criterion Experimental Results  Toy Datasets  Real-world Datasets Conclusions

13 Three Classes of Well Clustered Data Both eigenvalues are negative and comparable Need not perform dimensionality reduction

14 Two Classes of Data with Multimodal Distribution A big difference between two negative eigenvalues The leading eigenvector P 1 corresponding to will be kept.

15 Three Classes of Data Two eigenvectors corresponding to positive and negative eigenvalues, respectively. The eigenvector with positive eigenvalue should be discarded from the point of view of kNN classification.

16 Five Classes of Non-separable Data Both eigenvalues are positive, and it means that we could not perform kNN classification well both in the original and new spaces

17 UCI Sonar Dataset When eigenvalues < 0, the more dimensionality, the higher accuracy When eigenvalues near 0, its optimum can be achieved When eigenvalues > 0, the performance decreases Cumulative eigenvalue curve

18 Comparisons with the State-of-the-Art

19 UMIST Face Database

20 Comparisons with the State-of-the-Art UMIST Face Database

21 Outline Motivation  Related Work  Main Idea The Proposed Algorithm  Discriminant Neighborhood Embedding  Dimensionality Selection Criterion Experimental Results  Toy Datasets  Real-world Datasets Conclusions

22 Conclusions Summary  A low dimensional embedding can be LEARNED for better accuracy in kNN classification given finite training samples  Optimal dimensionality can be estimated Future work  For large scale datasets, how to reduce the computational complexity?

Thanks for your Attention! Any questions?