1. Joint Image Clustering and Labeling by Matrix Factorization
Seunghoon Hong
CV Lab.,
POSTECH

2. Motivation OBJECTIVE : From a set of unlabeled, unorganized images,
we want to find meaningful clusters and
associated labels about each cluster.
“Zoo”
“Park”, “Tree”
“Apple”, “Pear”
Ref-DB
Unorganized images

3. Approaches OPT 1. Supervised learning
Learn all possible categories from Ref-DB, and apply the model to test images
PROBLEMS
Learning a model for large number of categories is difficult
More importantly, it may not necessary

4. Approaches OPT 2. k-NN based approach
performing k-NN search for individual test image on Ref-DB to obtain labels for images
PROBLEMS
Obtained labels are noisy because,
K-NN is obtained based on visual similarity
K-NN is obtained for each test image independently

5. Approaches
OPT 3. Cluster test images, and Obtain labels for each cluster
Cluster test images, and obtain labels for each cluster
PROBLEMS
Images in each cluster may not semantically related.
Finding labels for each cluster may not trivial

6. Joint Clustering and Annotation
Proposed method
Ref-DB
car
dog
bird …
deer
Word Feature
SO-NMF
Visual Feature
Joint Clustering and Annotation 1
각 순서에 대한 notation같은걸 만들어 놓으면 좋을듯

7. Proposed method
Obtain human-interpretable mid-level features for test images based on k-NN search on Ref-DB (word feature).
Perform clustering on test images with suitable constraints on mid-level semantic feature.
Assign labels for each cluster directly from mid-level feature.
BENEFITS
Clustering is performed considering semantic relationship b/w images.
Candidate labels are bounded by test set.
(extension : learn relevant concepts and do classification)

8. Step 1. Word feature extraction

9. Joint Clustering and Annotation
Ref-DB
car
dog
bird …
deer
Word Feature
SO-NMF
Visual Feature
Joint Clustering and Annotation 1
각 순서에 대한 notation같은걸 만들어 놓으면 좋을듯

10. Word-feature Construction
Transform feature domain from visual to word space.
Procedure
Extract k-Nearest Neighbors from database
Construct weighted histogram based on labels of k-NNs

13. Step 2. Clustering

14. Joint Clustering and Annotation
Ref-DB
car
dog
bird …
deer
Word Feature
SO-NMF
Visual Feature
Joint Clustering and Annotation 1
각 순서에 대한 notation같은걸 만들어 놓으면 좋을듯

15. Step 2. Clustering - NMF
matrix factorization

16. Step 2. Clustering - NMF By low rank approximation,
tag1
By low rank approximation,
Noise can be cleaned and
Result become more homogeneous
tag2
tag3
바로 앞 슬라이드의 basis가 여기서의 cluster center를 관통하는 화살표랑 연결된다는 그림을 추가하면 좋겠다

17. Step 2. Clustering - NMF Limitation of NMF
Diverse form of basis component can be found !!
Bird
Snail
Car
Dog
Cat
Wardrobe
television
Snu
mouse .
Bird
Snail
Car
Dog
Cat
Wardrobe
television
Snu
mouse .
Not helpful basis to find relevant concepts for dataset!
Need to find Sparse Basis

18. Step 2. Clustering - NMF Limitation of NMF
Membership of each data can be diverse. 1.
= [0.6, 0.4]
= [0.9, 0.1]

19. Step 2. Clustering - NMF
Desirable properties of cluster in word-feature space:
(Sparseness) : Cluster should associated with small number of representative keywords.
(Orthogonality) : Data should associated with one cluster.

20. Step 2. Clustering – NMFSC
NMF with sparse constraints
(Hoyer, 2004)

21. Step 1: Initialize W, H to random positive matrices
Step 2: If constraints apply to W or H or both, project each column or row respectively to have unchanged L2 norm and desired L1 norm
Step 3: iterate
If sparseness constraints on W apply,
Set W=W-μw(WH-A)HT
Project columns of W as in step 2
Else, take standard multiplicative step
If sparseness constraints on H apply
Set H=H- μHWT(WH-A)
Project rows of H as in step 2

23. Step 2. Clustering - ONMF
Algorithms for orthogonal matrix factorization (S.choi, 2008)
Optimize NMF with orthogonality constrained stiefel manifold

24. Optimize in stiefel manifold Constrained on

25. Step 2. Clustering - NMF
So can we just integrate these two constraint simply? W H
NO !
There is three constraints on two variable
Introduce additional variable to make up error in original obj.func.

26. Step 2. Clustering - SONMF
Sparse Orthogonal NMF (SO-NMF)

28. Step 2. Clustering - NMF
Final Labeling
Per cluster
Per Image

29. Experiment result Experiment settings : two dataset
1. CIFAR-100 (100 categories)
2. Image-net (30 categories, challenging variation)
CIFAR-100
Image-net
Categories
100
30 (more variation)
Test/training images
10K
50K
Features
gist
Color-histogram
Bag of words(SIFT)

30. Experiment result - Categorization Performance

31. Experiment result Effect of RDB quality
Categorization accuracy in the presence of missing labels in the Ref-DB
Categorization accuracy in the presence of incorrect labels in Ref-DB
Categorization accuracy by varying the number of clusters.

32. Experiment result Effect of RDB quality
Categorization accuracy in the presence of missing labels in the Ref-DB
Categorization accuracy in the presence of incorrect labels in Ref-DB
Categorization accuracy by varying the number of clusters.

33. Experiment result - Labeling Performance
Quality of Cluster Labels.
Quality of Image Labels.

34. Experiment result - Labeling Performance

35. Experiment result - Extension to Supervised Image Classification
Extension with extracted labels :
Learn on only relevant categories of dataset using supervised method
-in other words, Bound candidate classes on test imageset

36. Experiment result - Extension to Supervised Image Classification
Example confusion matrix of cifar-100
Example confusion matrix of ImageNet

37. Experiment result - Extension to Supervised Image Classification

38. Thank you