1. 5/30/2006EE 148, Spring 20061 Visual Categorization with Bags of Keypoints Gabriella Csurka Christopher R. Dance Lixin Fan Jutta Willamowski Cedric Bray Presented by Yun-hsueh Liu

2. 5/30/2006EE 148, Spring 20062 What is Generic Visual Categorization? Categorization: distinguish different classes Generic Visual Categorization: Generic  to cope with many object types simultaneously readily extended to new object types. Handle the variation in view, imaging, lighting, occlusion, and typical object and scene variations

3. 5/30/2006EE 148, Spring 20063 Previous Work in Computational Vision Single Category Detection Decide if a member of one visual category is present in a given image. (faces, cars, targets) Content Based Image Retrieval Retrieve images on the basis of low-level image features, such as colors or textures. Recognition Distinguish between images of structurally distinct objects within one class. (say, different cell phones)

4. 5/30/2006EE 148, Spring 20064 Bag-of-Keypoints Approach Interesting Point Detection Key Patch Extraction Feature Descriptors Bag of Keypoints Multi-class Classifier

5. 5/30/2006EE 148, Spring 20065 SIFT Descriptors Interesting Point Detection Key Patch Extraction Feature Descriptors Bag of Keypoints Multi-class Classifier

6. 5/30/2006EE 148, Spring 20066 Bag of Keypoints (1) Construction of a vocabulary Kmeans clustering  find “centroids” (on all the descriptors we find from all the training images) Define a “vocabulary” as a set of “centroids”, where every centroid represents a “word”. Interesting Point Detection Key Patch Extraction Feature Descriptors Bag of Keypoints Multi-class Classifier

7. 5/30/2006EE 148, Spring 20067 Bag of Keypoints (2) Histogram Counts the number of occurrences of different visual words in each image Interesting Point Detection Key Patch Extraction Feature Descriptors Bag of Keypoints Multi-class Classifier

8. 5/30/2006EE 148, Spring 20068 Multi-class Classifier In this paper, classification is based on conventional machine learning approaches Naïve Bayes Support Vector Machine (SVM) Interesting Point Detection Key Patch Extraction Feature Descriptors Bag of Keypoints Multi-class Classifier

9. 5/30/2006EE 148, Spring 20069 Multi-class classifier – Naïve Bayes (1) Let V = {v i }, i = 1,…,N, be a visual vocabulary, in which each v i represents a visual word (cluster centers) from the feature space. A set of labeled images I = {I i }. Denote C j to represent our Classes, where j = 1,..,M N(t,i) = number of times v i occurs in image I i (keypoint histogram) Score approach: want to determine P(C j |I i ), where (*)

10. 5/30/2006EE 148, Spring 200610 Multi-class Classifier – Naïve Bayes (2) Goal: Find one specific class C j so that has maximum value In order to avoid zero probability, use Laplace smoothing:

11. 5/30/2006EE 148, Spring 200611 Multi-class classifier – Support Vector Machine (SVM) Input: the keypoints histogram for each image Multi-class  one-against-all approach Linear SVM gives better performances than quadratic or cubic SVM Goal: find hyperplanes which separate multi-class data with maximun margin

12. 5/30/2006EE 148, Spring 200612 Multi-class classifier – SVM (2)

13. 5/30/2006EE 148, Spring 200613 Evaluation of Multi-class Classifiers Three performance measures: The confusion matrix Each column of the matrix represents the instances in a predicted class Each row represents the instances in an actual class The overall error rate  = Pr(output class = true class) The mean ranks The mean position of the correct labels when labels output by the multi- class classifier are sorted by the classifier score.

14. 5/30/2006EE 148, Spring 200614 n-Fold Cross Validation What is “fold”? Randomly break the dataset into n partitions Example: suppose n = 10 Training on 2, 3,…,10; testing on 1 = result 1 Training on 1, 3,…,10; testing on 2 = result 2 … Answer = Average of result 1, result 2, ….

15. 5/30/2006EE 148, Spring 200615 Experiment on Naïve Bayes – k’s effect Present the overal error rate as a function of # of clusters k Result Error rate decreases as k increases Selecting point: k = 1000 After passing the selecting point, the error rate decreases slowly

16. 5/30/2006EE 148, Spring 200616 Experiment on Naïve Bayes – Confusion Matrix facesbuildingstreescarsphonesbikesbooks faces764234413 buildings24450513 trees32800050 cars41075314 phones915116701411 bikes2151208730 books419067269 error rate 2456202527 31 mean rank 1.491.881.33 1.631.57

17. 5/30/2006EE 148, Spring 200617 Experiment on SVM – Confusion Matrix facesbuildingstreescarsphonesbikesbooks faces981410 34013 buildings16330316 trees110811060 cars01185505 phones05435523 bikes04101910 books03012073 error rate227191545927 mean rank 1.041.771.281.301.831.091.39

18. 5/30/2006EE 148, Spring 200618 Interpretation of Results The confusion matrix In general, SVM has more correct predictions than Naïve Bayes does The overall error rate In general, Naïve Bayes > SVM The Mean Rank In general, SVM < Naïve Bayes

19. 5/30/2006EE 148, Spring 200619 Why do we have errors? There are objects from more than 2 classes in one image The data set is not totally clean (noise) Each image is given only one training label

20. 5/30/2006EE 148, Spring 200620 Conclusion Bag-of-Keypoints is a new and efficient generic visual categorizer. Evaluated on a seven-category database, this method is proved that it is robust to Choice of clusters, background clutter, multiple objects Any Questions? Thank you for listening to my presentation!! :)