Bag-of-Words based Image Classification Joost van de Weijer

What is in the image ? image classification: answers the question what is in the image. Is there a person ? Is there a suit-case ? Is there car ?

Inspiration The VOC Pascal challenge: a competition on image classification. Participants have to classify 20 classes in over images.

Inspiration

The Event Data Set 7 event classes: basketball, polo, rowing, castells, marathon, sailing, skiing. each class has 50 images, devided in 30 training and 20 test images.

Project I goal: build an image classification system which can successfully classify sport images. title: Bag-of-Words based Image Classification. competition: do so better than the other groups.

Why is this difficult ? Variations in viewpoint and zoom. Variations in pose.

Why is this difficult ? Inter-class variation. lighting changes.

Why is this difficult ? Back-ground variation. Maybe the background could help ? similar backgrounds- different classes.

from images to frequency histogram Compute visual words: detect local regions from a set of images. describe every local region by a descriptor texture color cluster all descriptors into visual words Given a new image: detect local regions from a set of image. assign to visual word N assign every region to its nearest visual word. compute visual word-image histogram

Bag of Visual Words representation Feature Detection Bag-of-Words representation normalize patches No spatial relations.

p i (w|Miro) p i (w|Dali) Bag of Visual Words representation

Image Representation 2. Extraction shape texture color Image 1. Feature detection shape words 3.Learn vocabulary Shape Voc image classification image retrieval 4. BOW 5. SVM/ distance measures The Framework

Image Representation Image 1. Feature detection 1.random 2. Extraction shape texture color 2. RGB shape words 3.Learn vocabulary Shape Voc 3. random 4. BOW 4. nearest neighbor image classification image retrieval 5. SVM/ distance measures 5. linear SVM Existing Implementation: 50 % classification score

Existing Implementation: properties of BOW implementation: you can improve any of the subroutines and analyze the changes based on the classification results. several team members can work on feature detection while others work on feature description. the final classification results allow us to compare the results between the groups.

Project I: Bag Bag-of-Words based Image Classification goal : build an image classification system which can successfully classify sport images. teaching objectives you will learn: to represent images robust to changes of cameras, object orientation, and illuminant color. what photometric invariance theory is and how to apply it to a real-world problem. understand and use the SIFT descriptor. how to discretize image features (colors, shapes, and textures). what the strong and weak points of BOW representations for images are. how to evaluate retrieval and classification results.

Practical information: Group Size: The project has to be made in groups of 3 students. Each group should decide on the following roles: responsible competition. responsible presentation. responsible report Practical Information: All practical information can be found in the student guide ( ) If it is hard to work as a group you can partition the tasks: feature detection feature description vocabulary construction learning/evaluation All group-members should understand all steps in the final program !

Practical information: Important Dates: 22 jan - 19 Feb.: The project will last 1 month. 22 jan.: Start project. 29 Jan.: Extra assignment will be handed out. Submission of first results in AP. 5 Feb.: Discussion meeting + submission second results in AP. 11 Feb. : Publication of final test set. 12 Feb. : Discussion meeting with groups separately. 15 Feb.: Final submission of classification results in AP for all classes. 19 Feb. : Presentation of the project. 22 Feb.: Final submission date for report. Supervision: There will be project meetings on Tuesdays afternoon to discus progress. For any questions during the three weeks of the project or come to office O/119 in the CVC. Use “PROJECT I” as subject of your s, which makes it easier to manage.

Practical information: Notes The final note will be based on: participation (15%) presentation (25%) report (50%) competition (10%) Bugs: For sure there will be several bugs in the code. If you find one, mail me, and I will notify the other groups. Thanks !

Practical information: Competition: Dates: 29 Jan. : Submission of first results in AP (before 15:00). 5 Feb. : Submission second results in AP (before 15:00) labeled train set labeled test set 19/22 Feb.: Your report/final presentation is based on the labeled test set !

Practical information: labeled train set no labels for test set ! Competition: Dates: 11 Feb. : Publication of final test set. 15 Feb.: Final submission results in AP for all calsses.

Practical information: Final Report The final report has to be submitted on 22 th of February. The report should contain the following chapters. Introduction ( max 1 page ) Feature Detection (max 2 pages). Feature Description (max 3 pages). Visual Vocabulary and BOW representation (max 2 pages) Classification (max 2 pages) Object Detection (optional: max 2 pages) Results (max 2 pages). Conclusions (max 1 page)

What to do next ? make groups of and assign : responsible competition (send an to me today or tomorrow ) install the programs and play with the code. ( ) This week you should already start working on a feature detector.

What to do next ? Good Luck !