1. Relative Attributes Presenter: Shuai Zheng (Kyle) Supervised by Philip H.S. Torr Author: Devi Parikh (TTI-Chicago) and Kristen Grauman (UT-Austin)

2. What is visual attributes? Attributes are properties observable in images that have human-designated names, such as ‘Orange’, ‘striped’, or ‘Furry’.

3. Learning Binary Attributes In PASCAL VOC Challenge, we learn to predict binary attributes. (e.g., dog? Or not a dog?) Vittorio Ferrari, Andrew Zisserman. Learning Visual Attributes. NIPS 2007. O. Parkhi, A.Vedaldi C.V.Jawahar, A.Zisserman. The Truth About Cats and Dogs. ICCV 2011.

4. Problems within Binary Attributes Given an attribute it is easy to get labelled data on AMT(Amazon Mechanical Turk). But, where do attributes come from? Can we find a easier way to ask more people rather than experts to tag the images?

5. Problems within Binary Attributes Some tags are binary while some are relative. Is furry Has four-legs Has tail Tail longer than donkeys’ Legs shorter than horses’ Mule

6. Labeling data

7. What is relative attributes? Relative attribute indicates the strength of an attribute in an image with respect to other image rather than simply predicting the presence of an attribute.

8. Advantages of Relative Attributes Enhanced human-machine communication More informative Natural for humans Enhanced human-machine communication More informative Natural for humans 8

9. Contributions Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image

10. Contributions Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image

11. Learning Relative Attributes For each attribute Supervision is open

12. Learning Relative Attributes Learn a scoring function that best satisfies constraints: 12 Image features Learned parameters

13. Learning Relative Attributes 13 Max-margin learning to rank formulation Based on [Joachims 2002] 1 2 3 4 56 Rank Margin Image Relative Attribute Score

14. Learning binary attributes v.s. Learning relative attributes

15. Contributions Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image

16. Relative Zero-shot Learning Training: Images from S seen categories and Descriptions of U unseen categories Need not use all attributes, or all seen categories Testing: Categorize image into one of S+U categories 16 Age: Scarlett CliveHugh Jared Miley Smiling: Jared Miley

17. Relative Zero-shot Learning Clive Infer image category using max-likelihood Can predict new classes based on their relationships to existing classes – without training images 17 Age: Scarlett CliveHugh Jared Miley Smiling: Jared Miley Smiling Age Miley S J H

18. Contributions Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image

19. Automatic Relative Image Description Density Conventional binary description: not dense Dense:Not dense: Novel image 19

20. more dense than less dense than Density Novel image 20 Automatic Relative Image Description

21. CCHHHCFHHMFFIF more dense than Highways, less dense than Forests Density Novel image 21 Automatic Relative Image Description

22. Contributions Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image

23. Datasets Outdoor Scene Recognition (OSR) [Oliva 2001] 8 classes, ~2700 images, Gist 6 attributes: open, natural, etc. Public Figures Face (PubFig) [Kumar 2009] 8 classes, ~800 images, Gist+color 11 attributes: white, chubby, etc. 23 Attributes labeled at category level http://ttic.uchicago.edu/~dparikh/relative.html

24. Zero-shot learning – Binary attributes: Direct Attribute Prediction [Lampert 2009] – Relative attributes via classifier scores Automatic image-description – Binary attributes 24 + + + – – – Baselines 6 4 5 3 2 1 bearturtlerabbit furry big

25. Relative Zero-shot Learning An attribute is more discriminative when used relatively Binary attributes Rel. att. (classifier) 25 Rel. att.(ranke r)

26. Relative (ours): More natural than insidecity Less natural than highway More open than street Less open than coast Has more perspective than highway Has less perspective than insidecity Binary (existing): Not natural Not open Has perspective Automatic Relative Image Description 26

27. Automatic Relative Image Description 18 subjects Test cases: 10OSR, 20 PubFig 27

28. Traditional Recognition DogChimpanzeeTiger ??? Tiger 28

29. Attributes-based Recognition Furry White Black Big Striped Yellow Striped Black White Big Attributes provide a mode of communication between humans and machines! [Lampert 2009] [Farhadi 2009] [Kumar 2009] [Berg 2010] [Parikh 2010] … Zero-shot learning Describing objects Face verification Attribute discovery Nameable attributes … 29 DogChimpanzeeTiger

30. Conclusions and Future Work Relative attributes learnt as ranking functions – Natural and accurate zero-shot learning of novel concepts by relating them to existing concepts – Precise image descriptions for human interpretation Attributes-based recognition is an interesting direction for the future object/scenes recognition. 30 Enhanced human-machine communication

31. Cheers! – Shuai Zheng (Kyle) kylezheng04@gmail.com Created by Tag clouds