1. Enhancing Human-Machine Communication via Visual Attributes Devi Parikh Virginia Tech

2. Interacting with Vision Systems UserSupervisor 2

3. Interacting with Vision Systems Semantic Gap 3 Mode of communication is important

4. Interacting with Vision Systems Necessary for communication – Language that humans understand (semantic) – Language that machines understand (visual) Attributes – Example: furry, natural, chubby, shiny, etc. – Better features, deeper image understanding, etc. Farhadi et al., Kumar et al., Lampert et al., etc. – Human-machine communication 4

5. SupervisorUser Reading Between the Lines Supervisor Role of the Human Communicator SupervisorUser Human Machine Image SearchInstilling Domain Knowledge Characterizing Failure Modes Interpretable Models My missing brother is fuller-faced than this boy. Polar bears are white and larger than rabbits. If the image is blurry or the face is not frontal, I may fail. I think this is a polar bear because this is a white and furry animal. Active and Interactive Learning 5

6. SupervisorUser Reading Between the Lines Supervisor Role of the Human Communicator SupervisorUser Human Machine Image SearchInstilling Domain Knowledge Characterizing Failure Modes Interpretable Models My missing brother is fuller-faced than this boy. Polar bears are white and larger than rabbits. If the image is blurry or the face is not frontal, I may fail. I think this is a polar bear because this is a white and furry animal. Active and Interactive Learning 6

7. Image Search Query: “black shoes” … 7 Binary Relevance Feedback

8. Image Search Query: “black shoes” … “shinier than these” “more formal than these” … 8

9. Relative Attributes Openness 9 Linear ranking function: open Training Testing [Parikh and Grauman, ICCV 2011]

10. Image Search System has pre-trained relative attribute predictors Relevance of image = # constraints satisfied 10 … “shinier” “more formal”

11. WhittleSearch shiny formal … “shinier” “more formal” 11

12. WhittleSearch shiny formal 12

13. WhittleSearch 13 [Kovashka, Parikh and Grauman, CVPR 2012] (Patent pending) 13

14. Whittle Search: Demo (Online) 14 [Prepared by Naman Agrawal, Demo at CVPR 2013] (Patent pending) 14

15. SupervisorUser Reading Between the Lines Supervisor Role of the Human Communicator SupervisorUser Human Machine Image SearchInstilling Domain Knowledge Characterizing Failure Modes Interpretable Models My missing brother is fuller-faced than this boy. Polar bears are white and larger than rabbits. If the image is blurry or the face is not frontal, I may fail. I think this is a polar bear because this is a white and furry animal. Active and Interactive Learning 15

16. SupervisorUser Reading Between the Lines Supervisor Role of the Human Communicator SupervisorUser Human Machine Image SearchInstilling Domain Knowledge Characterizing Failure Modes Interpretable Models My missing brother is fuller-faced than this boy. Polar bears are white and larger than rabbits. If the image is blurry or the face is not frontal, I may fail. I think this is a polar bear because this is a white and furry animal. Active and Interactive Learning 16

17. SupervisorUser Reading Between the Lines Supervisor Role of the Human Communicator SupervisorUser Human Machine Image SearchInstilling Domain Knowledge Characterizing Failure Modes Interpretable Models My missing brother is fuller-faced than this boy. Polar bears are white and larger than rabbits. If the image is blurry or the face is not frontal, I may fail. I think this is a polar bear because this is a white and furry animal. Active and Interactive Learning 17

18. Traditional Active Learning Is this a forest? No, this is not a forest. 18

19. [Parkash and Parikh, ECCV 2012] Classifier Feedback I think this is a forest. What do you think ? No, this is too open to be a forest. … Ah! These images must not be forests either then. 19 [Images more open than query]

20. Classifier Feedback I think this is a forest. What do you think ? No, this is too open to be a forest. … Ah! These images must not be forests either then. 20 [Images more open than query] Pre-trained relative attributes

21. Classifier Feedback I think this is a forest. What do you think ? No, this is too open to be a forest. … Ah! These images must not be forests either then. 21 [Images more open than query] Learn attributes on the fly

22. Classifier Feedback I think this is a forest. What do you think ? No, this is too open to be a forest. Ah! These images must be less open than query 22 … [images labeled as forest]

23. [Biswas and Parikh, CVPR 2013] Classifier Feedback Learning attributes on the fly – Start only with unlabeled images (+ a supervisor) – Categories and attributes learnt from scratch Confidence in instances Active learning for learning with attributes- based classifier feedback 23

24. Classifier Feedback Number of iterationsAccuracy 24 Parkash and Parikh ECCV 2012 Biswas and Parikh CVPR 2013

25. SupervisorUser Reading Between the Lines Supervisor Role of the Human Communicator SupervisorUser Human Machine Image SearchInstilling Domain Knowledge Characterizing Failure Modes Interpretable Models My missing brother is fuller-faced than this boy. Polar bears are white and larger than rabbits. If the image is blurry or the face is not frontal, I may fail. I think this is a polar bear because this is a white and furry animal. Active and Interactive Learning 25

26. WhittleSearch Query: “black shoes” … “shinier than these” “more formal than these” … 26

27. Image Search 27 [Parikh and Grauman, ICCV 2013]

28. Saying the Right Thing Smiling more thanNot smiling 28 [Sadovnik, Gallagher, Parikh and Chen, ICCV 2013] Improved image search, description

29. Saliency of Attributes Improved image search, zero-shot learning, description White, furry Scary, sharp teeth 29 [Turakhia and Parikh, ICCV 2013]

30. SupervisorUser Reading Between the Lines Supervisor Role of the Human Communicator SupervisorUser Human Machine Image SearchInstilling Domain Knowledge Characterizing Failure Modes Interpretable Models My missing brother is fuller-faced than this boy. Polar bears are white and larger than rabbits. If the image is blurry or the face is not frontal, I may fail. I think this is a polar bear because this is a white and furry animal. Active and Interactive Learning 30 Accessing user’s intensions for mental image search More usable computer vision systems even with their imperfections Trustworthy systems: key for effective human- machine teams Integrating AI with today’s machine learning tools Getting more from what the human says without added human effort Enhanced human-machine communication via attributes for improved visual recognition

31. Thank you!