1. C ONSTRAINED S EMI -S UPERVISED L EARNING USING A TTRIBUTES AND C OMPARATIVE A TTRIBUTES Abhinav Shrivastava, Saurabh Singh, Abhinav Gupta The Robotics Institute Carnegie Mellon University

2. S UPERVISION S UPERVISED A CTIVE L EARNING B IG -D ATA 2 [von Ahn and Dabbish, 2004], [Russell et al., IJCV 2009] [Prakash and Parikh, ECCV 2012] [Vijayanarasimhan and Grauman, CVPR 2011] [Kapoor et al., ICCV 2007] [Qi et al., CVPR 2008] [Joshi et al., CVPR 2009] [Siddiquie and Gupta, CVPR 2010]

3. S UPERVISION S UPERVISED U NSUPERVISED A CTIVE L EARNING 3 [Russell et al., CVPR 2006] [Kang et al., ICCV 2011]

4. S UPERVISION S UPERVISED U NSUPERVISED A CTIVE L EARNING S EMI -S UPERVISED 4 [Zhu, TR, 2005], [Chunsheng Fang, Slides, 2009]

5. S UPERVISION S UPERVISED U NSUPERVISED A CTIVE L EARNING S EMI -S UPERVISED B OOTSTRAPPING Labeled Seed Examples Amphitheatre Unlabeled Data Select Candidates Train Models Add to Labeled Set Retrain Models Amphitheatre 5

6. S UPERVISION S UPERVISED U NSUPERVISED A CTIVE L EARNING S EMI -S UPERVISED B OOTSTRAPPING Retrain Models Labeled Seed Examples Amphitheatre Unlabeled Data Select Candidates Add to Labeled Set Amphitheatre 25 th Iteration 6 [Curran et al., PACL 2007] Semantic Drift Amphitheatre + Auditorium

7. S UPERVISION S UPERVISED U NSUPERVISED A CTIVE L EARNING S EMI -S UPERVISED G RAPH - BASED M ETHODS 7 [Ebert et al., ECCV 2010] [Fergus et al., NIPS 2009]

8. Amphitheatre O UR APPROACH Amphitheatre Auditorium Amphitheatre Auditorium 8

9. O UR APPROACH Amphitheatre Auditorium Amphitheatre Auditorium 9 Joint Learning [Carlson et al., NAACL HLT Workshop on SSL for NLP 2009] Share Data

10. Amphitheatre Auditorium Banquet Hall Banquet Hall Conference Room Conference Room B INARY A TTRIBUTES (BA) 10 [Farhadi et al., CVPR 2009] [Lampert et al., CVPR 2009]

11. B INARY A TTRIBUTES (BA) Tables and Chairs Conference Room Banquet Hall Auditorium Amphitheatre Indoor Large Seating Capacity Man-made [Patterson and Hays, CVPR 2012] Tables and Chairs Conference Room Banquet Hall Auditorium Amphitheatre Indoor Large Seating Capacity Man-made

12. Auditorium 12

13. S HARING VIA D ISSIMILARITY 13 AmphitheatreAuditorium Has Larger Circular Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]

14. AmphitheatreAuditorium ? 14

15. 15 AmphitheatreAuditorium

16. D ISSIMILARITY 16 Has Larger Circular Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] C OMPARATIVE A TTRIBUTES

17. D ISSIMILARITY 17 C OMPARATIVE A TTRIBUTES Has Larger Circular Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] ………… Features GIST RGB (Tiny Image) Line Histogram of:  Length  Orientation LAB histogram [Hays and Efros, SIGGRAPH 2007] [Oliva and Torralba, 2006] [Torralba et al., PAMI, 2008]

18. ………… D ISSIMILARITY 18 C OMPARATIVE A TTRIBUTES [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] ………… Has Larger Circular Structures Classifier Boosted Decision Tree [Hoiem et al., IJCV 2007] or Has Larger Circular Structures

19. C OMPARATIVE A TTRIBUTES 19 [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] Amphitheatre>Barn Amphitheatre>Conference Room Desert>Barn Is More Open Church (Outdoor)>Cemetery Barn>Cemetery Has Taller Structures

20. Amphitheatre Auditorium Amphitheatre Auditorium Labeled Seed Examples Bootstrapping Selected Candidates 20

21. Labeled Seed Examples Amphitheatre Auditorium Amphitheatre Auditorium Bootstrapping Amphitheatre Auditorium Our Approach (Constrained Bootstrapping) Selected Candidates Indoor Has Seat Rows Attributes Has Larger Circular Structures Comparative Attributes 21

22. Banquet Bedroom Labeled Data Unlabeled Data has more space has larger structures Training Pairwise Data Promoted Instances Conference Room Banquet Hall 22 [Gupta and Davis, ECCV 2008] Comparative Attribute Classifiers more space larger structures Attribute Classifiers indoor has grass Scene Classifiers bedroom banquet hall

23. Conference Room Iteration 1 Iteration 40 Seed Examples 23 Introspection

24. Bootstrapping BA Constraints Amphitheatre Our Approach Seed Images 24

25. BA Constraints Bridge Seed Images Bootstrapping Our Approach 25

26. E XPERIMENTAL E VALUATION 26

27. C ONTROL E XPERIMENTS 27 # Images (SUN Database) 15 Scene Classes2 (seed) Black-box Binary Attributes (BA) 25 (separate) Black-box Comparative Attributes (CA) 25 (separate) Unlabeled Dataset18,000 (Distractors)(9,500) Test Set50 SUN Database : [Xiao et al., CVPR 2010]

28. C ONTROL E XPERIMENTS 28

29. C ONTROL E XPERIMENTS 29

30. C ONTROL E XPERIMENTS 30

31. C ONTROL E XPERIMENTS 31

32. C ONTROL E XPERIMENTS 32 \\

33. C ONTROL E XPERIMENTS 33 \\ Eigen Functions: [Fergus et al., NIPS 2009]

34. 1 40 Banquet Hall 10 Iterations Seed Images 34

35. C ONTROL E XPERIMENTS 35 # Images (SUN Database) 15 Scene Classes2 (seed) Black-box Binary Attributes (BA) 25 (separate) Black-box Comparative Attributes (CA) 25 (separate) Unlabeled Dataset18,000 (Distractors)(9,500) Test Set50 SUN Database : [Xiao et al., CVPR 2010]

36. C O - TRAINING (S MALL S CALE ) 36 # Images (SUN Database) 15 Scene Classes2 (seed) Black-box Binary Attributes (BA) 15x2 (seed) Black-box Comparative Attributes (CA) 15x2 (seed) Unlabeled Dataset18,000 (Distractors)(9,500) Test Set50 SUN Database : [Xiao et al., CVPR 2010]

37. Iteration-1 Iteration-60 Bootstrapping Our Approach Iteration-1 Iteration-60 Seed Images Bedroom 37

38. S CENE C LASSIFICATION 38 Eigen Functions: [Fergus et al., NIPS 2009]

39. C O - TRAINING ( LARGE S CALE ) 15 Scene Categories  25 Seed images / category Unlabeled Set  1Million (SUN Database + ImageNet)  >95% distractors 39 SUN Database: [Xiao et al., CVPR 2010] ImageNet: [Deng et al., CVPR 2009] Improve 12 out of 15 scene classifiers

40. C ONCLUSION Sharing via Dissimilarities Constrained Bootstrapping 40 AuditoriumAmphitheatre Has Larger Circular Structures Labeled Seed Examples Amphitheatre Auditorium Amphitheatre Auditorium Bootstrapping Amphitheatre Auditorium Constrained Bootstrapping

41. 41 Banquet Bedroom Labeled Data Unlabeled Data has more space has larger structures Training Pairwise Data Promoted Instances Conference Room Banquet Hall Comparative Attribute Classifiers more space larger structures Attribute Classifiers indoor has grass Scene Classifiers bedroom banquet hall

42. 42

43. 43 Unary Binary

44. E IGEN F UNCTIONS 44

45. F EATURES 960D GIST 75D RGB (image is resized to 5x5) 30D histogram of line lengths 200D histogram of orientation of lines 784D 3D-histogram Lab color space (14x14x4) Total 2049 45 [Hays and Efros, SIGGRAPH 2007] [Oliva and Torralba, 2006] [Torralba et al., PAMI, 2008]

46. C ATEGORIES 46

47. B INARY A TTRIBUTE R ELATIONSHIP 47

48. C OMPARATIVE A TTRIBUTE R ELATIONSHIPS 48

49. C LASSIFIERS Boosted Decision Trees – From [Hoiem et al., IJCV 2007] Scene – 20 Trees, 8 Nodes Binary Attribute – 40 Trees, 8 Nodes Comparative Attribute 20 Trees, 4 Nodes Differential features as in [Gupta and Davis, ECCV 2008] 49

50. S UPERVISION S UPERVISED A CTIVE L EARNING 50 [Torralba et al., PAMI 2008]

51. S UPERVISION S UPERVISED U NSUPERVISED A CTIVE L EARNING S EMI -S UPERVISED G RAPH - BASED M ETHODS Train Bus 51

52. M UTUAL E XCLUSION (ME) 52

53. E XPERIMENTAL E VALUATION Experiments Control Small-scale Large-scale Evaluation Metrics Scene Classification – Mean AP Purity of Labels Datasets SUN Database ImageNet 15 Scene Categories 53

54. B ASELINES Bootstrapping (Self-learning) -Binary Classifiers -Multi-class Classifiers Eigen Function based Graph Laplacian 54

55. S CENE C LASSIFICATION 55 Eigen Functions: [Fergus et al., NIPS 2009]

56. P URITY OF L ABELING 56 Eigen Functions: [Fergus et al., NIPS 2009]

57. 57

58. 1 40 Banquet Hall 90 10 Iterations Seed Images 58

59. 59

60. P URITY OF L ABELING 60 Eigen Functions: [Fergus et al., NIPS 2009]

61. S CENE C LASSIFICATION 61 Mean