C ONSTRAINED S EMI -S UPERVISED L EARNING USING A TTRIBUTES AND C OMPARATIVE A TTRIBUTES Abhinav Shrivastava, Saurabh Singh, Abhinav Gupta The Robotics Institute Carnegie Mellon University

S UPERVISION S UPERVISED A CTIVE L EARNING B IG -D ATA 2 [von Ahn and Dabbish, 2004], [Russell et al., IJCV 2009] [Prakash and Parikh, ECCV 2012] [Vijayanarasimhan and Grauman, CVPR 2011] [Kapoor et al., ICCV 2007] [Qi et al., CVPR 2008] [Joshi et al., CVPR 2009] [Siddiquie and Gupta, CVPR 2010]

S UPERVISION S UPERVISED U NSUPERVISED A CTIVE L EARNING 3 [Russell et al., CVPR 2006] [Kang et al., ICCV 2011]

S UPERVISION S UPERVISED U NSUPERVISED A CTIVE L EARNING S EMI -S UPERVISED 4 [Zhu, TR, 2005], [Chunsheng Fang, Slides, 2009]

S UPERVISION S UPERVISED U NSUPERVISED A CTIVE L EARNING S EMI -S UPERVISED B OOTSTRAPPING Labeled Seed Examples Amphitheatre Unlabeled Data Select Candidates Train Models Add to Labeled Set Retrain Models Amphitheatre 5

S UPERVISION S UPERVISED U NSUPERVISED A CTIVE L EARNING S EMI -S UPERVISED B OOTSTRAPPING Retrain Models Labeled Seed Examples Amphitheatre Unlabeled Data Select Candidates Add to Labeled Set Amphitheatre 25 th Iteration 6 [Curran et al., PACL 2007] Semantic Drift Amphitheatre + Auditorium

S UPERVISION S UPERVISED U NSUPERVISED A CTIVE L EARNING S EMI -S UPERVISED G RAPH - BASED M ETHODS 7 [Ebert et al., ECCV 2010] [Fergus et al., NIPS 2009]

Amphitheatre O UR APPROACH Amphitheatre Auditorium Amphitheatre Auditorium 8

O UR APPROACH Amphitheatre Auditorium Amphitheatre Auditorium 9 Joint Learning [Carlson et al., NAACL HLT Workshop on SSL for NLP 2009] Share Data

Amphitheatre Auditorium Banquet Hall Banquet Hall Conference Room Conference Room B INARY A TTRIBUTES (BA) 10 [Farhadi et al., CVPR 2009] [Lampert et al., CVPR 2009]

B INARY A TTRIBUTES (BA) Tables and Chairs Conference Room Banquet Hall Auditorium Amphitheatre Indoor Large Seating Capacity Man-made [Patterson and Hays, CVPR 2012] Tables and Chairs Conference Room Banquet Hall Auditorium Amphitheatre Indoor Large Seating Capacity Man-made

Auditorium 12

S HARING VIA D ISSIMILARITY 13 AmphitheatreAuditorium Has Larger Circular Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]

AmphitheatreAuditorium ? 14

15 AmphitheatreAuditorium

D ISSIMILARITY 16 Has Larger Circular Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] C OMPARATIVE A TTRIBUTES

D ISSIMILARITY 17 C OMPARATIVE A TTRIBUTES Has Larger Circular Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] ………… Features GIST RGB (Tiny Image) Line Histogram of:  Length  Orientation LAB histogram [Hays and Efros, SIGGRAPH 2007] [Oliva and Torralba, 2006] [Torralba et al., PAMI, 2008]

………… D ISSIMILARITY 18 C OMPARATIVE A TTRIBUTES [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] ………… Has Larger Circular Structures Classifier Boosted Decision Tree [Hoiem et al., IJCV 2007] or Has Larger Circular Structures

C OMPARATIVE A TTRIBUTES 19 [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] Amphitheatre>Barn Amphitheatre>Conference Room Desert>Barn Is More Open Church (Outdoor)>Cemetery Barn>Cemetery Has Taller Structures

Amphitheatre Auditorium Amphitheatre Auditorium Labeled Seed Examples Bootstrapping Selected Candidates 20

Labeled Seed Examples Amphitheatre Auditorium Amphitheatre Auditorium Bootstrapping Amphitheatre Auditorium Our Approach (Constrained Bootstrapping) Selected Candidates Indoor Has Seat Rows Attributes Has Larger Circular Structures Comparative Attributes 21

Banquet Bedroom Labeled Data Unlabeled Data has more space has larger structures Training Pairwise Data Promoted Instances Conference Room Banquet Hall 22 [Gupta and Davis, ECCV 2008] Comparative Attribute Classifiers more space larger structures Attribute Classifiers indoor has grass Scene Classifiers bedroom banquet hall

Conference Room Iteration 1 Iteration 40 Seed Examples 23 Introspection

Bootstrapping BA Constraints Amphitheatre Our Approach Seed Images 24

BA Constraints Bridge Seed Images Bootstrapping Our Approach 25

E XPERIMENTAL E VALUATION 26

C ONTROL E XPERIMENTS 27 # Images (SUN Database) 15 Scene Classes2 (seed) Black-box Binary Attributes (BA) 25 (separate) Black-box Comparative Attributes (CA) 25 (separate) Unlabeled Dataset18,000 (Distractors)(9,500) Test Set50 SUN Database : [Xiao et al., CVPR 2010]

C ONTROL E XPERIMENTS 28

C ONTROL E XPERIMENTS 29

C ONTROL E XPERIMENTS 30

C ONTROL E XPERIMENTS 31

C ONTROL E XPERIMENTS 32 \\

C ONTROL E XPERIMENTS 33 \\ Eigen Functions: [Fergus et al., NIPS 2009]

1 40 Banquet Hall 10 Iterations Seed Images 34

C ONTROL E XPERIMENTS 35 # Images (SUN Database) 15 Scene Classes2 (seed) Black-box Binary Attributes (BA) 25 (separate) Black-box Comparative Attributes (CA) 25 (separate) Unlabeled Dataset18,000 (Distractors)(9,500) Test Set50 SUN Database : [Xiao et al., CVPR 2010]

C O - TRAINING (S MALL S CALE ) 36 # Images (SUN Database) 15 Scene Classes2 (seed) Black-box Binary Attributes (BA) 15x2 (seed) Black-box Comparative Attributes (CA) 15x2 (seed) Unlabeled Dataset18,000 (Distractors)(9,500) Test Set50 SUN Database : [Xiao et al., CVPR 2010]

Iteration-1 Iteration-60 Bootstrapping Our Approach Iteration-1 Iteration-60 Seed Images Bedroom 37

S CENE C LASSIFICATION 38 Eigen Functions: [Fergus et al., NIPS 2009]

C O - TRAINING ( LARGE S CALE ) 15 Scene Categories  25 Seed images / category Unlabeled Set  1Million (SUN Database + ImageNet)  >95% distractors 39 SUN Database: [Xiao et al., CVPR 2010] ImageNet: [Deng et al., CVPR 2009] Improve 12 out of 15 scene classifiers

C ONCLUSION Sharing via Dissimilarities Constrained Bootstrapping 40 AuditoriumAmphitheatre Has Larger Circular Structures Labeled Seed Examples Amphitheatre Auditorium Amphitheatre Auditorium Bootstrapping Amphitheatre Auditorium Constrained Bootstrapping

41 Banquet Bedroom Labeled Data Unlabeled Data has more space has larger structures Training Pairwise Data Promoted Instances Conference Room Banquet Hall Comparative Attribute Classifiers more space larger structures Attribute Classifiers indoor has grass Scene Classifiers bedroom banquet hall

42

43 Unary Binary

E IGEN F UNCTIONS 44

F EATURES 960D GIST 75D RGB (image is resized to 5x5) 30D histogram of line lengths 200D histogram of orientation of lines 784D 3D-histogram Lab color space (14x14x4) Total [Hays and Efros, SIGGRAPH 2007] [Oliva and Torralba, 2006] [Torralba et al., PAMI, 2008]

C ATEGORIES 46

B INARY A TTRIBUTE R ELATIONSHIP 47

C OMPARATIVE A TTRIBUTE R ELATIONSHIPS 48

C LASSIFIERS Boosted Decision Trees – From [Hoiem et al., IJCV 2007] Scene – 20 Trees, 8 Nodes Binary Attribute – 40 Trees, 8 Nodes Comparative Attribute 20 Trees, 4 Nodes Differential features as in [Gupta and Davis, ECCV 2008] 49

S UPERVISION S UPERVISED A CTIVE L EARNING 50 [Torralba et al., PAMI 2008]

S UPERVISION S UPERVISED U NSUPERVISED A CTIVE L EARNING S EMI -S UPERVISED G RAPH - BASED M ETHODS Train Bus 51

M UTUAL E XCLUSION (ME) 52

E XPERIMENTAL E VALUATION Experiments Control Small-scale Large-scale Evaluation Metrics Scene Classification – Mean AP Purity of Labels Datasets SUN Database ImageNet 15 Scene Categories 53

B ASELINES Bootstrapping (Self-learning) -Binary Classifiers -Multi-class Classifiers Eigen Function based Graph Laplacian 54

S CENE C LASSIFICATION 55 Eigen Functions: [Fergus et al., NIPS 2009]

P URITY OF L ABELING 56 Eigen Functions: [Fergus et al., NIPS 2009]

57

1 40 Banquet Hall Iterations Seed Images 58

59

P URITY OF L ABELING 60 Eigen Functions: [Fergus et al., NIPS 2009]

S CENE C LASSIFICATION 61 Mean