1. Transfer Learning of Object Classes: From Cartoons to Photographs NIPS Workshop Inductive Transfer: 10 Years Later Geremy Heitz Gal Elidan Daphne Koller December 9 th, 2005

2. Localization vs. Recognition Traditional question: “Is there an object of type X in this image?” Airplane? NO Human? YES Dog? YES Our question: “Where in this image is the object of type X?” MAN DOG The man is walking the dog

3. Outline Landmark-based shape model Localization as inference Transfer learning from cartoon drawings Results

4. Shape Model Set of landmarks Piecewise-linear contour between neighbors Features of individual landmarks Features of pairs of landmarks tail nose

5. Outline Landmark-based shape model Localization as inference Transfer learning from cartoon drawings Results

6. “Registering” the Model to an Image Requires assigning each landmark to a pixel location ? ?

7. Localization Are local cues enough? Need to jointly consider all cues (features) “Correct” pixel is often not the best match! Markov Random Field Potentials = Functions of local and global features Inference using max-product Registration = Most Likely Assignment L nose L tail L under L cockpit

8. Outline Landmark-based shape model Localization as inference Transfer learning from cartoon drawings Results

9. Bootstrap from simple instances where outlining is easy = cartoons / drawings Learning Challenge Hand LabelHidden Variables Costly, and time-consuming Where to start? Local optima problem no confusing background outline (shape) is easily recovered using snake ? ? ? ? ?

10. Learning from Cartoon Drawings Registration Shape Learning Shape and Appearance Learning +

11. Phase I: Learning from Cartoons Extract high resolution contour using snake Create shape-based model from training contours Pairwise merging of models Selection of landmarks Registration Pyramid Final Shape Model

12. Training Set Selection high score low score Phase II: Learning from Images Correspond initial model to training images Select best correspondences as training instances Learn final shape- and appearance-based model Cartoon Phase Model Natural Image Model Transfer

13. Outline Landmark-based shape model Localization as inference Transfer learning from cartoon drawings Results

14. Localization Results 0.84 0.750.840.720.18 0.81 0.660.770.40 sample training cartoons sample registration

15. Transfer of Object Shape Transfer of shape speeds up learning Benefit of shape transfer 0246810 0 0.1 0.2 0.3 0.4 0.5 0.6 # images in phase II Average overlap transfer no transfer

16. Learning Appearance No Appearance FG/BG Appearance 0246810 0.46 0.48 0.5 0.52 0.54 0.56 0.58 0.6 0.62 0.64 Average overlap Shape template shape + appearance # images in phase II

17. Training Instance Selection AUTOPICKED 024681012 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 AUTO PICKED HAND Average overlap # images in phase II

18. Summary and Future Work  Flexible probabilistic shape model  Effective registration to images  Transfer  Shape from cartoons  Appearance from real images Develop a better appearance model Investigate self-training issues Transfer from one class to another

19. Thanks!

20. Cartoon vs. Hand Segmentation 012345 Number of Training Instances 0.1 0.3 0.5 0.7 0.9 Mean Overlap Score Learned from Drawings Hand Constructed Human Inter-Observer cartoon hand segmented Learning shape from cartoons is competitive with hand segmentation!

21. Landmark Features Shape Template Patch Appearance (Foreground/Background) Location

22. Prediction 00.20.40.60.81 False positive rate 0 0.2 0.4 0.6 0.8 1 True positive rate objectrecognition car side86% cougar86% airplane86% buddha84% bass76% rooster73%  Comparable to constellation w/ 5 instances (Fei Fei et. Al)  Leading (discriminative) methods require many instances