1. Part 4: Combined segmentation and recognition by Rob Fergus (MIT)

2. Aim Given an image and object category, to segment the object Segmentation should (ideally) be shaped like the object e.g. cow-like obtained efficiently in an unsupervised manner able to handle self-occlusion Segmentation Object Category Model Cow Image Segmented Cow Slide from Kumar ‘05

3. Feature-detector view

7. Examples of bottom-up segmentation Using Normalized Cuts, Shi & Malik, 1997 Borenstein and Ullman, ECCV 2002

8. Jigsaw approach: Borenstein and Ullman, 2002

9. Perceptual and Sensory Augmented Computing Interleaved Object Categorization and Segmentation Implicit Shape Model - Liebe and Schiele, 2003 Backprojected Hypotheses Interest Points Matched Codebook Entries Probabilistic Voting Voting Space (continuous) Backprojection of Maxima Segmentation Refined Hypotheses (uniform sampling) Liebe and Schiele, 2003, 2005

10. Random Fields for segmentation I = Image pixels (observed) h = foreground/background labels (hidden) – one label per pixel  = Parameters Prior LikelihoodPosteriorJoint 1.Generative approach models joint  Markov random field (MRF) 2. Discriminative approach models posterior directly  Conditional random field (CRF)

11. I (pixels) Image Plane i j h (labels)  {foreground, background} hihi hjhj Unary Potential  i ( I |h i,  i ) Pairwise Potential (MRF)  ij (h i, h j |  ij ) MRF PriorLikelihood Generative Markov Random Field Prior has no dependency on I

12. Conditional Random Field Lafferty, McCallum and Pereira 2001 PairwiseUnary Dependency on I allows introduction of pairwise terms that make use of image. For example, neighboring labels should be similar only if pixel colors are similar  Contrast term Discriminative approach I (pixels) Image Plane i j hihi hjhj e.g Kumar and Hebert 2003

13. I (pixels) Image Plane i j hihi hjhj Figure from Kumar et al., CVPR 2005 OBJCUT Ω (shape parameter) Kumar, Torr & Zisserman 2005 PairwiseUnary Ω is a shape prior on the labels from a Layered Pictorial Structure (LPS) model Segmentation by: - Match LPS model to image (get number of samples, each with a different pose -Marginalize over the samples using a single graph cut [Boykov & Jolly, 2001] Label smoothness Contrast Distance from Ω Color Likelihood

14. OBJCUT: Shape prior - Ω - Layered Pictorial Structures (LPS) Generative model Composition of parts + spatial layout Layer 2 Layer 1 Parts in Layer 2 can occlude parts in Layer 1 Spatial Layout (Pairwise Configuration) Kumar, et al. 2004, 2005

15. In the absence of a clear boundary between object and background SegmentationImage OBJCUT: Results Using LPS Model for Cow

16. Levin & Weiss [ECCV 2006] Segmentation alignment with image edges Resulting min-cut segmentation Consistency with fragments segmentation

17. [Lepetit et al. CVPR 2005] Decision forest classifier Features are differences of pixel intensities Classifier Winn and Shotton 2006 Layout Consistent Random Field

18. Layout consistency (8,3)(9,3)(7,3) (8,2)(9,2)(7,2) (8,4)(9,4)(7,4) Neighboring pixels (p,q) ? (p,q+1) (p,q) (p+1,q+1) (p-1,q+1) Layout consistent Winn and Shotton 2006

19. Layout Consistent Random Field Layout consistency Part detector Winn and Shotton 2006

20. Stability of part labelling Part color key

21. Object-Specific Figure-Ground Segregation Stella X. Yu and Jianbo Shi, 2002

22. Image parsing: Tu, Zhu and Yuille 2003

24. Segment out all the cars …. fused tree model for cars Unseen image Training images Segmented Cars Segmentation Trees Overview Multiscale Seg. Todorovic and Ahuja, CVPR 2006 Slide from T. Wu

25. LOCUS model Deformation field D Position & size T Class shape π Class edge sprite μ o,σ o Edge image e Image Object appearance λ 1 Background appearance λ 0 Mask m Shared between images Different for each image Kannan, Jojic and Frey 2004 Winn and Jojic, 2005

26. In this section: brief paper reviews Jigsaw approach: Borenstein & Ullman, 2001, 2002 Concurrent recognition and segmentation: Yu and Shi, 2002 Image parsing: Tu, Zhu & Yuille 2003 Interleaved segmentation: Liebe & Schiele, 2004, 2005 OBJCUT: Kumar, Torr, Zisserman 2005 LOCUS: Winn and Jojic, 2005 LayoutCRF: Winn and Shotton, 2006 Levin and Weiss, 2006 Todorovic and Ahuja, 2006

27. Summary Strength –Explains every pixel of the image –Useful for image editing, layering, etc. Issues –Invariance issues (especially) scale, view-point variations –Inference difficulties

30. Conditional Random Fields for Segmentation Segmentation map x Image I Low-level pairwise termHigh-level local term Pixel-wise similarity

31. Object-Specific Figure-Ground Segregation Some segmentation/detection results Yu and Shi, 2002

32. Multiscale Conditional Random Fields for Image Labeling Xuming He Richard S. Zemel Miguel A´. Carreira-Perpin˜a´n Conditional Random Fields for Object Recognition Ariadna Quattoni Michael Collins Trevor Darrell

33. OBJCUT Probability of labelling in addition has Unary potential which depend on distance from Θ (shape parameter) D (pixels) m (labels) Θ (shape parameter) Image Plane Object Category Specific MRF x y mxmx mymy Unary Potential Φ x (m x |Θ) Kumar, et al. 2004, 2005

34. Localization using features

35. Levin and Weiss 2006 Levin and Weiss, ECCV 2006

36. Results: horses

38. Cows: Results Segmentations from interest points Single-frame recognition - No temporal continuity used! Liebe and Schiele, 2003, 2005

40. Examples of low-level image segmentation Normalized Cuts, Shi & Malik, 1997 Borenstein & Ullman, ECCV 2002

42. Jigsaw approach Each patch has foreground/background mask

43. LayoutCRF

45. Interpretation of p(figure) map –per-pixel confidence in object hypothesis –Use for hypothesis verification p(figure) p(ground) Segmentation p(figure) p(ground) Original image Liebe and Schiele, 2003, 2005