Part 4: Combined segmentation and recognition by Rob Fergus (MIT)

Aim Given an image and object category, to segment the object Segmentation should (ideally) be shaped like the object e.g. cow-like obtained efficiently in an unsupervised manner able to handle self-occlusion Segmentation Object Category Model Cow Image Segmented Cow Slide from Kumar ‘05

Feature-detector view

Examples of bottom-up segmentation Using Normalized Cuts, Shi & Malik, 1997 Borenstein and Ullman, ECCV 2002

Jigsaw approach: Borenstein and Ullman, 2002

Perceptual and Sensory Augmented Computing Interleaved Object Categorization and Segmentation Implicit Shape Model - Liebe and Schiele, 2003 Backprojected Hypotheses Interest Points Matched Codebook Entries Probabilistic Voting Voting Space (continuous) Backprojection of Maxima Segmentation Refined Hypotheses (uniform sampling) Liebe and Schiele, 2003, 2005

Random Fields for segmentation I = Image pixels (observed) h = foreground/background labels (hidden) – one label per pixel  = Parameters Prior LikelihoodPosteriorJoint 1.Generative approach models joint  Markov random field (MRF) 2. Discriminative approach models posterior directly  Conditional random field (CRF)

I (pixels) Image Plane i j h (labels)  {foreground, background} hihi hjhj Unary Potential  i ( I |h i,  i ) Pairwise Potential (MRF)  ij (h i, h j |  ij ) MRF PriorLikelihood Generative Markov Random Field Prior has no dependency on I

Conditional Random Field Lafferty, McCallum and Pereira 2001 PairwiseUnary Dependency on I allows introduction of pairwise terms that make use of image. For example, neighboring labels should be similar only if pixel colors are similar  Contrast term Discriminative approach I (pixels) Image Plane i j hihi hjhj e.g Kumar and Hebert 2003

I (pixels) Image Plane i j hihi hjhj Figure from Kumar et al., CVPR 2005 OBJCUT Ω (shape parameter) Kumar, Torr & Zisserman 2005 PairwiseUnary Ω is a shape prior on the labels from a Layered Pictorial Structure (LPS) model Segmentation by: - Match LPS model to image (get number of samples, each with a different pose -Marginalize over the samples using a single graph cut [Boykov & Jolly, 2001] Label smoothness Contrast Distance from Ω Color Likelihood

OBJCUT: Shape prior - Ω - Layered Pictorial Structures (LPS) Generative model Composition of parts + spatial layout Layer 2 Layer 1 Parts in Layer 2 can occlude parts in Layer 1 Spatial Layout (Pairwise Configuration) Kumar, et al. 2004, 2005

In the absence of a clear boundary between object and background SegmentationImage OBJCUT: Results Using LPS Model for Cow

Levin & Weiss [ECCV 2006] Segmentation alignment with image edges Resulting min-cut segmentation Consistency with fragments segmentation

[Lepetit et al. CVPR 2005] Decision forest classifier Features are differences of pixel intensities Classifier Winn and Shotton 2006 Layout Consistent Random Field

Layout consistency (8,3)(9,3)(7,3) (8,2)(9,2)(7,2) (8,4)(9,4)(7,4) Neighboring pixels (p,q) ? (p,q+1) (p,q) (p+1,q+1) (p-1,q+1) Layout consistent Winn and Shotton 2006

Layout Consistent Random Field Layout consistency Part detector Winn and Shotton 2006

Stability of part labelling Part color key

Object-Specific Figure-Ground Segregation Stella X. Yu and Jianbo Shi, 2002

Image parsing: Tu, Zhu and Yuille 2003

Segment out all the cars …. fused tree model for cars Unseen image Training images Segmented Cars Segmentation Trees Overview Multiscale Seg. Todorovic and Ahuja, CVPR 2006 Slide from T. Wu

LOCUS model Deformation field D Position & size T Class shape π Class edge sprite μ o,σ o Edge image e Image Object appearance λ 1 Background appearance λ 0 Mask m Shared between images Different for each image Kannan, Jojic and Frey 2004 Winn and Jojic, 2005

In this section: brief paper reviews Jigsaw approach: Borenstein & Ullman, 2001, 2002 Concurrent recognition and segmentation: Yu and Shi, 2002 Image parsing: Tu, Zhu & Yuille 2003 Interleaved segmentation: Liebe & Schiele, 2004, 2005 OBJCUT: Kumar, Torr, Zisserman 2005 LOCUS: Winn and Jojic, 2005 LayoutCRF: Winn and Shotton, 2006 Levin and Weiss, 2006 Todorovic and Ahuja, 2006

Summary Strength –Explains every pixel of the image –Useful for image editing, layering, etc. Issues –Invariance issues (especially) scale, view-point variations –Inference difficulties

Conditional Random Fields for Segmentation Segmentation map x Image I Low-level pairwise termHigh-level local term Pixel-wise similarity

Object-Specific Figure-Ground Segregation Some segmentation/detection results Yu and Shi, 2002

Multiscale Conditional Random Fields for Image Labeling Xuming He Richard S. Zemel Miguel A´. Carreira-Perpin˜a´n Conditional Random Fields for Object Recognition Ariadna Quattoni Michael Collins Trevor Darrell

OBJCUT Probability of labelling in addition has Unary potential which depend on distance from Θ (shape parameter) D (pixels) m (labels) Θ (shape parameter) Image Plane Object Category Specific MRF x y mxmx mymy Unary Potential Φ x (m x |Θ) Kumar, et al. 2004, 2005

Localization using features

Levin and Weiss 2006 Levin and Weiss, ECCV 2006

Results: horses

Cows: Results Segmentations from interest points Single-frame recognition - No temporal continuity used! Liebe and Schiele, 2003, 2005

Examples of low-level image segmentation Normalized Cuts, Shi & Malik, 1997 Borenstein & Ullman, ECCV 2002

Jigsaw approach Each patch has foreground/background mask

LayoutCRF

Interpretation of p(figure) map –per-pixel confidence in object hypothesis –Use for hypothesis verification p(figure) p(ground) Segmentation p(figure) p(ground) Original image Liebe and Schiele, 2003, 2005