Presentation is loading. Please wait.

Presentation is loading. Please wait.

Shape Sharing for Object Segmentation Jaechul Kim and Kristen Grauman University of Texas at Austin.

Similar presentations

Presentation on theme: "Shape Sharing for Object Segmentation Jaechul Kim and Kristen Grauman University of Texas at Austin."— Presentation transcript:

1 Shape Sharing for Object Segmentation Jaechul Kim and Kristen Grauman University of Texas at Austin

2 Problem statement Category-independent object segmentation: Generate object segments in the image regardless of their categories.

3 Related work Top-down, category-specific approach e.g., Active Contours (IJCV 1987), Borenstein and Ullman (ECCV 2002), Levin and Weiss (ECCV 2006), Kumar (CVPR 2005) Use generic knowledge on object shapes e.g., Levinshtein (ICCV 2009) Category-independent multiple segmentations e.g., Malisiewicz and Efros (BMVC 2007), Carreira and Sminchisescu (CVPR 2010), Endres and Hoiem (ECCV 2010) Learn local shapes e.g., Ren (ECCV 2006), Opelt (CVPR 2006)

4 Spectrum of existing approaches Class-specificBottom-up + coherent mid-level regions + applicable to any image - prone to over/under-segment + robustness to low-level cues - typically viewpoint specific - requires class knowledge! How to model top-down shape in a category-independent way? horse shape priors color, textures, edges… Malisiewicz and Efros (BMVC 2007), Arbelaez et al. (CVPR 2009) Carreira and Sminchisescu (CVPR 2010) Endres and Hoiem (ECCV 2010) Active Contours (IJCV 1987) Borenstein and Ullman (ECCV 2002) Levin and Weiss (ECCV 2006) Kumar (CVPR 2005) e.g.,

5 Our goal Segment even unfamiliar objects with category-independent top-down cues Top-down segmentation with shape prior We don’t want to care what is in the image. Cow? Sheep?

6 Our idea: Shape sharing Semantically close Semantically disparate Object shapes are shared among different categories. Shapes from one class can be used to segment another (possibly unknown) class: Enable category-independent shape priors

7 Basis of approach: transfer through matching Transfer category-independent shape prior Exemplar image Test image Partial shape match Global shape projection ground truth object boundaries

8 1. Shape projection via local shape matches … Exemplars Test image Approach: Overview + Shape priorColor model Segmentation model per each group 2. Aggregating the shape projections Graph-cut Segmentation hypotheses 3. Multiple figure- ground segmentations with shape prior

9 Approach: Shape projection Test image Exemplars Projection AggregationSegmentation Vs. BPLRsSuperpixels Boundary-Preserving Local Regions (BPLR): Distinctively shaped Dense Repeatable [Kim & Grauman, CVPR 2011]

10 Approach: Shape projection Test image Exemplars … … Shape projections via similarity transform of BPLR matches Projection AggregationSegmentation Matched Exemplar 1 Matched Exemplar 2 Shape hypotheses

11 Approach: Refinement of projections Refined shape Initial projection Exemplar jigsaw Projection AggregationSegmentation Include superpixels where majority of pixels overlap projection Align with bottom-up evidence

12 Approach: Aggregating projections Projection Aggregation Segmentation … Exploit partial agreement from multiple exemplars Grouping based on overlap

13 + Shape priorColor model Segmentation model per each group Approach: Segmentation ProjectionAggregation Segmentation Figure-ground segmentation using graph-cut

14 n-links ProjectionAggregation Segmentation Approach: Graph-cut data term smoothness term Define a graph over image pixels: node = pixel edge = cost of a cut between pixels Energy function to minimize:

15 ProjectionAggregation Segmentation Approach: Segmentation Color likelihood Fg Bg NA Fg color histogram Bg color histogram Shape likelihood Graph-cut optimization Data term + Smoothness term

16 ProjectionAggregation Segmentation Approach: Multiple segmentations Parameter controlling data term bias Compute multiple segmentations by varying foreground bias: … Output: Carreira and Sminchisescu, CPMC: Automatic Object Segmentation Using Constrained Parametric Min-Cuts PAMI 2012.

17 Experiments Exemplar database: PASCAL 2010 segmentation task training set (20 classes, 2075 objects) Test datasets: PASCAL 2010 segmentation task validation set (20 classes, 964 images) Berkeley segmentation dataset (natural scenes and objects, 300 images) Baselines: CPMC [Carreira and Sminchisescu, PAMI 2012] Object proposals [Endres and Hoiem, ECCV 2012] gPb+owt+ucm [Arbelaez et al., PAMI 2011] Evaluation metric: Best covering score w.r.t # of segments … Ground truth0.920.750.71 Best covering score: 0.92

18 Segmentation quality ApproachCovering (%)Num of segments Shape sharing (Ours)84.31448 CPMC [Carreira and Sminchisescu]81.61759 Object proposals [Endres and Hoiem]81.71540 gPb-owt-ucm [Arbelaez et al.]62.81242 PASCAL 2010 dataset ApproachCovering (%)Num of segments Shape sharing (Ours)75.61449 CPMC [Carreira and Sminchisescu]74.11677 Object proposals [Endres and Hoiem]72.31275 gPb-owt-ucm [Arbelaez et al.]61.61483 Berkeley segmentation dataset *Exemplars = PASCAL

19 When does shape sharing help most? Gain as a function of color easiness and object size Easy to segment by color Hard to segment by color Compared to CPMC [Carreira and Sminchisescu., PAMI 2012]

20 Which classes share shapes? Animals Vehicles Semantically disparate Unexpected pose variations

21 Shape sharing (ours) CPMC (Carreira and Sminchisescu) 0.8890.8590.9030.935 0.5990.6380.630 0.694 Objects with diverse colors Example results (good)

22 Shape sharing (ours) CPMC (Carreira and Sminchisescu) Objects confused by surrounding colors Example results (good) 0.9660.875 0.999 0.928 0.5080.533 0.526 0.685

23 Shape sharing (ours) CPMC (Carreira and Sminchisescu) Example results (failure cases) 0.220 0.199 0.7130.406 0.818 0.934 0.973 0.799

24 Shape sharing: highlights Non-parametric transfer of shapes across categories Partial shape agreement from multiple exemplars Multiple hypothesis approach Most impact for heterogeneous objects Code is available: Top-down shape prior in a category-independent way

25 Approach: Refinement and pruning Initial projection Refined shape Exemplar jigsaw Pruned out Exemplar Projection AggregationSegmentation

26 Segmentation quality Quality of initial shape projections Initial shape prior Exemplar ApproachCovering (%)Num of segments Exemplar-based merge (Ours)77.0607 Neighbor merge [1]72.25005 Bottom-up segmentation [2]62.81242 [ 1] Malisiewicz and efros, BMVC 2007. [2] Arbelaez, PAMI 2011.

27 Impact of shapes CPMC, PASCALCPMC, BSDObject proposal, PASCALObject proposal, BSD Shape sharing’s gain in recall as a function of overlap

28 Category-independent vs. dependent ApproachCovering (%) Category-specific84.7 Category-independent (default)84.3 Strictly category-independent83.9 CPMC81.6 Object proposals81.7 Comparison of category-independent shape prior and category-specific variants

Download ppt "Shape Sharing for Object Segmentation Jaechul Kim and Kristen Grauman University of Texas at Austin."

Similar presentations

Ads by Google