1. Shape Sharing for Object Segmentation
Jaechul Kim and Kristen Grauman
University of Texas at Austin

2. Problem statement Category-independent object segmentation:
Generate object segments in the image
regardless of their categories.

3. Related work Top-down, category-specific approach
e.g., Active Contours (IJCV 1987), Borenstein and Ullman (ECCV 2002), Levin and Weiss (ECCV 2006), Kumar et.al. (CVPR 2005)
Use generic knowledge on object shapes
e.g., Levinshtein et.al. (ICCV 2009)
Learn local shapes
e.g., Ren et.al. (ECCV 2006), Opelt et.al. (CVPR 2006)
Category-independent multiple segmentations
e.g., Malisiewicz and Efros (BMVC 2007), Carreira and Sminchisescu (CVPR 2010), Endres and Hoiem (ECCV 2010)

4. Spectrum of existing approaches
horse shape priors
color, textures, edges…
How to model
top-down shape in a category-independent
way?
Bottom-up
Class-specific
+ coherent mid-level regions
+ applicable to any image
- prone to over/under-segment
+ robustness to low-level cues
- typically viewpoint specific
- requires class knowledge!
e.g.,
e.g.,
Malisiewicz and Efros (BMVC 2007),
Arbelaez et al. (CVPR 2009)
Carreira and Sminchisescu (CVPR 2010)
Endres and Hoiem (ECCV 2010)
Active Contours (IJCV 1987)
Borenstein and Ullman (ECCV 2002)
Levin and Weiss (ECCV 2006)
Kumar et.al. (CVPR 2005)

5. Our goal
Segment even unfamiliar objects with category-independent top-down cues
Cow? Sheep?
Top-down segmentation with shape prior
We don’t want to care what is in the image.

6. Our idea: Shape sharing
Semantically close
Semantically disparate
Object shapes are shared among different categories.
Shapes from one class can be used to segment another (possibly unknown) class:
Enable category-independent shape priors

7. Basis of approach: transfer through matching
Exemplar image
Test image
ground truth object boundaries
Partial shape match
Global shape projection
Transfer category-independent shape prior

8. … + Approach: Overview 1. Shape projection via local shape matches
Exemplars
Test image +
Shape prior
Color model
Segmentation model per each group
2. Aggregating the shape projections
Graph-cut
Segmentation hypotheses
3. Multiple figure-ground segmentations with shape prior

9. Approach: Shape projection
Aggregation
Segmentation
Test image
Exemplars
Vs.
BPLRs
Superpixels
Boundary-Preserving Local Regions (BPLR):
Distinctively shaped
Dense
Repeatable
[Kim & Grauman, CVPR 2011]

10. Approach: Shape projection
Aggregation
Segmentation
Test image
Exemplars …
Shape projections via
similarity transform of BPLR matches
Matched
Exemplar 1 …
Matched
Exemplar 2
Shape hypotheses

11. Approach: Refinement of projections
Aggregation
Segmentation
Exemplar
jigsaw
Initial projection
Refined shape
Align with bottom-up evidence
Include superpixels where majority of pixels overlap projection

12. Approach: Aggregating projections
Aggregation
Segmentation …
Grouping based on overlap
Exploit partial agreement from multiple exemplars

13. + Approach: Segmentation Figure-ground segmentation using graph-cut
Projection
Approach: Segmentation
Aggregation
Segmentation +
Shape prior
Color model
Segmentation model per each group
Figure-ground segmentation using graph-cut

14. Approach: Graph-cut n-links Define a graph over image pixels:
Projection
Approach: Graph-cut
Aggregation
Segmentation
n-links
Define a graph over image pixels:
node = pixel
edge = cost of a cut between pixels
Energy function to minimize:
data term
smoothness term

15. + Approach: Segmentation Bg color histogram Bg NA Fg color histogram
Projection
Approach: Segmentation
Aggregation
Segmentation
Bg color histogram Bg NA
Fg color histogram Fg +
Shape likelihood
Color likelihood
Data term
Smoothness term
Graph-cut optimization

16. … Approach: Multiple segmentations
Projection
Approach: Multiple segmentations
Aggregation
Segmentation
Compute multiple segmentations by varying foreground bias:
Parameter controlling
data term bias
Output: …
Carreira and Sminchisescu,
CPMC: Automatic Object Segmentation Using Constrained Parametric Min-Cuts PAMI 2012.

17. Experiments … Exemplar database:
PASCAL 2010 segmentation task training set (20 classes, 2075 objects)
Test datasets:
PASCAL 2010 segmentation task validation set (20 classes, 964 images)
Berkeley segmentation dataset (natural scenes and objects, 300 images)
Evaluation metric:
Best covering score w.r.t # of segments
Baselines:
CPMC
[Carreira and Sminchisescu, PAMI 2012]
Object proposals
[Endres and Hoiem, ECCV 2012]
gPb+owt+ucm
[Arbelaez et al., PAMI 2011] …
Ground truth
0.92
0.75
0.71
Best covering score: 0.92

18. Segmentation quality Approach Covering (%) Num of segments
Shape sharing (Ours)
84.3
1448
CPMC [Carreira and Sminchisescu]
81.6
1759
Object proposals [Endres and Hoiem]
81.7
1540
gPb-owt-ucm [Arbelaez et al.]
62.8
1242
PASCAL 2010 dataset
Approach
Covering (%)
Num of segments
Shape sharing (Ours)
75.6
1449
CPMC [Carreira and Sminchisescu]
74.1
1677
Object proposals [Endres and Hoiem]
72.3
1275
gPb-owt-ucm [Arbelaez et al.]
61.6
1483
Berkeley segmentation dataset
*Exemplars = PASCAL

19. When does shape sharing help most?
Gain as a function of color easiness and object size
Easy to segment by color
Hard to segment by color
Compared to CPMC [Carreira and Sminchisescu., PAMI 2012]

20. Which classes share shapes?
Semantically disparate
Animals
Unexpected
pose variations
Vehicles

21. Objects with diverse colors
Example results (good)
Shape sharing (ours)
0.889
0.859
0.903
0.935
CPMC (Carreira and Sminchisescu)
0.599
0.638
0.630
0.694
Objects with diverse colors

22. Objects confused by surrounding colors
Example results (good)
Shape sharing (ours)
0.966
0.875
0.999
0.928
CPMC (Carreira and Sminchisescu)
0.508
0.533
0.526
0.685
Objects confused by surrounding colors

23. Example results (failure cases)
Shape sharing (ours)
0.220
0.199
0.713
0.406
CPMC (Carreira and Sminchisescu)
0.818
0.934
0.973
0.799

24. Shape sharing: highlights
Top-down shape prior in a category-independent way
Non-parametric transfer of shapes across categories
Partial shape agreement from multiple exemplars
Multiple hypothesis approach
Most impact for heterogeneous objects
Code is available:

25. Approach: Refinement and pruning
Projection
Approach: Refinement and pruning
Aggregation
Segmentation
Exemplar
jigsaw
Initial projection
Refined shape
Pruned out
Exemplar

26. Segmentation quality Quality of initial shape projections Approach
Covering (%)
Num of segments
Exemplar-based merge (Ours)
77.0
607
Neighbor merge [1]
72.2
5005
Bottom-up segmentation [2]
62.8
1242
[1] Malisiewicz and efros, BMVC 2007.
[2] Arbelaez et.al., PAMI 2011.
Initial shape prior
Exemplar

27. Object proposal, PASCAL
Impact of shapes
Shape sharing’s gain in recall as a function of overlap
CPMC, PASCAL
CPMC, BSD
Object proposal, PASCAL
Object proposal, BSD

28. Category-independent vs. dependent
Approach
Covering (%)
Category-specific
84.7
Category-independent (default)
84.3
Strictly category-independent
83.9
CPMC
81.6
Object proposals
81.7
Comparison of category-independent shape prior and category-specific variants