1. Learning to Combine Bottom-Up and Top-Down Segmentation
Anat Levin and Yair Weiss
School of CS&Eng,
The Hebrew University of Jerusalem, Israel

2. Bottom-up segmentation
Bottom-up approaches: Use low level cues to group similar pixels
Malik et al, 2000
Sharon et al, 2001
Comaniciu and Meer, 2002 …

3. Bottom-up segmentation is ill posed
Many possible segmentation are equally good based on low level cues alone.
Some segmentation example (maybe horses from Eran’s paper)
images from Borenstein and Ullman 02

4. Top-down segmentation
Class-specific, top-down segmentation (Borenstein & Ullman Eccv02)
Winn and Jojic 05
Leibe et al 04
Yuille and Hallinan 02.
Liu and Sclaroff 01
Yu and Shi 03

5. Combining top-down and bottom-up segmentation+
Find a segmentation:
Similar to the top-down model
Aligns with image edges

6. Previous approaches
Borenstein et al 04 Combining top-down and bottom up segmentation.
Tu et al ICCV03 Image parsing: segmentation, detection, and recognition.
Kumar et al CVPR05 Obj-Cut.
Shotton et al ECCV06: TextonBoost
Previous approaches: Train top-down and bottom-up models independently

7. Why learning top-down and bottom-up models simultaneously?
Large number of freedom degrees in tentacles configuration- requires a complex deformable top down model
On the other hand: rather uniform colors- low level segmentation is easy

8. Our approach Learn top-down and bottom-up models simultaneously
Reduces at run time to energy minimization with binary labels (graph min cut)

9. Segmentation alignment with image edges
Energy model
Segmentation alignment with image edges
Consistency with fragments segmentation

10. Segmentation alignment with image edges
Energy model
Segmentation alignment with image edges
Consistency with fragments segmentation

11. Segmentation alignment with image edges
Energy model
Segmentation alignment with image edges
Consistency with fragments segmentation
Resulting min-cut segmentation

12. Learning from segmented class images
Training data:
Goal: Learn fragments for an energy function

13. Learning energy functions using conditional random fields
Theory of CRFs:
Lafferty et al 2001
LeCun and Huang 2005
CRFs For vision:
Kumar and Hebert 2003
Ren et al 2006
He et al 2004, 2006
Quattoni et al 2005
Torralba et al 04

14. Learning energy functions using conditional random fields
Maximize energy of all other configurations
Minimize energy of true segmentation
E(x)
“It's not enough to succeed. Others must fail.” –Gore Vidal

15. Learning energy functions using conditional random fields
Maximize energy of all other configurations
Minimize energy of true segmentation
P(x)
P(x)
“It's not enough to succeed. Others must fail.” –Gore Vidal

16. Differentiating CRFs log-likelihood
Log-likelihood is convex with respect to
Log-likelihood gradients with respect to :
Expected feature response minus observed feature response
Yair- in the original version of this slide I had another equation expressing the expectation as a sum of marginals (see next hidden slide). At least for me, it wasn’t originally clear what this expectation means before I saw the other equation. However, I try to delete un necessary equations..

17. Conditional random fields-computational challenges
CRFs cost- evaluating partition function
Derivatives- evaluating marginal probabilities
Use approximate estimations:
Sampling
Belief Propagation and Bethe free energy
Used in this work: Tree reweighted belief propagation and Tree reweighted upper bound (Wainwright et al 03)

18. Fragments selection
Candidate fragments pool:
Greedy energy design:

19. Fragments selection challenges
Straightforward computation of likelihood improvement is impractical
2000 Fragments
50 Training images
10 Fragments selection iterations
1,000,000 inference operations!

20. Fragment with low error on the training set
Fragments selection
First order approximation to log-likelihood gain:
Fragment with low error on the training set
Fragment not accounted for by the existing model
Similar idea in different contexts:
Zhu et al 1997
Lafferty et al 2004
McCallum 2003

21. Fragments selection First order approximation to log-likelihood gain:
Requires a single inference process on the previous iteration energy to evaluate approximations with respect to all fragments
First order approximation evaluation is linear in the fragment size

22. Fragments selection- summary
Initialization: Low- level term
For k=1:K
Run TRBP inference using the previous iteration energy.
Approximate likelihood gain of candidate fragments
Add to energy the fragment with maximal gain.

23. Training horses model

24. Training horses model-one fragment

25. Training horses model-two fragments

26. Training horses model-three fragments

27. Results- horses dataset

28. Results- horses dataset
Mislabeled pixels percent
Fragments number
Comparable to previous results (Kumar et al, Borenstein et al.) but with far fewer fragments

29. Results- artificial octopi

30. From the TU Darmstadt Database
Results- cows dataset
From the TU Darmstadt Database

31. Results- cows dataset
Mislabeled pixels percent
Fragments number

32. Conclusions
Simultaneously learning top-down and bottom-up segmentation cues.
Learning formulated as estimation in Conditional Random Fields
Novel, efficient fragments selection algorithm
Algorithm achieves state of the art performance with a significantly smaller number of fragments