1. Graph-based Segmentation
02/25/10
Graph-based Segmentation
Computer Vision
CS 543 / ECE 549
University of Illinois
Derek Hoiem

2. Last class Gestalt cues and principles of organization
Mean-shift segmentation
Good general-purpose segmentation method
Generally useful clustering, tracking technique
Watershed segmentation
Good for hierarchical segmentation
Use in combination with boundary prediction

3. Today’s class Treating the image as a graph Recap
Normalized cuts segmentation
MRFs Graph cuts segmentation
Recap
Go over HW2 instructions

4. Images as graphs Fully-connected graph node for every pixel
wij c j
Fully-connected graph
node for every pixel
link between every pair of pixels, p,q
similarity wij for each link
Source: Seitz

5. Similarity matrix
Increasing sigma

6. Segmentation by Graph Cutsw A B C
Break Graph into Segments
Delete links that cross between segments
Easiest to break links that have low cost (low similarity)
similar pixels should be in the same segments
dissimilar pixels should be in different segments
Source: Seitz

7. Cuts in a graph B A
Link Cut
set of links whose removal makes a graph disconnected
cost of a cut:
One idea: Find minimum cut
gives you a segmentation
fast algorithms exist for doing this
Source: Seitz

8. But min cut is not always the best cut...

9. Cuts in a graph a cut penalizes large segmentsB A
Normalized Cut
a cut penalizes large segments
fix by normalizing for size of segments
volume(A) = sum of costs of all edges that touch A
Source: Seitz

10. Recursive normalized cuts
Given an image or image sequence, set up a weighted graph: G=(V, E)
Vertex for each pixel
Edge weight for nearby pairs of pixels
Solve for eigenvectors with the smallest eigenvalues: (D − W)y = λDy
Use the eigenvector with the second smallest eigenvalue to bipartition the graph
Note: this is an approximation
4. Recursively repartition the segmented parts if necessary
Details:

11. Normalized cuts results

12. Normalized cuts: Pro and con
Pros
Generic framework, can be used with many different features and affinity formulations
Provides regular segments
Cons
Need to chose number of segments
High storage requirement and time complexity
Bias towards partitioning into equal segments
Usage
Use for oversegmentation when you want regular segments

13. Graph cuts segmentation

14. Markov Random Fields Node yi: pixel label Edge: constrained pairs
Cost to assign a label to each pixel
Cost to assign a pair of labels to connected pixels

15. Markov Random Fields Example: “label smoothing” grid Unary potential
0: -logP(yi = 0 ; data)
1: -logP(yi = 1 ; data)
Pairwise Potential
0 1 K
1 K 0

16. Solving MRFs with graph cuts
Source (Label 0)
Cost to assign to 0
Cost to split nodes
Cost to assign to 1
Sink (Label 1)

17. Solving MRFs with graph cuts
Source (Label 0)
Cost to assign to 0
Cost to split nodes
Cost to assign to 1
Sink (Label 1)

18. Grab cuts and graph cuts
Magic Wand (198?)
Intelligent Scissors Mortensen and Barrett (1995)
GrabCut
User Input
Result
Basic idea is to combine the information which was used in the 2 well known tools: MW and IS.
MS select a region of similar colour according to your input. Fat pen and the boundary snapps to high contrast edges.
Grabcut combines it. And this allows us to simplify the user interface considerably by …
Regions
Boundary
Regions & Boundary
Source: Rother

19. Colour Model R R G G Gaussian Mixture Model (typically 5-8 components)
Iterated graph cut
Foreground & Background
Foreground
Color enbergy.
Iterations have the effect of pulling them away;
D is – log likelyhood of the GMM.
Background G
Background G
Gaussian Mixture Model (typically 5-8 components)
Source: Rother

20. Graph cuts Boykov and Jolly (2001)
Foreground (source)
Image
Min Cut
Cut: separating source and sink; Energy: collection of edges
Min Cut: Global minimal enegry in polynomial time
Optimization engine we use Graph Cuts. By now everybody should know what graph cut is. IN case you missed it here is a very brief introduction.
To image. 3D view. First task to construct a graph.
All pixels on a certain scanline. Just a few of them.
Next step introduce artificial nodes. fgd and background.
Edges – contrast. Boundary is quite likely between black and white.
Min Cut - Minimum edge strength.
Background (sink)
Source: Rother

21. Graph cuts segmentation
Define graph
usually 4-connected or 8-connected
Define unary potentials
Color histogram or mixture of Gaussians for background and foreground
Define pairwise potentials
Apply graph cuts
Return to 2, using current labels to compute foreground, background models

22. Moderately straightforward examples
Moderately straightforward examples- after the user input automnatically
… GrabCut completes automatically
GrabCut – Interactive Foreground Extraction

23. Difficult Examples Camouflage & Low Contrast Fine structure
Harder Case
Initial Rectangle
Initial Result
You might wonder when does it fail. 4 cases. Low contrats – an edge not good visible
GrabCut – Interactive Foreground Extraction

24. Using graph cuts for recognition
TextonBoost (Shotton et al IJCV)

25. Using graph cuts for recognition
Unary Potentials
Alpha Expansion Graph Cuts
TextonBoost (Shotton et al IJCV)

26. Limits of graph cuts
Associative: edge potentials penalize different labels
If not associative, can sometimes clip potentials
Approximate for multilabel
Alpha-expansion or alpha-beta swaps
Must satisfy

27. Graph cuts: Pros and Cons
Very fast inference
Can incorporate recognition or high-level priors
Applies to a wide range of problems (stereo, image labeling, recognition)
Cons
Not always applicable (associative only)
Need unary terms (not used for generic segmentation)
Use whenever applicable

28. Further reading and resources
Normalized cuts and image segmentation (Shi and Malik)
N-cut implementation
Graph cuts
Classic paper: What Energy Functions can be Minimized via Graph Cuts? (Kolmogorov and Zabih, ECCV '02/PAMI '04)

29. Recap of Grouping and Fitting

30. Line detection and Hough transform
Canny edge detector = smooth  derivative  thin  threshold  link
Generalized Hough transform = points vote for shape parameters
Straight line detector = canny + gradient orientations  orientation binning  linking  check for straightness

31. Robust fitting and registration
Key algorithm
RANSAC

32. Clustering
Key algorithm
Kmeans

33. EM and Mixture of Gaussians
Tutorials:

34. Segmentation Mean-shift segmentation Watershed segmentation
Flexible clustering method, good segmentation
Watershed segmentation
Hierarchical segmentation from soft boundaries
Normalized cuts
Produces regular regions
Slow but good for oversegmentation
MRFs with Graph Cut
Incorporates foreground/background/object model and prefers to cut at image boundaries
Good for interactive segmentation or recognition

35. Next section: Recognition
How to recognize
Specific object instances
Faces
Scenes
Object categories
Materials