1. Machine Learning – Lecture 15
Efficient MRF Inference with Graph Cuts
Bastian Leibe
RWTH Aachen
TexPoint fonts used in EMF.
Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAAAAAAAAAAAAAAAAA

2. Announcements Last lecture next Tuesday: Repetition
Summary of all topics in the lecture
“Big picture” and current research directions
Opportunity to ask questions
Please use this opportunity and prepare questions!
B. Leibe

3. Course Outline Fundamentals (2 weeks)
Bayes Decision Theory
Probability Density Estimation
Discriminative Approaches (5 weeks)
Linear Discriminant Functions
Statistical Learning Theory & SVMs
Ensemble Methods & Boosting
Decision Trees & Randomized Trees
Generative Models (4 weeks)
Bayesian Networks
Markov Random Fields
Exact Inference
B. Leibe

4. Recap: Junction Tree Algorithm
Motivation
Exact inference on general graphs.
Works by turning the initial graph into a junction tree and then running a sum-product-like algorithm.
Intractable on graphs with large cliques.
Main steps
If starting from directed graph, first convert it to an undirected graph by moralization.
Introduce additional links by triangulation in order to reduce the size of cycles.
Find cliques of the moralized, triangulated graph.
Construct a new graph from the maximal cliques.
Remove minimal links to break cycles and get a junction tree.
 Apply regular message passing to perform inference.
B. Leibe

5. Recap: Junction Tree Example
Without triangulation step
The final graph will contain cycles that we cannot break without losing the running intersection property!
B. Leibe
Image source: J. Pearl, 1988

6. Recap: Junction Tree Example
When applying the triangulation
Only small cycles remain that are easy to break.
Running intersection property is maintained.
B. Leibe
Image source: J. Pearl, 1988

7. Recap: MRF Structure for Images
Basic structure
Two components
Observation model
How likely is it that node xi has label Li given observation yi?
This relationship is usually learned from training data.
Neighborhood relations
Simplest case: 4-neighborhood
Serve as smoothing terms.
 Discourage neighboring pixels to have different labels.
This can either be learned or be set to fixed “penalties”.
Noisy observations
“True” image content
B. Leibe

8. Recap: How to Set the Potentials?
Unary potentials
E.g. color model, modeled with a Mixture of Gaussians
 Learn color distributions for each label
B. Leibe

9. Recap: How to Set the Potentials?
Pairwise potentials
Potts Model
Simplest discontinuity preserving model.
Discontinuities between any pair of labels are penalized equally.
Useful when labels are unordered or number of labels is small.
Extension: “contrast sensitive Potts model” where
Discourages label changes except in places where there is also a large change in the observations.
B. Leibe

10. Energy Minimization Goal:
Infer the optimal labeling of the MRF.
Many inference algorithms are available, e.g.
Simulated annealing
Iterated conditional modes (ICM)
Belief propagation
Graph cuts
Variational methods
Monte Carlo sampling
Recently, Graph Cuts have become a popular tool
Only suitable for a certain class of energy functions.
But the solution can be obtained very fast for typical vision problems (~1MPixel/sec).
What you saw in the movie.
Too simple.
Last lecture
Today!
For more complex problems
B. Leibe

11. Topics of This Lecture Solving MRFs with Graph Cuts
Graph cuts for image segmentation
s-t mincut algorithm
Extension to non-binary case
Applications
B. Leibe

12. Graph Cuts for Binary Problems
Idea: convert MRF into source-sink graph s t
a cut
n-links
hard
constraint
Minimum cost cut can be computed in polynomial time
(max-flow/min-cut algorithms)
Slide credit: Yuri Boykov
B. Leibe
[Boykov & Jolly, ICCV’01]

13. Simple Example of Energy
unary potentials
pairwise potentials
t-links
n-links s t
a cut
(binary object segmentation)
Slide credit: Yuri Boykov
B. Leibe

14. Adding Regional Properties
a cut
n-links
t-link
t-link
Regional bias example
Suppose are given
“expected” intensities
of object and background
NOTE: hard constrains are not required, in general.
Slide credit: Yuri Boykov
B. Leibe
[Boykov & Jolly, ICCV’01]

15. Adding Regional Properties
a cut
n-links
t-link
t-link
“expected” intensities of
object and background
can be re-estimated
EM-style optimization
Slide credit: Yuri Boykov
B. Leibe
[Boykov & Jolly, ICCV’01]

16. Adding Regional Properties
More generally, unary potentials can be based on any intensity/color models of object and background.
a cut s t
Object and background color distributions
Slide credit: Yuri Boykov
B. Leibe
[Boykov & Jolly, ICCV’01]

17. Topics of This Lecture Solving MRFs with Graph Cuts
Graph cuts for image segmentation
s-t mincut algorithm
Extension to non-binary case
Applications
Source
Sink v1 v2 2 5 9 4 1
B. Leibe

18. How Does it Work? The s-t-Mincut Problem
Source
Sink v1 v2 2 5 9 4 1
Graph (V, E, C)
Vertices V = {v1, v2 ... vn}
Edges E = {(v1, v2) ....}
Costs C = {c(1, 2) ....}
Slide credit: Pushmeet Kohli
B. Leibe

19. The s-t-Mincut Problem
What is an st-cut?
An st-cut (S,T) divides the nodes between source and sink.
Source 2 9
What is the cost of a st-cut? 1
Sum of cost of all edges going from S to T v1 v2 2 5 4
Sink
= 16
Slide credit: Pushmeet Kohli
B. Leibe

20. The s-t-Mincut Problem
What is an st-cut?
An st-cut (S,T) divides the nodes between source and sink.
Source 2 9
What is the cost of a st-cut? 1
Sum of cost of all edges going from S to T v1 v2 2 5 4
What is the st-mincut?
Sink
st-cut with the minimum cost
= 7
Slide credit: Pushmeet Kohli
B. Leibe

21. How to Compute the s-t-Mincut?
Solve the dual maximum flow problem
Compute the maximum flow between Source and Sink
Source
Sink v1 v2 2 5 9 4 1
Constraints
Edges: Flow < Capacity
Nodes: Flow in = Flow out
In every network, the maximum flow equals the cost of the st-mincut
Min-cut/Max-flow Theorem
Slide credit: Pushmeet Kohli
B. Leibe

22. History of Maxflow Algorithms
Augmenting Path and Push-Relabel
n: #nodes
m: #edges
U: maximum edge weight
Algorithms assume non-negative edge weights
Slide credit: Andrew Goldberg
B. Leibe

23. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 0
Augmenting Path Based Algorithms
Source
Sink v1 v2 2 5 9 4 1
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

24. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 0
Augmenting Path Based Algorithms
Source
Sink v1 v2 9 4 2 1 2
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found 5
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

25. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 0 + 2
Augmenting Path Based Algorithms
Source
Sink v1 v2 9 4 2 1
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found
2-2
5-2
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

26. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 2
Augmenting Path Based Algorithms
Source
Sink v1 v2 9 4 2 1
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found 3
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

27. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 2
Augmenting Path Based Algorithms
Source
Sink v1 v2 3 9 4 2 1
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

28. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 2
Augmenting Path Based Algorithms
Source
Sink v1 v2 3 2 1
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found 9 4
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

29. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 2 + 4
Augmenting Path Based Algorithms
Source
Sink v1 v2 3 2 1
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found 5
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

30. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 6
Augmenting Path Based Algorithms
Source
Sink v1 v2 3 5 2 1
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

31. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 6
Augmenting Path Based Algorithms
Source
Sink v1 v2 2
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found 5 1 3
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

32. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 6 + 1
Augmenting Path Based Algorithms
Source
Sink v1 v2 2
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found 4
1-1 2
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

33. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 7
Augmenting Path Based Algorithms
Source
Sink v1 v2 2 4
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

34. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 7
Augmenting Path Based Algorithms
Source
Sink v1 v2 2 4
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

35. Applications: Maxflow in Computer Vision
Specialized algorithms for vision problems
Grid graphs
Low connectivity (m ~ O(n))
Dual search tree augmenting path algorithm
[Boykov and Kolmogorov PAMI 2004]
Finds approximate shortest augmenting paths efficiently.
High worst-case time complexity.
Empirically outperforms other algorithms on vision problems.
Efficient code available on the web
Slide credit: Pushmeet Kohli
B. Leibe

36. When Can s-t Graph Cuts Be Applied?
s-t graph cuts can only globally minimize binary energies that are submodular.
Submodularity is the discrete equivalent to convexity.
Implies that every local energy minimum is a global minimum.
 Solution will be globally optimal.
unary potentials
pairwise potentials
t-links
n-links
[Boros & Hummer, 2002, Kolmogorov & Zabih, 2004]
E(L) can be minimized by s-t graph cuts
Submodularity (“convexity”)
B. Leibe

37. Recap: Simple Binary Image Denoising Model
MRF Structure
Example: simple energy function
Smoothness term: fixed penalty ¯ if neighboring labels disagree.
Observation term: fixed penalty ´ if label and observation disagree.
Noisy observations
Observation process
“True” image content
“Smoothness constraints”
Prior
Smoothness
("Potts model")
Observation
B. Leibe
Image source: C. Bishop, 2006

38. Converting an MRF into an s-t Graph
Conversion:
Energy:
Unary potentials are straightforward to set.
How?
Just insert xi = 1 and xi = -1 into the unary terms above...
Source
Sink xi xj
B. Leibe

39. Converting an MRF into an s-t Graph
Conversion:
Energy:
Unary potentials are straightforward to set. How?
Pairwise potentials are more tricky, since we don’t know xi!
Trick: the pairwise energy only has an influence if xi  xj.
(Only!) in this case, the cut will go through the edge {xi,xj}.
Source
Sink xi xj
B. Leibe

40. Topics of This Lecture Solving MRFs with Graph Cuts
Graph cuts for image segmentation
s-t mincut algorithm
Extension to non-binary case
Applications
other labels 
B. Leibe

41. Dealing with Non-Binary Cases
Limitation to binary energies is often a nuisance.
 E.g. binary segmentation only…
We would like to solve also multi-label problems.
The bad news: Problem is NP-hard with 3 or more labels!
There exist some approximation algorithms which extend graph cuts to the multi-label case:
-Expansion
-Swap
They are no longer guaranteed to return the globally optimal result.
But -Expansion has a guaranteed approximation quality (2-approx) and converges in a few iterations.
B. Leibe

42. -Expansion Move Basic idea:
Break multi-way cut computation into a sequence of binary s-t cuts.
other labels 
Slide credit: Yuri Boykov
B. Leibe

43. -Expansion Algorithm
Start with any initial solution
For each label “” in any (e.g. random) order:
Compute optimal -expansion move (s-t graph cuts).
Decline the move if there is no energy decrease.
Stop when no expansion move would decrease energy.
Slide credit: Yuri Boykov
B. Leibe

44. Example: Stereo Vision
Depth map
ground truth
Depth map
Original pair of “stereo” images
Slide credit: Yuri Boykov
B. Leibe

45. -Expansion Moves
In each -expansion a given label “” grabs space from other labels
initial solution
-expansion
-expansion
-expansion
-expansion
-expansion
-expansion
-expansion
For each move, we choose the expansion that gives the largest decrease in the energy:  binary optimization problem
Slide credit: Yuri Boykov
B. Leibe

46. Topics of This Lecture Solving MRFs with Graph Cuts
Graph cuts for image segmentation
s-t mincut algorithm
Extension to non-binary case
Applications
B. Leibe

47. GraphCut Applications: “GrabCut”
Interactive Image Segmentation [Boykov & Jolly, ICCV’01]
Rough region cues sufficient
Segmentation boundary can be extracted from edges
Procedure
User marks foreground and background regions with a brush.
This is used to create an initial segmentation which can then be corrected by additional brush strokes.
Add interactive image segmentation slide
Additional
segmentation
cues
User segmentation cues
Slide credit: Matthieu Bray

48. Global optimum of the energy
GrabCut: Data Model
Obtained from interactive user input
User marks foreground and background regions with a brush
Alternatively, user can specify a bounding box
Foreground color
Background color
Global optimum of the energy
Slide credit: Carsten Rother
B. Leibe

49. Color model (Mixture of Gaussians) Energy after each iteration
Iterated Graph Cuts
Foreground
Background G R
Foreground & Background
Background G R
Color model (Mixture of Gaussians) 1 2 3 4
Result
Energy after each iteration
Slide credit: Carsten Rother
B. Leibe

50. GrabCut: Example Results
This is included in the newest version of MS Office!
B. Leibe
Image source: Carsten Rother

51. Applications: Interactive 3D Segmentation
Slide credit: Yuri Boykov
B. Leibe
[Y. Boykov, V. Kolmogorov, ICCV’03]

52. Bayesian Networks What we’ve learned so far…
We know they are directed graphical models.
Their joint probability factorizes into conditional probabilities,
We know how to convert them into undirected graphs.
We know how to perform inference for them.
Sum/Max-Product BP for exact inference in (poly)tree-shaped BNs.
Loopy BP for approximate inference in arbitrary BNs.
Junction Tree algorithm for converting arbitrary BNs into trees.
But what are they actually good for?
How do we apply them in practice?
And how do we learn their parameters?
B. Leibe
Image source: C. Bishop, 2006

53. References and Further Reading
A gentle introduction to Graph Cuts can be found in the following paper:
Y. Boykov, O. Veksler, Graph Cuts in Vision and Graphics: Theories and Applications. In Handbook of Mathematical Models in Computer Vision, edited by N. Paragios, Y. Chen and O. Faugeras, Springer, 2006.
Try the GraphCut implementation at
B. Leibe