Relaxations and Moves for MAP Estimation in MRFs M. Pawan Kumar STANFORDSTANFORD Vladimir KolmogorovPhilip TorrDaphne Koller

Our Problem v1v1 v2v2 v3v3 v4v Label l 1 Label l 2 Random Variables V = {v 1,...,v 4 } Label Set L = {l 1, l 2 } Labeling f: V  L (shown in red)

Our Problem v1v1 v2v2 v3v3 v4v Label l 1 Label l 2 Random Variables V = {v 1,...,v 4 } Label Set L = {l 1, l 2 } Labeling f: V  L (shown in red) Energy of Labeling E(f) = 13 (shown in green)

Our Problem v1v1 v2v2 v3v3 v4v Label l 1 Label l 2 Find f* = argmin f E(f) Arbitrary topology, discrete label set, potentials (NP-hard) Pairwise energy function: unary and pairwise potentials (still NP-hard)

Outline Convex Relaxations – Integer Programming Formulation – LP Relaxation – SDP Relaxation – SOCP Relaxation – Comparing Relaxations Move Making Algorithms Some Interesting Open Problems

Integer Programming Formulation v1v1 v2v Unary Potentials Unary Potential u = [ 5 Cost of v 1 = 1 2 Cost of v 1 = 2 ; 2 4 ] Labeling f shown in red Label l 1 Label l 2

Label vector x = [ -1 v 1  1 1 v 1 = 2 ; 1 -1 ] T Recall that the aim is to find the optimal x Integer Programming Formulation v1v1 v2v Unary Potentials Labeling f shown in red Label l 1 Label l 2 Unary Potential u = [ 5 2 ; 2 4 ]

Label vector x = [ -11; 1 -1 ] T Sum of Unary Potentials = 1 2 ∑ i u i (1 + x i ) Integer Programming Formulation v1v1 v2v Unary Potentials Labeling f shown in red Label l 1 Label l 2 Unary Potential u = [ 5 2 ; 2 4 ]

0 Cost of v 1 = 1 and v 1 = Cost of v 1 = 1 and v 2 = 1 3 Cost of v 1 = 1 and v 2 = Pairwise Potential P Integer Programming Formulation v1v1 v2v Pairwise Potentials Labeling f shown in red Label l 1 Label l 2

Pairwise Potential P Sum of Pairwise Potentials 1 4 ∑ ij P ij (1 + x i )(1+x j ) Integer Programming Formulation v1v1 v2v Pairwise Potentials Labeling f shown in red Label l 1 Label l 2

Sum of Pairwise Potentials 1 4 ∑ ij P ij (1 + x i +x j + x i x j ) 1 4 ∑ ij P ij (1 + x i + x j + X ij )= X = x x T X ij = x i x j Integer Programming Formulation v1v1 v2v Pairwise Potentials Labeling f shown in red Label l 1 Label l 2 Pairwise Potential P

Constraints Uniqueness Constraint ∑ x i = 2 - |L| i  v a Integer Constraints x i  {-1,1} X = x x T Integer Programming Formulation

x* = argmin 1 2 ∑ u i (1 + x i ) ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  {-1,1} X = x x T Convex Non-Convex Integer Programming Formulation

Outline Convex Relaxations – Integer Programming Formulation – LP Relaxation – SDP Relaxation – SOCP Relaxation – Comparing Relaxations Move Making Algorithms Some Interesting Open Problems

LP Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  {-1,1} X = x x T Retain Convex Part Schlesinger, 1976 Relax Non-Convex Constraint

LP Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  [-1,1] X = x x T Retain Convex Part Schlesinger, 1976 Relax Non-Convex Constraint

LP Relaxation X = x x T Schlesinger, 1976 X ij  [-1,1] 1 + x i + x j + X ij ≥ 0 ∑ X ij = (2 - |L|) x i j  v b

LP Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  [-1,1] X = x x T Retain Convex Part Schlesinger, 1976 Relax Non-Convex Constraint

LP Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  [-1,1], Retain Convex Part Schlesinger, 1976 X ij  [-1,1] 1 + x i + x j + X ij ≥ 0 ∑ X ij = (2 - |L|) x i j  v b

Outline Convex Relaxations – Integer Programming Formulation – LP Relaxation – SDP Relaxation – SOCP Relaxation – Comparing Relaxations Move Making Algorithms Some Interesting Open Problems

SDP Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  {-1,1} X = x x T Retain Convex Part Lasserre, 2000 Relax Non-Convex Constraint

SDP Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  [-1,1] X = x x T Relax Non-Convex Constraint Lasserre, 2000 Retain Convex Part

x1x1 x2x2 xnxn x1x1 x2x2... xnxn 1xTxT x X = Rank = 1 X ii = 1 Positive Semidefinite Convex Non-Convex SDP Relaxation

x1x1 x2x2 xnxn x1x1 x2x2... xnxn 1xTxT x X = X ii = 1 Positive Semidefinite Convex SDP Relaxation

x* = argmin 1 2 ∑ u i (1 + x i ) ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  [-1,1] X = x x T Relax Non-Convex Constraint Lasserre, 2000 Retain Convex Part

SDP Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  [-1,1] X ii = 1 X - xx T 0 Accurate Inefficient Lasserre, 2000 Retain Convex Part Positive Semidefinite

Outline Convex Relaxations – Integer Programming Formulation – LP Relaxation – SDP Relaxation – SOCP Relaxation – Comparing Relaxations Move Making Algorithms Some Interesting Open Problems

SOCP Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  [-1,1] X ii = 1 X - xx T 0 Derive SOCP relaxation from the SDP relaxation Further Relaxation

SOCP Relaxation Choose a matrix C 1 = UU T 0 Kim and Kojima, 2000 Choose a sub-graph G Variables x G and X G (X G - x G x G T ) C1C1 ≥ 0 Choose a matrix C 2 = UU T 0 REPEAT

Outline Convex Relaxations – Integer Programming Formulation – LP Relaxation – SDP Relaxation – SOCP Relaxation – Comparing Relaxations Move Making Algorithms Some Interesting Open Problems

Dominating Relaxation For all MAP Estimation problem (u, P) A dominates B A B ≥ Dominating relaxations are better

SOCP Relaxation Choose a matrix C 1 = UU T 0 Kim and Kojima, 2000 Choose a sub-graph G Variables x G and X G (X G - x G x G T ) C1C1 ≥ 0 If G is a tree, LP dominates SOCP

Examples Muramatsu and Suzuki, 2003 (MAXCUT) Ravikumar and Lafferty, 2006 (QP Relaxation) Kumar, Torr and Zisserman, 2006 (Equivalent SOCP Relaxation)

SOCP Relaxation Choose a matrix C 1 = UU T 0 Kim and Kojima, 2000 Choose a sub-graph G Variables x G and X G (X G - x G x G T ) C1C1 ≥ 0 If G is a cycle with non-negative P

SOCP Relaxation Choose a matrix C 1 = UU T 0 Kim and Kojima, 2000 Choose a sub-graph G Variables x G and X G (X G - x G x G T ) C1C1 ≥ 0 If G is an even cycle with non-positive P

SOCP Relaxation Choose a matrix C 1 = UU T 0 Kim and Kojima, 2000 Choose a sub-graph G Variables x G and X G (X G - x G x G T ) C1C1 ≥ 0 If G is an odd cycle with 1 non-positive P

SOCP Relaxation What about other cycles? Dominated by linear cycle inequalities Cliques? Dominated by clique inequalities Kumar, Kolmogorov and Torr, 2007

Outline Convex Relaxations Move Making Algorithms – State of the Art – Comparison with LP Relaxation – Improved Moves Some Interesting Open Problems

MRFs in Vision vava vbvb lili lklk P ab (i,k) P ab (i,k) = w ab min{ d(i-k), M } w ab is non-negative Truncated Linear Truncated Quadratic d(.) is a semi-metric distance u a (i) u b (k)

Move Making Search Neighbourhood Current Solution Optimal Move Solution Space Energy Slide courtesy of Pushmeet Kohli

Outline Convex Relaxations Move Making Algorithms – State of the Art – Comparison with LP Relaxation – Improved Moves Some Interesting Open Problems

Expansion Move Variables take label  or retain current label Boykov, Veksler, Zabih 2001 Slide courtesy of Pushmeet Kohli

Sky House Tree Ground Initialize with TreeStatus:Expand GroundExpand HouseExpand Sky [Boykov, Veksler, Zabih] Expansion Move Variables take label  or retain current label Boykov, Veksler, Zabih 2001 Slide courtesy of Pushmeet Kohli

Outline Convex Relaxations Move Making Algorithms – State of the Art – Comparison with LP Relaxation – Improved Moves Some Interesting Open Problems

Multiplicative Bounds LP Move- Making Potts Truncated Linear Truncated Quadratic Metric Labeling √22M O(√M)2M O(log h) 2M Expansion Bounds as bad as ICM Bounds

Outline Convex Relaxations Move Making Algorithms – State of the Art – Comparison with LP Relaxation – Improved Moves Some Interesting Open Problems

Randomized Rounding 0y’ 0 y’ i y’ k y’ h = 1 y’ i = y 0 + y 1 + … + y i Choose an interval of length L’ y i = (1 + x i )/2

Randomized Rounding 0y’ 0 y’ i y’ k y’ h = 1 Generate a random number r  (0,1] r y’ i = y 0 + y 1 + … + y i y i = (1 + x i )/2

Randomized Rounding 0y’ 0 y’ i y’ k y’ h = 1 Assign label next to r (if within the interval) r y’ i = y 0 + y 1 + … + y i y i = (1 + x i )/2

Move Making vava vbvb Initialize the labeling Choose interval I of L’ labels Each variable can Retain old label Choose a label from I Choose best labeling Iterate over intervals Non-submodular move? Submodular overestimation

Truncated Convex Models P ab (i,k) = w ab min{ d(i-k), M } Truncated Linear Truncated Quadratic d(.) is convexd(x+1) - 2d(x) + d(x-1) ≥ 0

Move Making vava vbvb Choose interval I of L’ labels Each variable can Retain old label Choose a label from I Choose best labeling Large L’ => Non-submodular

Move Making vava vbvb Submodular problem

Move Making vava vbvb Non-submodular Problem

Move Making vava vbvb Submodular problem Ishikawa, 2003; Veksler, 2007

Move Making vava vbvb a m+1 a m+2 anan t b m+1 b m+2 bnbn s

LP Bounds Kumar and Torr, NIPS 08 Type of ProblemBound Potts2 Truncated Linear 2 + √2 Truncated Quadratic O(√M) Metric Labeling O(log h) Kumar and Koller, UAI 09 Move Making

Outline Convex Relaxations Move Making Algorithms Some Interesting Open Problems

Problem 1 Relationship between rounding and move-making? What happens when n < h ?? (Should we even use move-making here??) What about semi-metric MRFs??

Problem 2 Graph-cuts based image segmentation Vicente, Kolmogorov, Rother, 2008

Problem 2 Image segmentation with connectivity prior Vicente, Kolmogorov, Rother, 2008

Problem 2++ Kumar and Koller, 20??

Questions??