Presentation is loading. Please wait.

Presentation is loading. Please wait.

MAP Estimation Algorithms in M. Pawan Kumar, University of Oxford Pushmeet Kohli, Microsoft Research Computer Vision - Part II.

Similar presentations


Presentation on theme: "MAP Estimation Algorithms in M. Pawan Kumar, University of Oxford Pushmeet Kohli, Microsoft Research Computer Vision - Part II."— Presentation transcript:

1 MAP Estimation Algorithms in M. Pawan Kumar, University of Oxford Pushmeet Kohli, Microsoft Research Computer Vision - Part II

2 Example: Image Segmentation E(x) = c i x i + c ij x i (1-x j ) E: {0,1} n R 0 fg 1 bg Image (D) ii,j n = number of pixels

3 Example: Image Segmentation E(x) = c i x i + c ij x i (1-x j ) E: {0,1} n R 0 fg 1 bg ii,j Unary Cost (c i ) Dark ( negative ) Bright (positive) n = number of pixels

4 Example: Image Segmentation E(x) = c i x i + c ij x i (1-x j ) E: {0,1} n R 0 fg 1 bg ii,j Discontinuity Cost (c ij ) n = number of pixels

5 Example: Image Segmentation E(x) = c i x i + c ij x i (1-x j ) E: {0,1} n R 0 fg 1 bg ii,j Global Minimum (x * ) x * = arg min E(x) x How to minimize E(x)? n = number of pixels

6 Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Recent Advances and Open Problems Connection between st-mincut and energy minimization?

7 Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems

8 The st-Mincut Problem Source Sink v1v1 v2v2 2 5 9 4 2 1 Graph (V, E, C) Vertices V = {v 1, v 2... v n } Edges E = {(v 1, v 2 )....} Costs C = {c (1, 2)....}

9 The st-Mincut Problem Source Sink v1v1 v2v2 2 5 9 4 2 1 What is a st-cut?

10 The st-Mincut Problem Source Sink v1v1 v2v2 2 5 9 4 2 1 What is a st-cut? An st-cut (S,T) divides the nodes between source and sink. What is the cost of a st-cut? Sum of cost of all edges going from S to T 5 + 2 + 9 = 16

11 The st-Mincut Problem What is a st-cut? An st-cut (S,T) divides the nodes between source and sink. What is the cost of a st-cut? Sum of cost of all edges going from S to T What is the st-mincut? st-cut with the minimum cost Source Sink v1v1 v2v2 2 5 9 4 2 1 2 + 1 + 4 = 7

12 How to compute the st-mincut? Source Sink v1v1 v2v2 2 5 9 4 2 1 Solve the dual maximum flow problem In every network, the maximum flow equals the cost of the st-mincut Min-cut\Max-flow Theorem Compute the maximum flow between Source and Sink Constraints Edges: Flow < Capacity Nodes: Flow in = Flow out

13 Maxflow Algorithms Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Source Sink v1v1 v2v2 2 5 9 4 2 1 Algorithms assume non-negative capacity Flow = 0

14 Maxflow Algorithms Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Source Sink v1v1 v2v2 2 5 9 4 2 1 Algorithms assume non-negative capacity Flow = 0

15 Maxflow Algorithms Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Source Sink v1v1 v2v2 2-2 5-2 9 4 2 1 Algorithms assume non-negative capacity Flow = 0 + 2

16 Maxflow Algorithms Source Sink v1v1 v2v2 0 3 9 4 2 1 Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Algorithms assume non-negative capacity Flow = 2

17 Maxflow Algorithms Source Sink v1v1 v2v2 0 3 9 4 2 1 Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Algorithms assume non-negative capacity Flow = 2

18 Maxflow Algorithms Source Sink v1v1 v2v2 0 3 9 4 2 1 Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Algorithms assume non-negative capacity Flow = 2

19 Maxflow Algorithms Source Sink v1v1 v2v2 0 3 5 0 2 1 Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Algorithms assume non-negative capacity Flow = 2 + 4

20 Maxflow Algorithms Source Sink v1v1 v2v2 0 3 5 0 2 1 Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Algorithms assume non-negative capacity Flow = 6

21 Maxflow Algorithms Source Sink v1v1 v2v2 0 3 5 0 2 1 Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Algorithms assume non-negative capacity Flow = 6

22 Maxflow Algorithms Source Sink v1v1 v2v2 0 2 4 0 2+1 1-1 Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Algorithms assume non-negative capacity Flow = 6 + 1

23 Maxflow Algorithms Source Sink v1v1 v2v2 0 2 4 0 3 0 Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Algorithms assume non-negative capacity Flow = 7

24 Maxflow Algorithms Source Sink v1v1 v2v2 0 2 4 0 3 0 Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Algorithms assume non-negative capacity Flow = 7

25 History of Maxflow Algorithms [Slide credit: Andrew Goldberg] Augmenting Path and Push-Relabel n: # nodes m: # edges U: maximum edge weight Algorithms assume non- negative edge weights

26 History of Maxflow Algorithms [Slide credit: Andrew Goldberg] Augmenting Path and Push-Relabel n: # nodes m: # edges U: maximum edge weight Algorithms assume non- negative edge weights

27 Augmenting Path based Algorithms a1a1 a2a2 1000 1 Sink Source 1000 0 Ford Fulkerson: Choose any augmenting path

28 a1a1 a2a2 1000 1 Sink Source 1000 0 Augmenting Path based Algorithms Bad Augmenting Paths Ford Fulkerson: Choose any augmenting path

29 a1a1 a2a2 1000 1 Sink Source 1000 0 Augmenting Path based Algorithms Bad Augmenting Path Ford Fulkerson: Choose any augmenting path

30 a1a1 a2a2 999 0 Sink Source 1000 999 1 Augmenting Path based Algorithms Ford Fulkerson: Choose any augmenting path

31 a1a1 a2a2 999 0 Sink Source 1000 999 1 Ford Fulkerson: Choose any augmenting path n: # nodes m: # edges We will have to perform 2000 augmentations! Worst case complexity: O (m x Total_Flow) (Pseudo-polynomial bound: depends on flow) Augmenting Path based Algorithms

32 Dinic: Choose shortest augmenting path n: # nodes m: # edges Worst case Complexity: O (m n 2 ) Augmenting Path based Algorithms a1a1 a2a2 1000 1 Sink Source 1000 0

33 Maxflow in Computer Vision Specialized algorithms for vision problems – Grid graphs – Low connectivity (m ~ O(n)) Dual search tree augmenting path algorithm [Boykov and Kolmogorov PAMI 2004] Finds approximate shortest augmenting paths efficiently High worst-case time complexity Empirically outperforms other algorithms on vision problems

34 Maxflow in Computer Vision Specialized algorithms for vision problems – Grid graphs – Low connectivity (m ~ O(n)) Dual search tree augmenting path algorithm [Boykov and Kolmogorov PAMI 2004] Finds approximate shortest augmenting paths efficiently High worst-case time complexity Empirically outperforms other algorithms on vision problems Efficient code available on the web http://www.adastral.ucl.ac.uk/~vladkolm/software.html

35 Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems

36 St-mincut and Energy Minimization T Sst-mincut E: {0,1} n R Minimizing a Qudratic Pseudoboolean function E(x) Functions of boolean variablesPseudoboolean? Polynomial time st-mincut algorithms require non-negative edge weights E(x) = c i x i + c ij x i (1-x j ) c ij 0 i,ji

37 So how does this work? Construct a graph such that: 1.Any st-cut corresponds to an assignment of x 2.The cost of the cut is equal to the energy of x : E(x) Solution T S st-mincut E(x)

38 Graph Construction Sink (1) Source (0) a1a1 a2a2 E(a 1,a 2 )

39 Graph Construction Sink (1) Source (0) a1a1 a2a2 E(a 1,a 2 ) = 2a 1 2

40 Graph Construction a1a1 a2a2 E(a 1,a 2 ) = 2a 1 + 5ā 1 2 5 Sink (1) Source (0)

41 Graph Construction a1a1 a2a2 E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 2 5 9 4 Sink (1) Source (0)

42 Graph Construction a1a1 a2a2 E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 2 5 9 4 2 Sink (1) Source (0)

43 Graph Construction a1a1 a2a2 E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0)

44 Graph Construction a1a1 a2a2 E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0)

45 Graph Construction a1a1 a2a2 E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 a 1 = 1 a 2 = 1 E (1,1) = 11 Cost of cut = 11 Sink (1) Source (0)

46 Graph Construction a1a1 a2a2 E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0) a 1 = 1 a 2 = 0 E (1,0) = 8 st-mincut cost = 8

47 Energy Function Reparameterization Two functions E 1 and E 2 are reparameterizations if E 1 (x) = E 2 (x) for all x For instance: E 1 (a 1 ) = 1+ 2a 1 + 3ā 1 E 2 (a 1 ) = 3 + ā 1 a1a1 ā1ā1 1+ 2a 1 + 3ā 1 3 + ā 1 0144 1033

48 Flow and Reparametrization a1a1 a2a2 E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0)

49 Flow and Reparametrization a1a1 a2a2 E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0) 2a 1 + 5ā 1 = 2(a 1 +ā 1 ) + 3ā 1 = 2 + 3ā 1

50 Flow and Reparametrization Sink (1) Source (0) a1a1 a2a2 E(a 1,a 2 ) = 2 + 3ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 0 3 9 4 2 1 2a 1 + 5ā 1 = 2(a 1 +ā 1 ) + 3ā 1 = 2 + 3ā 1

51 Sink (0) Source (1) a1a1 a2a2 E(a 1,a 2 ) = 2 + 3ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 0 3 9 4 2 1 Flow and Reparametrization 9a 2 + 4ā 2 = 4(a 2 +ā 2 ) + 5ā 2 = 4 + 5ā 2

52 Sink (0) Source (1) a1a1 a2a2 E(a 1,a 2 ) = 2 + 3ā 1 + 5a 2 + 4 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1 9a 2 + 4ā 2 = 4(a 2 +ā 2 ) + 5ā 2 = 4 + 5ā 2 Flow and Reparametrization

53 Sink (0) Source (1) a1a1 a2a2 E(a 1,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1 Flow and Reparametrization

54 Sink (0) Source (1) a1a1 a2a2 E(a 1,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1 Flow and Reparametrization

55 Sink (0) Source (1) a1a1 a2a2 E(a 1,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1 Flow and Reparametrization 3ā 1 + 5a 2 + 2a 1 ā 2 = 2(ā 1 +a 2 +a 1 ā 2 ) +ā 1 +3a 2 = 2(1+ā 1 a 2 ) +ā 1 +3a 2 a1a1 a2a2 F1F2 0011 0122 1011 1111 F1 = ā 1 +a 2 +a 1 ā 2 F2 = 1+ā 1 a 2

56 Sink (0) Source (1) a1a1 a2a2 E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 0 1 3 0 0 3 Flow and Reparametrization 3ā 1 + 5a 2 + 2a 1 ā 2 = 2(ā 1 +a 2 +a 1 ā 2 ) +ā 1 +3a 2 = 2(1+ā 1 a 2 ) +ā 1 +3a 2 a1a1 a2a2 F1F2 0011 0122 1011 1111 F1 = ā 1 +a 2 +a 1 ā 2 F2 = 1+ā 1 a 2

57 Sink (0) Source (1) a1a1 a2a2 0 1 3 0 0 3 Flow and Reparametrization E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 No more augmenting paths possible

58 Sink (0) Source (1) a1a1 a2a2 0 1 3 0 0 3 Flow and Reparametrization E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 Total Flow Residual Graph (positive coefficients) bound on the optimal solution Inference of the optimal solution becomes trivial because the bound is tight

59 Sink (0) Source (1) a1a1 a2a2 0 1 3 0 0 3 Flow and Reparametrization E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 a 1 = 1 a 2 = 0 E (1,0) = 8 st-mincut cost = 8 Total Flow bound on the optimal solution Inference of the optimal solution becomes trivial because the bound is tight Residual Graph (positive coefficients)

60 Example: Image Segmentation E(x) = c i x i + c ij x i (1-x j ) E: {0,1} n R 0 fg 1 bg ii,j Global Minimum (x * ) x * = arg min E(x) x How to minimize E(x)?

61 How does the code look like? Sink (1) Source (0) Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1)

62 How does the code look like? Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) a1a1 a2a2 fgCost(a 1 ) Sink (1) Source (0) fgCost(a 2 ) bgCost(a 1 ) bgCost(a 2 )

63 Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost(p,q)); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) How does the code look like? a1a1 a2a2 fgCost(a 1 ) Sink (1) Source (0) fgCost(a 2 ) bgCost(a 1 ) bgCost(a 2 ) cost(p,q)

64 Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost(p,q)); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) How does the code look like? a1a1 a2a2 fgCost(a 1 ) Sink (1) Source (0) fgCost(a 2 ) bgCost(a 1 ) bgCost(a 2 ) cost(p,q) a 1 = bg a 2 = fg

65 Image Segmentation in Video Image Flow Global Optimum n-links st-cut s t = 0 = 1 E(x) x*x*

66 Image Segmentation in Video Image Flow Global Optimum

67 Dynamic Energy Minimization EBEB SBSB computationally expensive operation EAEA SASA minimize Recycling Solutions Can we do better? Boykov & Jolly ICCV01, Kohli & Torr (ICCV05, PAMI07)

68 Dynamic Energy Minimization EBEB SBSB computationally expensive operation EAEA SASA minimize cheaper operation Simpler energy E B* differences between A and B similar Kohli & Torr (ICCV05, PAMI07) 3 – 100000 time speedup! Reparametrization Reuse flow Boykov & Jolly ICCV01, Kohli & Torr (ICCV05, PAMI07)

69 Reparametrized Energy Dynamic Energy Minimization Kohli & Torr (ICCV05, PAMI07) Boykov & Jolly ICCV01, Kohli & Torr (ICCV05, PAMI07) E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 Original Energy E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 7a 1 ā 2 + ā 1 a 2 E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 + 5a 1 ā 2 New Energy New Reparametrized Energy

70 Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems

71 Minimizing Energy Functions Space of Function Minimization Problems Submodular Functions NP-Hard General Energy Functions – NP-hard to minimize – Only approximate minimization possible Easy energy functions – Solvable in polynomial time – Submodular ~ O(n 6 ) MAXCUT Functions defined on trees

72 Submodular Set Functions Set function f 2 |E| 2 |E| = #subsets of E Let E= {a 1,a 2,.... a n } be a set

73 Submodular Set Functions Set function f 2 |E| is submodular if E A B f(A) + f(B) f(A B) + f(A B) for all A,B E 2 |E| = #subsets of E Let E= {a 1,a 2,.... a n } be a set Important Property Sum of two submodular functions is submodular

74 Minimizing Submodular Functions Minimizing general submodular functions – O(n 5 Q + n 6 ) where Q is function evaluation time [Orlin, IPCO 2007] Symmetric submodular functions – E (x) = E (1 - x) – O(n 3 ) [Queyranne 1998] Quadratic pseudoboolean – Can be transformed to st-mincut – One node per variable [ O(n 3 ) complexity] – Very low empirical running time

75 Submodular Pseudoboolean Functions All functions for one boolean variable (f: {0,1} -> R) are submodular A function of two boolean variables (f: {0,1} 2 -> R) is submodular if f(0,1) + f(1,0) f(0,0) + f(1,1) A general pseudoboolean function f 2 n is submodular if all its projections f p are submodular i.e. f p (0,1) + f p (1,0) f p (0,0) + f p (1,1) Function defined over boolean vectors x = {x 1,x 2,.... x n } Definition :

76 E(x) = θ i (x i ) + θ ij (x i,x j ) i,ji Quadratic Submodular Pseudoboolean Functions θ ij (0,1) + θ ij (1,0) θ ij (0,0) + θ ij (1,1)For all ij

77 E(x) = θ i (x i ) + θ ij (x i,x j ) i,ji Quadratic Submodular Pseudoboolean Functions θ ij (0,1) + θ ij (1,0) θ ij (0,0) + θ ij (1,1)For all ij E(x) = c i x i + c ij x i (1-x j ) c ij 0 i,ji Equivalent (transformable) i.e. All submodular QPBFs are st-mincut solvable

78 AB CD 0 1 0 1 xixi xjxj = A + 00 C-A 0 1 0 1 0 D-C 0 0 1 0 1 0 B+C -A-D 00 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D 0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i,x j ) = θ ij (0,0) + (θ ij (1,0)-θ ij (0,0)) x i + (θ ij (1,0)-θ ij (0,0)) x j + (θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j

79 AB CD 0 1 0 1 xixi xjxj = A + 00 C-A 0 1 0 1 0 D-C 0 0 1 0 1 0 B+C -A-D 00 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D 0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i,x j ) = θ ij (0,0) + (θ ij (1,0)-θ ij (0,0)) x i + (θ ij (1,0)-θ ij (0,0)) x j + (θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j

80 AB CD 0 1 0 1 xixi xjxj = A + 00 C-A 0 1 0 1 0 D-C 0 0 1 0 1 0 B+C -A-D 00 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D 0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i,x j ) = θ ij (0,0) + (θ ij (1,0)-θ ij (0,0)) x i + (θ ij (1,0)-θ ij (0,0)) x j + (θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j

81 AB CD 0 1 0 1 xixi xjxj = A + 00 C-A 0 1 0 1 0 D-C 0 0 1 0 1 0 B+C -A-D 00 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D 0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i,x j ) = θ ij (0,0) + (θ ij (1,0)-θ ij (0,0)) x i + (θ ij (1,0)-θ ij (0,0)) x j + (θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j

82 AB CD 0 1 0 1 xixi xjxj = A + 00 C-A 0 1 0 1 0 D-C 0 0 1 0 1 0 B+C -A-D 00 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D 0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i,x j ) = θ ij (0,0) + (θ ij (1,0)-θ ij (0,0)) x i + (θ ij (1,0)-θ ij (0,0)) x j + (θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j

83 E(x) = θ i (x i ) + θ ij (x i,x j ) i,ji Quadratic Submodular Pseudoboolean Functions θ ij (0,1) + θ ij (1,0) θ ij (0,0) + θ ij (1,1)For all ij Equivalent (transformable) T S st-mincut x in {0,1} n

84 Minimizing Non-Submodular Functions Minimizing general non-submodular functions is NP-hard. Commonly used method is to solve a relaxation of the problem E(x) = θ i (x i ) + θ ij (x i,x j ) i,ji θ ij (0,1) + θ ij (1,0) θ ij (0,0) + θ ij (1,1) for some ij [Slide credit: Carsten Rother]

85 pairwise nonsubmodular unary pairwise submodular Minimization using Roof-dual Relaxation [Slide credit: Carsten Rother]

86 Double number of variables: Minimization using Roof-dual Relaxation [Slide credit: Carsten Rother]

87 Double number of variables: Minimization using Roof-dual Relaxation Non- submodular Submodular

88 Double number of variables: Minimization using Roof-dual Relaxation is submodular ! Ignore (solvable using st-mincut) Property of the problem :

89 is the optimal label Property of the solution : Double number of variables: Minimization using Roof-dual Relaxation

90 Recap Exact minimization of Submodular QBFs using graph cuts. Obtaining partially optimal solutions of non- submodular QBFs using graph cuts.

91 But... Need higher order energy functions to model image structure – Field of experts [Roth and Black] Many problems in computer vision involve multiple labels E(x) = θ i (x i ) + θ ij (x i,x j ) + θ c (x c ) i,ji c x Labels L = {l 1, l 2, …, l k }Clique c V

92 Transforming problems in QBFs Multi-label Functions Pseudoboolean Functions Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions

93 Transforming problems in QBFs Multi-label Functions Pseudoboolean Functions Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions

94 Higher order to Quadratic Simple Example using Auxiliary variables { 0 if all x i = 0 C 1 otherwise f(x) = min f(x) min C 1 a + C 1 ā x i x L = {0,1} n x = x,a {0,1} Higher Order Submodular Function Quadratic Submodular Function x i = 0 a=0 (ā=1)f(x) = 0 x i 1 a=1 (ā=0)f(x) = C 1

95 Higher order to Quadratic min f(x) min C 1 a + C 1 ā x i x = x,a {0,1} Higher Order Submodular Function Quadratic Submodular Function x i 1 23 C1C1 C 1 x i

96 Higher order to Quadratic min f(x) min C 1 a + C 1 ā x i x = x,a {0,1} Higher Order Submodular Function Quadratic Submodular Function x i 1 23 C1C1 C 1 x i a=1 a=0 Lower envelop of concave functions is concave

97 Higher order to Quadratic min f(x) min f 1 (x)a + f 2 (x)ā x = x,a {0,1} Higher Order Submodular Function Quadratic Submodular Function x i 1 23 a=1 Lower envelop of concave functions is concave f 2 (x) f 1 (x)

98 Higher order to Quadratic min f(x) min f 1 (x)a + f 2 (x)ā x = x,a {0,1} Higher Order Submodular Function Quadratic Submodular Function x i 1 23 a=1 a=0 Lower envelop of concave functions is concave f 2 (x) f 1 (x)

99 Transforming problems in QBFs Multi-label Functions Pseudoboolean Functions Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions

100 Multi-label to Pseudo-boolean So what is the problem? E b (x 1,x 2,..., x m )E m (y 1,y 2,..., y n ) Multi-label ProblemBinary label Problem y i L = {l 1, l 2, …, l k }x i L = {0,1} such that: Let Y and X be the set of feasible solutions, then 1.For each binary solution x X with finite energy there exists exactly one multi-label solution y Y -> One-One encoding function T:X->Y 2. arg min E m (y) = T(arg min E b (x))

101 Multi-label to Pseudo-boolean Popular encoding scheme [Roy and Cox 98, Ishikawa 03, Schlesinger & Flach 06]

102 Multi-label to Pseudo-boolean Popular encoding scheme [Roy and Cox 98, Ishikawa 03, Schlesinger & Flach 06] Ishikawas result: E(y) = θ i (y i ) + θ ij (y i,y j ) i,ji y Labels L = {l 1, l 2, …, l k } θ ij (y i,y j ) = g(|y i -y j |) Convex Function g(|y i -y j |) |y i -y j |

103 Multi-label to Pseudo-boolean Popular encoding scheme [Roy and Cox 98, Ishikawa 03, Schlesinger & Flach 06] Schlesinger & Flach 06: E(y) = θ i (y i ) + θ ij (y i,y j ) i,ji y Labels L = {l 1, l 2, …, l k } θ ij (l i+1,l j ) + θ ij (l i,l j+1 ) θ ij (l i,l j ) + θ ij (l i+1,l j+1 ) Covers all Submodular multi-label functions More general than Ishikawa

104 Multi-label to Pseudo-boolean Applicability – Only solves restricted class of energy functions – Cannot handle Potts model potentials Computational Cost – Very high computational cost – Problem size = |Variables| x |Labels| – Gray level image denoising (1 Mpixel image) (~2.5 x 10 8 graph nodes) Problems

105 Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems

106 St-mincut based Move algorithms Commonly used for solving non-submodular multi-label problems Extremely efficient and produce good solutions Not Exact: Produce local optima E(x) = θ i (x i ) + θ ij (x i,x j ) i,ji x Labels L = {l 1, l 2, …, l k }

107 Move Making Algorithms Solution Space Energy

108 Move Making Algorithms Search Neighbourhood Current Solution Optimal Move Solution Space Energy

109 Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move xcxc (t)(t) Key Property Move Space Bigger move space Solution Space Energy Better solutions Finding the optimal move hard

110 Moves using Graph Cuts Expansion and Swap move algorithms [Boykov Veksler and Zabih, PAMI 2001] Makes a series of changes to the solution (moves) Each move results in a solution with smaller energy Space of Solutions (x) : L N Move Space (t) : 2 N Search Neighbourhood Current Solution N Number of Variables L Number of Labels

111 Moves using Graph Cuts Expansion and Swap move algorithms [Boykov Veksler and Zabih, PAMI 2001] Makes a series of changes to the solution (moves) Each move results in a solution with smaller energy Current Solution Construct a move function Minimize move function to get optimal move Move to new solution How to minimize move functions?

112 General Binary Moves Minimize over move variables t to get the optimal move x = t x 1 + (1- t ) x 2 New solution Current Solution Second solution E m ( t ) = E( t x 1 + (1- t ) x 2 ) Boykov, Veksler and Zabih, PAMI 2001 Move energy is a submodular QPBF (Exact Minimization Possible)

113 Swap Move Variables labeled α, β can swap their labels [Boykov, Veksler, Zabih]

114 Swap Move Sky House Tree Ground Swap Sky, House [Boykov, Veksler, Zabih] Variables labeled α, β can swap their labels

115 Swap Move Variables labeled α, β can swap their labels Move energy is submodular if: – Unary Potentials: Arbitrary – Pairwise potentials: Semimetric [Boykov, Veksler, Zabih] θ ij (l a,l b ) 0 θ ij (l a,l b ) = 0 a = b Examples: Potts model, Truncated Convex

116 Expansion Move [Boykov, Veksler, Zabih] Variables take label or retain current label

117 Expansion Move Sky House Tree Ground Initialize with TreeStatus:Expand GroundExpand HouseExpand Sky [Boykov, Veksler, Zabih] Variables take label or retain current label

118 Expansion Move Move energy is submodular if: – Unary Potentials: Arbitrary – Pairwise potentials: Metric [Boykov, Veksler, Zabih] θ ij (l a,l b ) + θ ij (l b,l c ) θ ij (l a,l c ) Semi metric + Triangle Inequality Variables take label or retain current label Examples: Potts model, Truncated linear Cannot solve truncated quadratic

119 General Binary Moves Move TypeFirst Solution Second Solution Guarantee ExpansionOld solutionAll alphaMetric FusionAny solution Minimize over move variables t x = t x 1 + (1-t) x 2 New solution First solution Second solution Move functions can be non-submodular!!

120 Solving Continuous Problems using Fusion Move x = t x 1 + (1-t) x 2 (Lempitsky et al. CVPR08, Woodford et al. CVPR08) x 1, x 2 can be continuous F x1x1 x2x2 x Optical Flow Example Final Solution Solution from Method 1 Solution from Method 2

121 Range Moves Move variables can be multi-label Optimal move found out by using the Ishikawa Useful for minimizing energies with truncated convex pairwise potentials O. Veksler, CVPR 2007 θ ij (y i,y j ) = min(|y i -y j |,T) |y i -y j | θ ij (y i,y j ) T x = ( t == 1) x 1 + (t == 2) x 2 +… +(t == k) x k

122 Move Algorithms for Solving Higher Order Energies Higher order functions give rise to higher order move energies Move energies for certain classes of higher order energies can be transformed to QPBFs. E(x) = θ i (x i ) + θ ij (x i,x j ) + θ c (x c ) i,ji c x Labels L = {l 1, l 2, …, l k }Clique c V [Kohli, Kumar and Torr, CVPR07] [Kohli, Ladicky and Torr, CVPR08]

123 Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems

124 Solving Mixed Programming Problems x – binary image segmentation (x i {0,1}) ω – non-local parameter (lives in some large set Ω) constant unary potentials pairwise potentials E(x, ω ) = C( ω ) + θ i ( ω, x i ) + θ ij ( ω, x i,x j ) i,ji 0 Rough Shape Prior Stickman Model ω Pose θ i ( ω, x i ) Shape Prior

125 Open Problems Submodular Functions st-mincut Equivalent Characterization of Problems Solvable using st-mincut What functions can be transformed to submodular QBFs?

126 Minimizing General Higher Order Functions We saw how simple higher order potentials can be solved How more sophisticated higher order potentials can be solved?

127 Exact Transformation (global optimum) Or Relaxed transformation (partially optimal) Summary T S st-mincut Labelling Problem Submodular Quadratic Pseudoboolean Function Move making algorithms Sub-problem

128 Thanks. Questions?

129 Use of Higher order Potentials Stereo - Woodford et al. CVPR 2008 P1P1 P2P2 P3P3 5 6 7 8 Pixels Disparity Labels E(x 1,x 2,x 3 ) = θ 12 (x 1,x 2 ) + θ 23 (x 2,x 3 ) θ ij (x i,x j ) = 0 if x i =x j C otherwise { E(6,6,6) = 0 + 0 = 0

130 Use of Higher order Potentials Stereo - Woodford et al. CVPR 2008 P1P1 P2P2 P3P3 5 6 7 8 Pixels Disparity Labels E(x 1,x 2,x 3 ) = θ 12 (x 1,x 2 ) + θ 23 (x 2,x 3 ) θ ij (x i,x j ) = 0 if x i =x j C otherwise { E(6,6,6) = 0 + 0 = 0 E(6,7,7) = 1 + 0 = 1

131 Use of Higher order Potentials Stereo - Woodford et al. CVPR 2008 P1P1 P2P2 P3P3 5 6 7 8 Pixels Disparity Labels Pairwise potential penalize slanted planar surfaces E(x 1,x 2,x 3 ) = θ 12 (x 1,x 2 ) + θ 23 (x 2,x 3 ) θ ij (x i,x j ) = 0 if x i =x j C otherwise { E(6,6,6) = 0 + 0 = 0 E(6,7,7) = 1 + 0 = 1 E(6,7,8) = 1 + 1 = 2

132 Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move x E(x)E(x) xcxc Transformation function T (t)(t) T(x c, t) = x n = x c + t

133 Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move E(x)E(x) xcxc Transformation function T EmEm Move Energy (t)(t) x E m (t) = E(T(x c, t)) T(x c, t) = x n = x c + t

134 Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move E(x)E(x) xcxc E m (t) = E(T(x c, t)) Transformation function T EmEm Move Energy T(x c, t) = x n = x c + t minimize t* Optimal Move (t)(t) x


Download ppt "MAP Estimation Algorithms in M. Pawan Kumar, University of Oxford Pushmeet Kohli, Microsoft Research Computer Vision - Part II."

Similar presentations


Ads by Google