Download presentation

Presentation is loading. Please wait.

Published byBrooke Ferguson Modified over 3 years ago

1
MAP Estimation Algorithms in M. Pawan Kumar, University of Oxford Pushmeet Kohli, Microsoft Research Computer Vision - Part II

2
Example: Image Segmentation E(x) = c i x i + c ij x i (1-x j ) E: {0,1} n R 0 fg 1 bg Image (D) ii,j n = number of pixels

3
Example: Image Segmentation E(x) = c i x i + c ij x i (1-x j ) E: {0,1} n R 0 fg 1 bg ii,j Unary Cost (c i ) Dark ( negative ) Bright (positive) n = number of pixels

4
Example: Image Segmentation E(x) = c i x i + c ij x i (1-x j ) E: {0,1} n R 0 fg 1 bg ii,j Discontinuity Cost (c ij ) n = number of pixels

5
Example: Image Segmentation E(x) = c i x i + c ij x i (1-x j ) E: {0,1} n R 0 fg 1 bg ii,j Global Minimum (x * ) x * = arg min E(x) x How to minimize E(x)? n = number of pixels

6
Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Recent Advances and Open Problems Connection between st-mincut and energy minimization?

7
Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems

8
The st-Mincut Problem Source Sink v1v1 v2v2 2 5 9 4 2 1 Graph (V, E, C) Vertices V = {v 1, v 2... v n } Edges E = {(v 1, v 2 )....} Costs C = {c (1, 2)....}

9
The st-Mincut Problem Source Sink v1v1 v2v2 2 5 9 4 2 1 What is a st-cut?

10
The st-Mincut Problem Source Sink v1v1 v2v2 2 5 9 4 2 1 What is a st-cut? An st-cut (S,T) divides the nodes between source and sink. What is the cost of a st-cut? Sum of cost of all edges going from S to T 5 + 2 + 9 = 16

11
The st-Mincut Problem What is a st-cut? An st-cut (S,T) divides the nodes between source and sink. What is the cost of a st-cut? Sum of cost of all edges going from S to T What is the st-mincut? st-cut with the minimum cost Source Sink v1v1 v2v2 2 5 9 4 2 1 2 + 1 + 4 = 7

12
How to compute the st-mincut? Source Sink v1v1 v2v2 2 5 9 4 2 1 Solve the dual maximum flow problem In every network, the maximum flow equals the cost of the st-mincut Min-cut\Max-flow Theorem Compute the maximum flow between Source and Sink Constraints Edges: Flow < Capacity Nodes: Flow in = Flow out

13
Maxflow Algorithms Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Source Sink v1v1 v2v2 2 5 9 4 2 1 Algorithms assume non-negative capacity Flow = 0

14
Maxflow Algorithms Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Source Sink v1v1 v2v2 2 5 9 4 2 1 Algorithms assume non-negative capacity Flow = 0

15
Maxflow Algorithms Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Source Sink v1v1 v2v2 2-2 5-2 9 4 2 1 Algorithms assume non-negative capacity Flow = 0 + 2

16
Maxflow Algorithms Source Sink v1v1 v2v2 0 3 9 4 2 1 Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Algorithms assume non-negative capacity Flow = 2

17
Maxflow Algorithms Source Sink v1v1 v2v2 0 3 9 4 2 1 Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Algorithms assume non-negative capacity Flow = 2

18
Maxflow Algorithms Source Sink v1v1 v2v2 0 3 9 4 2 1 Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Algorithms assume non-negative capacity Flow = 2

19
Maxflow Algorithms Source Sink v1v1 v2v2 0 3 5 0 2 1 Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Algorithms assume non-negative capacity Flow = 2 + 4

20
Maxflow Algorithms Source Sink v1v1 v2v2 0 3 5 0 2 1 Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Algorithms assume non-negative capacity Flow = 6

21
Maxflow Algorithms Source Sink v1v1 v2v2 0 3 5 0 2 1 Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Algorithms assume non-negative capacity Flow = 6

22
Maxflow Algorithms Source Sink v1v1 v2v2 0 2 4 0 2+1 1-1 Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Algorithms assume non-negative capacity Flow = 6 + 1

23
Maxflow Algorithms Source Sink v1v1 v2v2 0 2 4 0 3 0 Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Algorithms assume non-negative capacity Flow = 7

24
Maxflow Algorithms Source Sink v1v1 v2v2 0 2 4 0 3 0 Augmenting Path Based Algorithms 1.Find path from source to sink with positive capacity 2.Push maximum possible flow through this path 3.Repeat until no path can be found Algorithms assume non-negative capacity Flow = 7

25
History of Maxflow Algorithms [Slide credit: Andrew Goldberg] Augmenting Path and Push-Relabel n: # nodes m: # edges U: maximum edge weight Algorithms assume non- negative edge weights

26
History of Maxflow Algorithms [Slide credit: Andrew Goldberg] Augmenting Path and Push-Relabel n: # nodes m: # edges U: maximum edge weight Algorithms assume non- negative edge weights

27
Augmenting Path based Algorithms a1a1 a2a2 1000 1 Sink Source 1000 0 Ford Fulkerson: Choose any augmenting path

28
a1a1 a2a2 1000 1 Sink Source 1000 0 Augmenting Path based Algorithms Bad Augmenting Paths Ford Fulkerson: Choose any augmenting path

29
a1a1 a2a2 1000 1 Sink Source 1000 0 Augmenting Path based Algorithms Bad Augmenting Path Ford Fulkerson: Choose any augmenting path

30
a1a1 a2a2 999 0 Sink Source 1000 999 1 Augmenting Path based Algorithms Ford Fulkerson: Choose any augmenting path

31
a1a1 a2a2 999 0 Sink Source 1000 999 1 Ford Fulkerson: Choose any augmenting path n: # nodes m: # edges We will have to perform 2000 augmentations! Worst case complexity: O (m x Total_Flow) (Pseudo-polynomial bound: depends on flow) Augmenting Path based Algorithms

32
Dinic: Choose shortest augmenting path n: # nodes m: # edges Worst case Complexity: O (m n 2 ) Augmenting Path based Algorithms a1a1 a2a2 1000 1 Sink Source 1000 0

33
Maxflow in Computer Vision Specialized algorithms for vision problems – Grid graphs – Low connectivity (m ~ O(n)) Dual search tree augmenting path algorithm [Boykov and Kolmogorov PAMI 2004] Finds approximate shortest augmenting paths efficiently High worst-case time complexity Empirically outperforms other algorithms on vision problems

34
Maxflow in Computer Vision Specialized algorithms for vision problems – Grid graphs – Low connectivity (m ~ O(n)) Dual search tree augmenting path algorithm [Boykov and Kolmogorov PAMI 2004] Finds approximate shortest augmenting paths efficiently High worst-case time complexity Empirically outperforms other algorithms on vision problems Efficient code available on the web http://www.adastral.ucl.ac.uk/~vladkolm/software.html

35
Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems

36
St-mincut and Energy Minimization T Sst-mincut E: {0,1} n R Minimizing a Qudratic Pseudoboolean function E(x) Functions of boolean variablesPseudoboolean? Polynomial time st-mincut algorithms require non-negative edge weights E(x) = c i x i + c ij x i (1-x j ) c ij 0 i,ji

37
So how does this work? Construct a graph such that: 1.Any st-cut corresponds to an assignment of x 2.The cost of the cut is equal to the energy of x : E(x) Solution T S st-mincut E(x)

38
Graph Construction Sink (1) Source (0) a1a1 a2a2 E(a 1,a 2 )

39
Graph Construction Sink (1) Source (0) a1a1 a2a2 E(a 1,a 2 ) = 2a 1 2

40
Graph Construction a1a1 a2a2 E(a 1,a 2 ) = 2a 1 + 5ā 1 2 5 Sink (1) Source (0)

41
Graph Construction a1a1 a2a2 E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 2 5 9 4 Sink (1) Source (0)

42
Graph Construction a1a1 a2a2 E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 2 5 9 4 2 Sink (1) Source (0)

43
Graph Construction a1a1 a2a2 E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0)

44
Graph Construction a1a1 a2a2 E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0)

45
Graph Construction a1a1 a2a2 E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 a 1 = 1 a 2 = 1 E (1,1) = 11 Cost of cut = 11 Sink (1) Source (0)

46
Graph Construction a1a1 a2a2 E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0) a 1 = 1 a 2 = 0 E (1,0) = 8 st-mincut cost = 8

47
Energy Function Reparameterization Two functions E 1 and E 2 are reparameterizations if E 1 (x) = E 2 (x) for all x For instance: E 1 (a 1 ) = 1+ 2a 1 + 3ā 1 E 2 (a 1 ) = 3 + ā 1 a1a1 ā1ā1 1+ 2a 1 + 3ā 1 3 + ā 1 0144 1033

48
Flow and Reparametrization a1a1 a2a2 E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0)

49
Flow and Reparametrization a1a1 a2a2 E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0) 2a 1 + 5ā 1 = 2(a 1 +ā 1 ) + 3ā 1 = 2 + 3ā 1

50
Flow and Reparametrization Sink (1) Source (0) a1a1 a2a2 E(a 1,a 2 ) = 2 + 3ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 0 3 9 4 2 1 2a 1 + 5ā 1 = 2(a 1 +ā 1 ) + 3ā 1 = 2 + 3ā 1

51
Sink (0) Source (1) a1a1 a2a2 E(a 1,a 2 ) = 2 + 3ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 0 3 9 4 2 1 Flow and Reparametrization 9a 2 + 4ā 2 = 4(a 2 +ā 2 ) + 5ā 2 = 4 + 5ā 2

52
Sink (0) Source (1) a1a1 a2a2 E(a 1,a 2 ) = 2 + 3ā 1 + 5a 2 + 4 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1 9a 2 + 4ā 2 = 4(a 2 +ā 2 ) + 5ā 2 = 4 + 5ā 2 Flow and Reparametrization

53
Sink (0) Source (1) a1a1 a2a2 E(a 1,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1 Flow and Reparametrization

54
Sink (0) Source (1) a1a1 a2a2 E(a 1,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1 Flow and Reparametrization

55
Sink (0) Source (1) a1a1 a2a2 E(a 1,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1 Flow and Reparametrization 3ā 1 + 5a 2 + 2a 1 ā 2 = 2(ā 1 +a 2 +a 1 ā 2 ) +ā 1 +3a 2 = 2(1+ā 1 a 2 ) +ā 1 +3a 2 a1a1 a2a2 F1F2 0011 0122 1011 1111 F1 = ā 1 +a 2 +a 1 ā 2 F2 = 1+ā 1 a 2

56
Sink (0) Source (1) a1a1 a2a2 E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 0 1 3 0 0 3 Flow and Reparametrization 3ā 1 + 5a 2 + 2a 1 ā 2 = 2(ā 1 +a 2 +a 1 ā 2 ) +ā 1 +3a 2 = 2(1+ā 1 a 2 ) +ā 1 +3a 2 a1a1 a2a2 F1F2 0011 0122 1011 1111 F1 = ā 1 +a 2 +a 1 ā 2 F2 = 1+ā 1 a 2

57
Sink (0) Source (1) a1a1 a2a2 0 1 3 0 0 3 Flow and Reparametrization E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 No more augmenting paths possible

58
Sink (0) Source (1) a1a1 a2a2 0 1 3 0 0 3 Flow and Reparametrization E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 Total Flow Residual Graph (positive coefficients) bound on the optimal solution Inference of the optimal solution becomes trivial because the bound is tight

59
Sink (0) Source (1) a1a1 a2a2 0 1 3 0 0 3 Flow and Reparametrization E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 a 1 = 1 a 2 = 0 E (1,0) = 8 st-mincut cost = 8 Total Flow bound on the optimal solution Inference of the optimal solution becomes trivial because the bound is tight Residual Graph (positive coefficients)

60
Example: Image Segmentation E(x) = c i x i + c ij x i (1-x j ) E: {0,1} n R 0 fg 1 bg ii,j Global Minimum (x * ) x * = arg min E(x) x How to minimize E(x)?

61
How does the code look like? Sink (1) Source (0) Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1)

62
How does the code look like? Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) a1a1 a2a2 fgCost(a 1 ) Sink (1) Source (0) fgCost(a 2 ) bgCost(a 1 ) bgCost(a 2 )

63
Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost(p,q)); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) How does the code look like? a1a1 a2a2 fgCost(a 1 ) Sink (1) Source (0) fgCost(a 2 ) bgCost(a 1 ) bgCost(a 2 ) cost(p,q)

64
Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost(p,q)); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) How does the code look like? a1a1 a2a2 fgCost(a 1 ) Sink (1) Source (0) fgCost(a 2 ) bgCost(a 1 ) bgCost(a 2 ) cost(p,q) a 1 = bg a 2 = fg

65
Image Segmentation in Video Image Flow Global Optimum n-links st-cut s t = 0 = 1 E(x) x*x*

66
Image Segmentation in Video Image Flow Global Optimum

67
Dynamic Energy Minimization EBEB SBSB computationally expensive operation EAEA SASA minimize Recycling Solutions Can we do better? Boykov & Jolly ICCV01, Kohli & Torr (ICCV05, PAMI07)

68
Dynamic Energy Minimization EBEB SBSB computationally expensive operation EAEA SASA minimize cheaper operation Simpler energy E B* differences between A and B similar Kohli & Torr (ICCV05, PAMI07) 3 – 100000 time speedup! Reparametrization Reuse flow Boykov & Jolly ICCV01, Kohli & Torr (ICCV05, PAMI07)

69
Reparametrized Energy Dynamic Energy Minimization Kohli & Torr (ICCV05, PAMI07) Boykov & Jolly ICCV01, Kohli & Torr (ICCV05, PAMI07) E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 Original Energy E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 7a 1 ā 2 + ā 1 a 2 E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 + 5a 1 ā 2 New Energy New Reparametrized Energy

70
Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems

71
Minimizing Energy Functions Space of Function Minimization Problems Submodular Functions NP-Hard General Energy Functions – NP-hard to minimize – Only approximate minimization possible Easy energy functions – Solvable in polynomial time – Submodular ~ O(n 6 ) MAXCUT Functions defined on trees

72
Submodular Set Functions Set function f 2 |E| 2 |E| = #subsets of E Let E= {a 1,a 2,.... a n } be a set

73
Submodular Set Functions Set function f 2 |E| is submodular if E A B f(A) + f(B) f(A B) + f(A B) for all A,B E 2 |E| = #subsets of E Let E= {a 1,a 2,.... a n } be a set Important Property Sum of two submodular functions is submodular

74
Minimizing Submodular Functions Minimizing general submodular functions – O(n 5 Q + n 6 ) where Q is function evaluation time [Orlin, IPCO 2007] Symmetric submodular functions – E (x) = E (1 - x) – O(n 3 ) [Queyranne 1998] Quadratic pseudoboolean – Can be transformed to st-mincut – One node per variable [ O(n 3 ) complexity] – Very low empirical running time

75
Submodular Pseudoboolean Functions All functions for one boolean variable (f: {0,1} -> R) are submodular A function of two boolean variables (f: {0,1} 2 -> R) is submodular if f(0,1) + f(1,0) f(0,0) + f(1,1) A general pseudoboolean function f 2 n is submodular if all its projections f p are submodular i.e. f p (0,1) + f p (1,0) f p (0,0) + f p (1,1) Function defined over boolean vectors x = {x 1,x 2,.... x n } Definition :

76
E(x) = θ i (x i ) + θ ij (x i,x j ) i,ji Quadratic Submodular Pseudoboolean Functions θ ij (0,1) + θ ij (1,0) θ ij (0,0) + θ ij (1,1)For all ij

77
E(x) = θ i (x i ) + θ ij (x i,x j ) i,ji Quadratic Submodular Pseudoboolean Functions θ ij (0,1) + θ ij (1,0) θ ij (0,0) + θ ij (1,1)For all ij E(x) = c i x i + c ij x i (1-x j ) c ij 0 i,ji Equivalent (transformable) i.e. All submodular QPBFs are st-mincut solvable

78
AB CD 0 1 0 1 xixi xjxj = A + 00 C-A 0 1 0 1 0 D-C 0 0 1 0 1 0 B+C -A-D 00 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D 0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i,x j ) = θ ij (0,0) + (θ ij (1,0)-θ ij (0,0)) x i + (θ ij (1,0)-θ ij (0,0)) x j + (θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j

79
AB CD 0 1 0 1 xixi xjxj = A + 00 C-A 0 1 0 1 0 D-C 0 0 1 0 1 0 B+C -A-D 00 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D 0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i,x j ) = θ ij (0,0) + (θ ij (1,0)-θ ij (0,0)) x i + (θ ij (1,0)-θ ij (0,0)) x j + (θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j

80
AB CD 0 1 0 1 xixi xjxj = A + 00 C-A 0 1 0 1 0 D-C 0 0 1 0 1 0 B+C -A-D 00 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D 0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i,x j ) = θ ij (0,0) + (θ ij (1,0)-θ ij (0,0)) x i + (θ ij (1,0)-θ ij (0,0)) x j + (θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j

81
AB CD 0 1 0 1 xixi xjxj = A + 00 C-A 0 1 0 1 0 D-C 0 0 1 0 1 0 B+C -A-D 00 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D 0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i,x j ) = θ ij (0,0) + (θ ij (1,0)-θ ij (0,0)) x i + (θ ij (1,0)-θ ij (0,0)) x j + (θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j

82
AB CD 0 1 0 1 xixi xjxj = A + 00 C-A 0 1 0 1 0 D-C 0 0 1 0 1 0 B+C -A-D 00 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D 0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i,x j ) = θ ij (0,0) + (θ ij (1,0)-θ ij (0,0)) x i + (θ ij (1,0)-θ ij (0,0)) x j + (θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j

83
E(x) = θ i (x i ) + θ ij (x i,x j ) i,ji Quadratic Submodular Pseudoboolean Functions θ ij (0,1) + θ ij (1,0) θ ij (0,0) + θ ij (1,1)For all ij Equivalent (transformable) T S st-mincut x in {0,1} n

84
Minimizing Non-Submodular Functions Minimizing general non-submodular functions is NP-hard. Commonly used method is to solve a relaxation of the problem E(x) = θ i (x i ) + θ ij (x i,x j ) i,ji θ ij (0,1) + θ ij (1,0) θ ij (0,0) + θ ij (1,1) for some ij [Slide credit: Carsten Rother]

85
pairwise nonsubmodular unary pairwise submodular Minimization using Roof-dual Relaxation [Slide credit: Carsten Rother]

86
Double number of variables: Minimization using Roof-dual Relaxation [Slide credit: Carsten Rother]

87
Double number of variables: Minimization using Roof-dual Relaxation Non- submodular Submodular

88
Double number of variables: Minimization using Roof-dual Relaxation is submodular ! Ignore (solvable using st-mincut) Property of the problem :

89
is the optimal label Property of the solution : Double number of variables: Minimization using Roof-dual Relaxation

90
Recap Exact minimization of Submodular QBFs using graph cuts. Obtaining partially optimal solutions of non- submodular QBFs using graph cuts.

91
But... Need higher order energy functions to model image structure – Field of experts [Roth and Black] Many problems in computer vision involve multiple labels E(x) = θ i (x i ) + θ ij (x i,x j ) + θ c (x c ) i,ji c x Labels L = {l 1, l 2, …, l k }Clique c V

92
Transforming problems in QBFs Multi-label Functions Pseudoboolean Functions Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions

93
Transforming problems in QBFs Multi-label Functions Pseudoboolean Functions Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions

94
Higher order to Quadratic Simple Example using Auxiliary variables { 0 if all x i = 0 C 1 otherwise f(x) = min f(x) min C 1 a + C 1 ā x i x L = {0,1} n x = x,a {0,1} Higher Order Submodular Function Quadratic Submodular Function x i = 0 a=0 (ā=1)f(x) = 0 x i 1 a=1 (ā=0)f(x) = C 1

95
Higher order to Quadratic min f(x) min C 1 a + C 1 ā x i x = x,a {0,1} Higher Order Submodular Function Quadratic Submodular Function x i 1 23 C1C1 C 1 x i

96
Higher order to Quadratic min f(x) min C 1 a + C 1 ā x i x = x,a {0,1} Higher Order Submodular Function Quadratic Submodular Function x i 1 23 C1C1 C 1 x i a=1 a=0 Lower envelop of concave functions is concave

97
Higher order to Quadratic min f(x) min f 1 (x)a + f 2 (x)ā x = x,a {0,1} Higher Order Submodular Function Quadratic Submodular Function x i 1 23 a=1 Lower envelop of concave functions is concave f 2 (x) f 1 (x)

98
Higher order to Quadratic min f(x) min f 1 (x)a + f 2 (x)ā x = x,a {0,1} Higher Order Submodular Function Quadratic Submodular Function x i 1 23 a=1 a=0 Lower envelop of concave functions is concave f 2 (x) f 1 (x)

99
Transforming problems in QBFs Multi-label Functions Pseudoboolean Functions Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions

100
Multi-label to Pseudo-boolean So what is the problem? E b (x 1,x 2,..., x m )E m (y 1,y 2,..., y n ) Multi-label ProblemBinary label Problem y i L = {l 1, l 2, …, l k }x i L = {0,1} such that: Let Y and X be the set of feasible solutions, then 1.For each binary solution x X with finite energy there exists exactly one multi-label solution y Y -> One-One encoding function T:X->Y 2. arg min E m (y) = T(arg min E b (x))

101
Multi-label to Pseudo-boolean Popular encoding scheme [Roy and Cox 98, Ishikawa 03, Schlesinger & Flach 06]

102
Multi-label to Pseudo-boolean Popular encoding scheme [Roy and Cox 98, Ishikawa 03, Schlesinger & Flach 06] Ishikawas result: E(y) = θ i (y i ) + θ ij (y i,y j ) i,ji y Labels L = {l 1, l 2, …, l k } θ ij (y i,y j ) = g(|y i -y j |) Convex Function g(|y i -y j |) |y i -y j |

103
Multi-label to Pseudo-boolean Popular encoding scheme [Roy and Cox 98, Ishikawa 03, Schlesinger & Flach 06] Schlesinger & Flach 06: E(y) = θ i (y i ) + θ ij (y i,y j ) i,ji y Labels L = {l 1, l 2, …, l k } θ ij (l i+1,l j ) + θ ij (l i,l j+1 ) θ ij (l i,l j ) + θ ij (l i+1,l j+1 ) Covers all Submodular multi-label functions More general than Ishikawa

104
Multi-label to Pseudo-boolean Applicability – Only solves restricted class of energy functions – Cannot handle Potts model potentials Computational Cost – Very high computational cost – Problem size = |Variables| x |Labels| – Gray level image denoising (1 Mpixel image) (~2.5 x 10 8 graph nodes) Problems

105
Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems

106
St-mincut based Move algorithms Commonly used for solving non-submodular multi-label problems Extremely efficient and produce good solutions Not Exact: Produce local optima E(x) = θ i (x i ) + θ ij (x i,x j ) i,ji x Labels L = {l 1, l 2, …, l k }

107
Move Making Algorithms Solution Space Energy

108
Move Making Algorithms Search Neighbourhood Current Solution Optimal Move Solution Space Energy

109
Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move xcxc (t)(t) Key Property Move Space Bigger move space Solution Space Energy Better solutions Finding the optimal move hard

110
Moves using Graph Cuts Expansion and Swap move algorithms [Boykov Veksler and Zabih, PAMI 2001] Makes a series of changes to the solution (moves) Each move results in a solution with smaller energy Space of Solutions (x) : L N Move Space (t) : 2 N Search Neighbourhood Current Solution N Number of Variables L Number of Labels

111
Moves using Graph Cuts Expansion and Swap move algorithms [Boykov Veksler and Zabih, PAMI 2001] Makes a series of changes to the solution (moves) Each move results in a solution with smaller energy Current Solution Construct a move function Minimize move function to get optimal move Move to new solution How to minimize move functions?

112
General Binary Moves Minimize over move variables t to get the optimal move x = t x 1 + (1- t ) x 2 New solution Current Solution Second solution E m ( t ) = E( t x 1 + (1- t ) x 2 ) Boykov, Veksler and Zabih, PAMI 2001 Move energy is a submodular QPBF (Exact Minimization Possible)

113
Swap Move Variables labeled α, β can swap their labels [Boykov, Veksler, Zabih]

114
Swap Move Sky House Tree Ground Swap Sky, House [Boykov, Veksler, Zabih] Variables labeled α, β can swap their labels

115
Swap Move Variables labeled α, β can swap their labels Move energy is submodular if: – Unary Potentials: Arbitrary – Pairwise potentials: Semimetric [Boykov, Veksler, Zabih] θ ij (l a,l b ) 0 θ ij (l a,l b ) = 0 a = b Examples: Potts model, Truncated Convex

116
Expansion Move [Boykov, Veksler, Zabih] Variables take label or retain current label

117
Expansion Move Sky House Tree Ground Initialize with TreeStatus:Expand GroundExpand HouseExpand Sky [Boykov, Veksler, Zabih] Variables take label or retain current label

118
Expansion Move Move energy is submodular if: – Unary Potentials: Arbitrary – Pairwise potentials: Metric [Boykov, Veksler, Zabih] θ ij (l a,l b ) + θ ij (l b,l c ) θ ij (l a,l c ) Semi metric + Triangle Inequality Variables take label or retain current label Examples: Potts model, Truncated linear Cannot solve truncated quadratic

119
General Binary Moves Move TypeFirst Solution Second Solution Guarantee ExpansionOld solutionAll alphaMetric FusionAny solution Minimize over move variables t x = t x 1 + (1-t) x 2 New solution First solution Second solution Move functions can be non-submodular!!

120
Solving Continuous Problems using Fusion Move x = t x 1 + (1-t) x 2 (Lempitsky et al. CVPR08, Woodford et al. CVPR08) x 1, x 2 can be continuous F x1x1 x2x2 x Optical Flow Example Final Solution Solution from Method 1 Solution from Method 2

121
Range Moves Move variables can be multi-label Optimal move found out by using the Ishikawa Useful for minimizing energies with truncated convex pairwise potentials O. Veksler, CVPR 2007 θ ij (y i,y j ) = min(|y i -y j |,T) |y i -y j | θ ij (y i,y j ) T x = ( t == 1) x 1 + (t == 2) x 2 +… +(t == k) x k

122
Move Algorithms for Solving Higher Order Energies Higher order functions give rise to higher order move energies Move energies for certain classes of higher order energies can be transformed to QPBFs. E(x) = θ i (x i ) + θ ij (x i,x j ) + θ c (x c ) i,ji c x Labels L = {l 1, l 2, …, l k }Clique c V [Kohli, Kumar and Torr, CVPR07] [Kohli, Ladicky and Torr, CVPR08]

123
Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems

124
Solving Mixed Programming Problems x – binary image segmentation (x i {0,1}) ω – non-local parameter (lives in some large set Ω) constant unary potentials pairwise potentials E(x, ω ) = C( ω ) + θ i ( ω, x i ) + θ ij ( ω, x i,x j ) i,ji 0 Rough Shape Prior Stickman Model ω Pose θ i ( ω, x i ) Shape Prior

125
Open Problems Submodular Functions st-mincut Equivalent Characterization of Problems Solvable using st-mincut What functions can be transformed to submodular QBFs?

126
Minimizing General Higher Order Functions We saw how simple higher order potentials can be solved How more sophisticated higher order potentials can be solved?

127
Exact Transformation (global optimum) Or Relaxed transformation (partially optimal) Summary T S st-mincut Labelling Problem Submodular Quadratic Pseudoboolean Function Move making algorithms Sub-problem

128
Thanks. Questions?

129
Use of Higher order Potentials Stereo - Woodford et al. CVPR 2008 P1P1 P2P2 P3P3 5 6 7 8 Pixels Disparity Labels E(x 1,x 2,x 3 ) = θ 12 (x 1,x 2 ) + θ 23 (x 2,x 3 ) θ ij (x i,x j ) = 0 if x i =x j C otherwise { E(6,6,6) = 0 + 0 = 0

130
Use of Higher order Potentials Stereo - Woodford et al. CVPR 2008 P1P1 P2P2 P3P3 5 6 7 8 Pixels Disparity Labels E(x 1,x 2,x 3 ) = θ 12 (x 1,x 2 ) + θ 23 (x 2,x 3 ) θ ij (x i,x j ) = 0 if x i =x j C otherwise { E(6,6,6) = 0 + 0 = 0 E(6,7,7) = 1 + 0 = 1

131
Use of Higher order Potentials Stereo - Woodford et al. CVPR 2008 P1P1 P2P2 P3P3 5 6 7 8 Pixels Disparity Labels Pairwise potential penalize slanted planar surfaces E(x 1,x 2,x 3 ) = θ 12 (x 1,x 2 ) + θ 23 (x 2,x 3 ) θ ij (x i,x j ) = 0 if x i =x j C otherwise { E(6,6,6) = 0 + 0 = 0 E(6,7,7) = 1 + 0 = 1 E(6,7,8) = 1 + 1 = 2

132
Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move x E(x)E(x) xcxc Transformation function T (t)(t) T(x c, t) = x n = x c + t

133
Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move E(x)E(x) xcxc Transformation function T EmEm Move Energy (t)(t) x E m (t) = E(T(x c, t)) T(x c, t) = x n = x c + t

134
Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move E(x)E(x) xcxc E m (t) = E(T(x c, t)) Transformation function T EmEm Move Energy T(x c, t) = x n = x c + t minimize t* Optimal Move (t)(t) x

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google