1. Measuring Uncertainty in Graph Cut Solutions Pushmeet Kohli Philip H.S. Torr Department of Computing Oxford Brookes University

2. Objective Labelling problem t s Graph Cut st-mincut Belief or Confidence Most Probable Solution No uncertainty measure associated with the solution

3. Outline n Inference in Graphical Models n Inference using Graph Cuts n Computing Min-marginals using Graph Cuts n Flow Potentials and Min-marginals n Results

4. Outline n Inference in Graphical Models n Inference using Graph Cuts n Computing Min-marginals using Graph Cuts n Flow Potentials and Min-marginals n Results

5. Inference in Graphical Models x* = arg max Pr(x|D) x E(x|D) = -log Pr(x|D) + constant x* = arg min E(x|D) x x: Set of latent variables D: Observed Data x*: Most Probable (MAP) Solution Pr: Joint Posterior Probability

6. Inference in Graphical Models.2.1.05.35.05.2.01.03.01 A B 012 0 1 2 Joint Distribution Pr(A,B) MAP Solution arg max Pr(A,B) {A=1, B=0}

7. Inference in Graphical Models Marginal - Sum the joint probability over all other variables..2.1.05.35.05.2.01.03.01 A B 012 0 1 2 P(A=1) = 0.6

8. Inference in Graphical Models.2.1.05.35.05.2.01.03.01 A B 012 0 1 2 Max- Marginals (µ) - Maximum joint probability over all other variables. µ A,1 = 0.35

9. Inference in Graphical Models.2.1.05.35.05.2.01.03.01 A B 012 0 1 2 µ A,1 = 0.35 µ A,0 = 0.2 µ A,2 = 0.03 Confidence or Belief (σ) - Normalized max-marginals σ A,1 = µ A,1 / Σ x µ A,x = 0.35/ 0.58 = 0.603

10. Inference in Graphical Models Min-Marginals Energies( ψ ) - Minimize joint energy over all other variables. - Related to max-marginals as: µ j = (1/z)*exp(- ψ j )

11. Inference in Graphical Models Min-Marginals Energies( ψ ) - Minimize joint energy over all other variables. - Related to max-marginals as: - Can be used to compute confidence as: σ j = µ j / Σ a µ a = exp(- ψ i ) / Σ a exp(- ψ a ) µ j = (1/z)*exp(- ψ j )

12. Inference in Graphical Models Min-Marginals Energies( ψ ) - Minimize joint energy over all other variables. - Related to max-marginals as: - Can be used to compute confidence as: How to compute min-marginal energies using Graph Cuts? σ j = µ j / Σ a µ a = exp(- ψ i ) / Σ a exp(- ψ a ) µ j = (1/z)*exp(- ψ j )

13. Inference in Graphical Models Graphical Model Topology TreeGraph with cycles Belief Propagation and variants Exact solution True Marginals/ min-marginals Approximate solution Approximate Marginals/ min-marginals Graph Cuts No Marginals/ Min-Marginals Class 1: Single Max-flow Computation, Exact Solution Class 2: Expansions/ Swap Moves, Approximate Solution

14. Inference in Graphical Models Graphical Model Topology TreeGraph with cycles Belief Propagation and variants Exact solution True Marginals/ min-marginals Approximate solution Approximate Marginals/ min-marginals Graph Cuts No Marginals/ Min-Marginals Class 1: Single Max-flow Computation, Exact Solution Class 2: Expansions/ Swap Moves, Approximate Solution

15. Outline n Inference in Graphical Models n Inference using Graph Cuts n Computing Min-marginals using Graph Cuts n Flow Potentials and Min-marginals n Results

16. Given energy functions E(x|D), Compute: arg min E(x|D) Inference using Graph cuts x

17. Certain E(x|D) can be minimized using graph cuts exactly. Given energy functions E(x|D), Compute: arg min E(x|D) x

18. Inference using Graph cuts Class of energy function and graph contruction - Binary random variables - Submodular functions (Kolmogorov & Zabih, ECCV 2002) - Multi-valued variables - Convex Pair-wise Terms (Ishikawa, PAMI 2003) Certain E(x|D) can be minimized using graph cuts exactly. Given energy functions E(x|D), Compute: arg min E(x|D) x

19. E MRF (a 1,a 2 ) Sink (0) Source (1) a1a1 a2a2 Graph Construction for Binary Random Variables Inference using Graph cuts

20. a1a1 a2a2 E MRF (a 1,a 2 ) = 2a 1 2 t-edges (unary terms) Inference using Graph cuts Sink (0) Source (1)

21. E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 a1a1 a2a2 2 5 Inference using Graph cuts Sink (0) Source (1)

22. E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 a1a1 a2a2 2 5 9 4 Inference using Graph cuts Sink (0) Source (1)

23. E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 a1a1 a2a2 2 5 9 4 2 n-edges (pair-wise term) Inference using Graph cuts Sink (0) Source (1)

24. E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 a1a1 a2a2 2 5 9 4 2 1 Inference using Graph cuts Sink (0) Source (1)

25. E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 a1a1 a2a2 2 5 9 4 2 1 Inference using Graph cuts Sink (0) Source (1)

26. E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 a1a1 a2a2 2 5 9 4 2 1 a 1 = 1 a 2 = 1 E MRF (1,1) = 11 Cost of st-cut = 11 Inference using Graph cuts Sink (0) Source (1)

27. E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 a1a1 a2a2 2 5 9 4 2 1 a 1 = 1 a 2 = 0 E MRF (1,0) = 8 Cost of st-cut = 8 Inference using Graph cuts Sink (0) Source (1)

28. E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 a1a1 a2a2 2 5 9 4 2 1 a 1 = 1 a 2 = 0 E MRF (1,0) = 8 Cost of st-cut = 8 Inference using Graph cuts MAP Solution a 1,map = 1 a 2,map = 0 Sink (0) Source (1)

29. Outline n Inference in Graphical Models n Inference using Graph Cuts n Computing Min-marginals using Graph Cuts n Flow Potentials and Min-marginals n Results

30. Computing Min-marginals using Graph Cuts

31. Instead of minimizing E(.), minimize a projection of E(.) where the value of latent variable x v is fixed to label j. All projections of a sub-modular function are sub-modular [Kolmogorov and Zabih, ECCV 2002]

32. Computing Min-marginals using Graph Cuts Instead of minimizing E(.), minimize a projection of E(.) where the value of latent variable x v is fixed to label j. Problem Solved? Not Really!

33. Computing Min-marginals using Graph Cuts Computing a min-marginal requires computation of a st-cut. Typical Segmentation problem - 640x480 image, 2 labels - Variables = 640x480 = 307200 - Number of Min-marginals = 307200x2 = 614400 - Time taken for 1 graph cut =.3 seconds - Total computation time = 614400x0.3 = 184320 sec = 51.2 hours!!!

34. Energy Projections and Graph Construction E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 a1a1 a2a2 2 5 9 4 2 1 Sink (0) Source (1)

35. Energy Projections and Graph Construction E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 a 2 = 1 E MRF (a 1,1) = 2a 1 + 5ā 1 + 9 + ā 1 a1a1 a2a2 2 5 9 4 2 1 Sink (0) Source (1)

36. Energy Projections and Graph Construction E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 a1a1 a2a2 2 6 9 Sink (0) Source (1) a 2 = 1 E MRF (a 1,1) = 2a 1 + 5ā 1 + 9 + ā 1 E MRF (a 1,1) = 2a 1 + 6ā 1 + 9

37. Energy Projections and Graph Construction E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 + Kā 2 a1a1 a2a2 2 5 9 4 2 1 Sink (0) Source (1) ∞ A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection. Alternative Construction K

38. Energy Projections and Graph Construction a1a1 a2a2 2 5 9 4 2 1 Sink (0) Source (1) A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection. The minimum value of the energy projection can be calculated by using the same graph. Alternative Construction E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 + Kā 2 ∞ K

39. Energy Projections and Graph Construction a1a1 a2a2 2 5 9 4 2 1 Sink (0) Source (1) A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection. The minimum value of the energy projection can be calculated by using the same graph. Alternative Construction K> Sum of Outgoing/Incoming edges E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 + Kā 2 ∞ K

40. Energy Projections and Graph Construction E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 + Kā 2 A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection. The minimum value of the energy projection can be calculated by using the same graph. Results in a small change in the graph. a1a1 a2a2 2 5 9 4 2 1 Sink (0) Source (1) ∞K Alternative Construction

41. Energy Projections and Graph Construction E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 + Kā 2 a1a1 a2a2 2 5 9 4 2 1 Sink (0) Source (1) ∞ A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection. K The minimum value of the energy projection can be calculated by using the same graph. Results in a small change in the graph. Solve using Dynamic Graph Cuts Alternative Construction

42. Dynamic Graph Cuts - Kohli and Torr [ICCV 2005] Graph 1 Graph 2 Energy function Projection of Energy function

43. Dynamic Graph Cuts - Kohli and Torr [ICCV 2005] Graph 1 Graph 2 Energy function Projection of Energy function Graph 1 and 2 are similar

44. Dynamic Graph Cuts - Kohli and Torr [ICCV 2005] Graph 1 Graph 2

45. Dynamic Graph Cuts - Kohli and Torr [ICCV 2005] Graph 1Residual Graph 1st-cut 1 Graph 2Residual Graph 2st-cut 2 Computationally Expensive Procedure Compute Max-flow Compute Max-flow

46. Dynamic Graph Cuts - Kohli and Torr [ICCV 2005] Graph 1Residual Graph 1st-cut 1 Graph 2Residual Graph 2st-cut 2 Re-parameterized Graph 2 Compute Max-flow

47. Dynamic Graph Cuts - Kohli and Torr [ICCV 2005] Graph 1Residual Graph 1st-cut 1 Graph 2 Residual Graph 2st-cut 2 Re-parameterized Graph 2 same solution

48. Dynamic Graph Cuts - Kohli and Torr [ICCV 2005] Graph 1 Residual Graph 1 st-cut 1 Graph 2 Residual Graph 2st-cut 2 Re-parameterized Graph 2 similar edge weights

49. Dynamic Graph Cuts - Kohli and Torr [ICCV 2005] Graph 1Residual Graph 1st-cut 1 Graph 2Residual Graph 2st-cut 2 Re-parameterized Graph 2 Extremely Fast Operation Compute Max-flow Compute Max-flow

50. Dynamic Graph Cuts - Kohli and Torr [ICCV 2005] Graph 1Residual Graph 1st-cut 1 Graph 2Residual Graph 2st-cut 2 Re-parameterized Graph 2 Extremely Fast Operation 300 msec 0.002 msec

51. Summary of the Algorithm Construct graph G for minimizing energy E For computing min-marginals do: - Obtain graph G* ≡ energy function projection E* (By adding constraining edges) - Find the maximum flow in G* using dynamic graph cut algorithm [Kohli and Torr, ICCV 2005]

52. Extension to Multiple Labels Graph construction for multi-label random variables (Ishikawa PAMI 2003) Labels: l 1 …. l n Latent variables: x 1 …. x n x 1 = l 3 x 2 = l 2 MAP Labels x 3 = l 2 x 4 = l 1

53. Extension to Multiple Labels Graph construction for multi-label random variables (Ishikawa PAMI 2003) Labels: l 1 …. l n Latent variables: x 1 …. x n x 4 = l 3 Graph for Projection

54. Outline n Inference in Graphical Models n Inference using Graph Cuts n Computing Min-marginals using Graph Cuts n Flow Potentials and Min-marginals n Results

55. Min-Marginals and Flow Potentials Flow Potentials - maximum amount of flow that can be passed from the node to a particular terminal.

56. Min-Marginals and Flow Potentials 123 4 5 s t 3 29 7 114 3 1 2 11 Graph Flow Potentials - maximum amount of flow that can be passed from the node to a particular terminal.

57. Min-Marginals and Flow Potentials 123 4 5 s t 3 29 7 114 3 1 2 11 Graph Flow Potentials - maximum amount of flow that can be passed from the node to a particular terminal. Source (s) flow potential of node 4

58. Min-Marginals and Flow Potentials 123 4 5 s t 3 29 7 114 3 1 2 11 Graph Flow Potentials - maximum amount of flow that can be passed from the node to a particular terminal. Source (s) flow potential of node 4

59. Min-Marginals and Flow Potentials 123 4 5 s t 3 29 7 114 3 1 2 11 Graph Source (s) flow potential of node 4 123 4 5 s t 11 1 1 Flow Potentials - maximum amount of flow that can be passed from the node to a particular terminal.

60. Min-Marginals and Flow Potentials 123 4 5 s t 3 29 7 114 3 1 2 11 Graph Flow Potentials - maximum amount of flow that can be passed from the node to a particular terminal. Sink (t) flow potential of node 4

61. Min-Marginals and Flow Potentials 123 4 5 s t 3 29 7 114 3 1 2 11 Graph Sink (t) flow potential of node 4 123 4 5 s t 2 29 Flow Potentials - maximum amount of flow that can be passed from the node to a particular terminal.

62. Min-Marginals and Flow Potentials Relationship between Min-marginal of binary latent variables and flow potential. MAP Solution energy Flow potential in residual graph + Min-marginal = For details: See theorem 1. Implications: Any algorithm for computing flow potentials can be used to compute min-marginals.

63. Outline n Inference in Graphical Models n Inference using Graph Cuts n Computing Min-marginals using Graph Cuts n Flow Potentials and Min-marginals n Results

64. Typical Segmentation problem - 640x480 image, 2 labels Computation Times - Naïve Approach = 51.2 hours!!! - Our Algorithm (Dynamic Graph Cuts) = 1.2 seconds Experimental Evaluation

65. Computation Times (in seconds) for binary variables Problem Size (No. of Variables/ Neighbourhood) MAP solution (single maxflow computation) Computing all min-marginals 1x10 5, 4-neighbourhood0.180.70 2x10 5, 4-neighbourhood0.461.34 4x10 5, 4-neighbourhood0.923.15 8x10 5, 4-neighbourhood2.178.21 1x10 5, 8-neighbourhood0.401.53 2x10 5, 8-neighbourhood1.393.59 4x10 5, 8-neighbourhood2.428.50 6x10 5, 8-neighbourhood5.1215.61

66. Min-Marginals in Image segmentation Unary likelihoodContrast Term Uniform Prior (Potts Model) Image Segmentation Energy Boykov and Jolly [ICCV 2001], Blake et al. [ECCV 2004] x i = binary variable representing label (‘fg’ or ‘bg’) of pixel i

67. Min-Marginals in Image segmentation Image MAP SolutionBelief - Foreground Low smoothness High smoothness Moderate smoothness Colour Scale 1 0 0.5 [MSR]

68. Min-Marginals in Image segmentation encourage smoothness Unary likelihoodContrast Term Uniform Prior (Potts Model) Image Segmentation Energy Boykov and Jolly [ICCV 2001], Blake et al. [ECCV 2004]

69. Min-Marginals in Image segmentation Unary likelihoodContrast Term Uniform Prior (Potts Model) Image Segmentation Energy Boykov and Jolly [ICCV 2001], Blake et al. [ECCV 2004] How smoothness effects solutions? encourage smoothness

70. Min-Marginals in Image segmentation Effect of increasing pair-wise terms of the energy function Image

71. Min-Marginals in Image segmentation Effect of increasing pair-wise terms of the energy function MAP Segmentation Foreground Confidence Map

72. Concluding Remarks Efficient method for computing exact min-marginals for certain labelling problems Relationship between Flow-potentials and min-marginals Applications Computing M Most Probable Solutions Min-marginals for parameter learning Hierarchical Segmentation Future Work Efficient min-marginal computation for general problems

73. Thank You