Measuring Uncertainty in Graph Cut Solutions Pushmeet Kohli Philip H.S. Torr Department of Computing Oxford Brookes University

Objective Labelling problem t s Graph Cut st-mincut Belief or Confidence Most Probable Solution No uncertainty measure associated with the solution

Outline n Inference in Graphical Models n Inference using Graph Cuts n Computing Min-marginals using Graph Cuts n Flow Potentials and Min-marginals n Results

Outline n Inference in Graphical Models n Inference using Graph Cuts n Computing Min-marginals using Graph Cuts n Flow Potentials and Min-marginals n Results

Inference in Graphical Models x* = arg max Pr(x|D) x E(x|D) = -log Pr(x|D) + constant x* = arg min E(x|D) x x: Set of latent variables D: Observed Data x*: Most Probable (MAP) Solution Pr: Joint Posterior Probability

Inference in Graphical Models A B Joint Distribution Pr(A,B) MAP Solution arg max Pr(A,B) {A=1, B=0}

Inference in Graphical Models Marginal - Sum the joint probability over all other variables A B P(A=1) = 0.6

Inference in Graphical Models A B Max- Marginals (µ) - Maximum joint probability over all other variables. µ A,1 = 0.35

Inference in Graphical Models A B µ A,1 = 0.35 µ A,0 = 0.2 µ A,2 = 0.03 Confidence or Belief (σ) - Normalized max-marginals σ A,1 = µ A,1 / Σ x µ A,x = 0.35/ 0.58 = 0.603

Inference in Graphical Models Min-Marginals Energies( ψ ) - Minimize joint energy over all other variables. - Related to max-marginals as: µ j = (1/z)*exp(- ψ j )

Inference in Graphical Models Min-Marginals Energies( ψ ) - Minimize joint energy over all other variables. - Related to max-marginals as: - Can be used to compute confidence as: σ j = µ j / Σ a µ a = exp(- ψ i ) / Σ a exp(- ψ a ) µ j = (1/z)*exp(- ψ j )

Inference in Graphical Models Min-Marginals Energies( ψ ) - Minimize joint energy over all other variables. - Related to max-marginals as: - Can be used to compute confidence as: How to compute min-marginal energies using Graph Cuts? σ j = µ j / Σ a µ a = exp(- ψ i ) / Σ a exp(- ψ a ) µ j = (1/z)*exp(- ψ j )

Inference in Graphical Models Graphical Model Topology TreeGraph with cycles Belief Propagation and variants Exact solution True Marginals/ min-marginals Approximate solution Approximate Marginals/ min-marginals Graph Cuts No Marginals/ Min-Marginals Class 1: Single Max-flow Computation, Exact Solution Class 2: Expansions/ Swap Moves, Approximate Solution

Inference in Graphical Models Graphical Model Topology TreeGraph with cycles Belief Propagation and variants Exact solution True Marginals/ min-marginals Approximate solution Approximate Marginals/ min-marginals Graph Cuts No Marginals/ Min-Marginals Class 1: Single Max-flow Computation, Exact Solution Class 2: Expansions/ Swap Moves, Approximate Solution

Outline n Inference in Graphical Models n Inference using Graph Cuts n Computing Min-marginals using Graph Cuts n Flow Potentials and Min-marginals n Results

Given energy functions E(x|D), Compute: arg min E(x|D) Inference using Graph cuts x

Certain E(x|D) can be minimized using graph cuts exactly. Given energy functions E(x|D), Compute: arg min E(x|D) x

Inference using Graph cuts Class of energy function and graph contruction - Binary random variables - Submodular functions (Kolmogorov & Zabih, ECCV 2002) - Multi-valued variables - Convex Pair-wise Terms (Ishikawa, PAMI 2003) Certain E(x|D) can be minimized using graph cuts exactly. Given energy functions E(x|D), Compute: arg min E(x|D) x

E MRF (a 1,a 2 ) Sink (0) Source (1) a1a1 a2a2 Graph Construction for Binary Random Variables Inference using Graph cuts

a1a1 a2a2 E MRF (a 1,a 2 ) = 2a 1 2 t-edges (unary terms) Inference using Graph cuts Sink (0) Source (1)

E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 a1a1 a2a2 2 5 Inference using Graph cuts Sink (0) Source (1)

E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 a1a1 a2a Inference using Graph cuts Sink (0) Source (1)

E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 a1a1 a2a n-edges (pair-wise term) Inference using Graph cuts Sink (0) Source (1)

E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 a1a1 a2a Inference using Graph cuts Sink (0) Source (1)

E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 a1a1 a2a Inference using Graph cuts Sink (0) Source (1)

E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 a1a1 a2a a 1 = 1 a 2 = 1 E MRF (1,1) = 11 Cost of st-cut = 11 Inference using Graph cuts Sink (0) Source (1)

E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 a1a1 a2a a 1 = 1 a 2 = 0 E MRF (1,0) = 8 Cost of st-cut = 8 Inference using Graph cuts Sink (0) Source (1)

E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 a1a1 a2a a 1 = 1 a 2 = 0 E MRF (1,0) = 8 Cost of st-cut = 8 Inference using Graph cuts MAP Solution a 1,map = 1 a 2,map = 0 Sink (0) Source (1)

Outline n Inference in Graphical Models n Inference using Graph Cuts n Computing Min-marginals using Graph Cuts n Flow Potentials and Min-marginals n Results

Computing Min-marginals using Graph Cuts

Instead of minimizing E(.), minimize a projection of E(.) where the value of latent variable x v is fixed to label j. All projections of a sub-modular function are sub-modular [Kolmogorov and Zabih, ECCV 2002]

Computing Min-marginals using Graph Cuts Instead of minimizing E(.), minimize a projection of E(.) where the value of latent variable x v is fixed to label j. Problem Solved? Not Really!

Computing Min-marginals using Graph Cuts Computing a min-marginal requires computation of a st-cut. Typical Segmentation problem - 640x480 image, 2 labels - Variables = 640x480 = Number of Min-marginals = x2 = Time taken for 1 graph cut =.3 seconds - Total computation time = x0.3 = sec = 51.2 hours!!!

Energy Projections and Graph Construction E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 a1a1 a2a Sink (0) Source (1)

Energy Projections and Graph Construction E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 a 2 = 1 E MRF (a 1,1) = 2a 1 + 5ā ā 1 a1a1 a2a Sink (0) Source (1)

Energy Projections and Graph Construction E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 a1a1 a2a Sink (0) Source (1) a 2 = 1 E MRF (a 1,1) = 2a 1 + 5ā ā 1 E MRF (a 1,1) = 2a 1 + 6ā 1 + 9

Energy Projections and Graph Construction E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 + Kā 2 a1a1 a2a Sink (0) Source (1) ∞ A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection. Alternative Construction K

Energy Projections and Graph Construction a1a1 a2a Sink (0) Source (1) A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection. The minimum value of the energy projection can be calculated by using the same graph. Alternative Construction E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 + Kā 2 ∞ K

Energy Projections and Graph Construction a1a1 a2a Sink (0) Source (1) A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection. The minimum value of the energy projection can be calculated by using the same graph. Alternative Construction K> Sum of Outgoing/Incoming edges E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 + Kā 2 ∞ K

Energy Projections and Graph Construction E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 + Kā 2 A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection. The minimum value of the energy projection can be calculated by using the same graph. Results in a small change in the graph. a1a1 a2a Sink (0) Source (1) ∞K Alternative Construction

Energy Projections and Graph Construction E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 + Kā 2 a1a1 a2a Sink (0) Source (1) ∞ A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection. K The minimum value of the energy projection can be calculated by using the same graph. Results in a small change in the graph. Solve using Dynamic Graph Cuts Alternative Construction

Dynamic Graph Cuts - Kohli and Torr [ICCV 2005] Graph 1 Graph 2 Energy function Projection of Energy function

Dynamic Graph Cuts - Kohli and Torr [ICCV 2005] Graph 1 Graph 2 Energy function Projection of Energy function Graph 1 and 2 are similar

Dynamic Graph Cuts - Kohli and Torr [ICCV 2005] Graph 1 Graph 2

Dynamic Graph Cuts - Kohli and Torr [ICCV 2005] Graph 1Residual Graph 1st-cut 1 Graph 2Residual Graph 2st-cut 2 Computationally Expensive Procedure Compute Max-flow Compute Max-flow

Dynamic Graph Cuts - Kohli and Torr [ICCV 2005] Graph 1Residual Graph 1st-cut 1 Graph 2Residual Graph 2st-cut 2 Re-parameterized Graph 2 Compute Max-flow

Dynamic Graph Cuts - Kohli and Torr [ICCV 2005] Graph 1Residual Graph 1st-cut 1 Graph 2 Residual Graph 2st-cut 2 Re-parameterized Graph 2 same solution

Dynamic Graph Cuts - Kohli and Torr [ICCV 2005] Graph 1 Residual Graph 1 st-cut 1 Graph 2 Residual Graph 2st-cut 2 Re-parameterized Graph 2 similar edge weights

Dynamic Graph Cuts - Kohli and Torr [ICCV 2005] Graph 1Residual Graph 1st-cut 1 Graph 2Residual Graph 2st-cut 2 Re-parameterized Graph 2 Extremely Fast Operation Compute Max-flow Compute Max-flow

Dynamic Graph Cuts - Kohli and Torr [ICCV 2005] Graph 1Residual Graph 1st-cut 1 Graph 2Residual Graph 2st-cut 2 Re-parameterized Graph 2 Extremely Fast Operation 300 msec msec

Summary of the Algorithm Construct graph G for minimizing energy E For computing min-marginals do: - Obtain graph G* ≡ energy function projection E* (By adding constraining edges) - Find the maximum flow in G* using dynamic graph cut algorithm [Kohli and Torr, ICCV 2005]

Extension to Multiple Labels Graph construction for multi-label random variables (Ishikawa PAMI 2003) Labels: l 1 …. l n Latent variables: x 1 …. x n x 1 = l 3 x 2 = l 2 MAP Labels x 3 = l 2 x 4 = l 1

Extension to Multiple Labels Graph construction for multi-label random variables (Ishikawa PAMI 2003) Labels: l 1 …. l n Latent variables: x 1 …. x n x 4 = l 3 Graph for Projection

Outline n Inference in Graphical Models n Inference using Graph Cuts n Computing Min-marginals using Graph Cuts n Flow Potentials and Min-marginals n Results

Min-Marginals and Flow Potentials Flow Potentials - maximum amount of flow that can be passed from the node to a particular terminal.

Min-Marginals and Flow Potentials s t Graph Flow Potentials - maximum amount of flow that can be passed from the node to a particular terminal.

Min-Marginals and Flow Potentials s t Graph Flow Potentials - maximum amount of flow that can be passed from the node to a particular terminal. Source (s) flow potential of node 4

Min-Marginals and Flow Potentials s t Graph Flow Potentials - maximum amount of flow that can be passed from the node to a particular terminal. Source (s) flow potential of node 4

Min-Marginals and Flow Potentials s t Graph Source (s) flow potential of node s t Flow Potentials - maximum amount of flow that can be passed from the node to a particular terminal.

Min-Marginals and Flow Potentials s t Graph Flow Potentials - maximum amount of flow that can be passed from the node to a particular terminal. Sink (t) flow potential of node 4

Min-Marginals and Flow Potentials s t Graph Sink (t) flow potential of node s t 2 29 Flow Potentials - maximum amount of flow that can be passed from the node to a particular terminal.

Min-Marginals and Flow Potentials Relationship between Min-marginal of binary latent variables and flow potential. MAP Solution energy Flow potential in residual graph + Min-marginal = For details: See theorem 1. Implications: Any algorithm for computing flow potentials can be used to compute min-marginals.

Outline n Inference in Graphical Models n Inference using Graph Cuts n Computing Min-marginals using Graph Cuts n Flow Potentials and Min-marginals n Results

Typical Segmentation problem - 640x480 image, 2 labels Computation Times - Naïve Approach = 51.2 hours!!! - Our Algorithm (Dynamic Graph Cuts) = 1.2 seconds Experimental Evaluation

Computation Times (in seconds) for binary variables Problem Size (No. of Variables/ Neighbourhood) MAP solution (single maxflow computation) Computing all min-marginals 1x10 5, 4-neighbourhood x10 5, 4-neighbourhood x10 5, 4-neighbourhood x10 5, 4-neighbourhood x10 5, 8-neighbourhood x10 5, 8-neighbourhood x10 5, 8-neighbourhood x10 5, 8-neighbourhood

Min-Marginals in Image segmentation Unary likelihoodContrast Term Uniform Prior (Potts Model) Image Segmentation Energy Boykov and Jolly [ICCV 2001], Blake et al. [ECCV 2004] x i = binary variable representing label (‘fg’ or ‘bg’) of pixel i

Min-Marginals in Image segmentation Image MAP SolutionBelief - Foreground Low smoothness High smoothness Moderate smoothness Colour Scale [MSR]

Min-Marginals in Image segmentation encourage smoothness Unary likelihoodContrast Term Uniform Prior (Potts Model) Image Segmentation Energy Boykov and Jolly [ICCV 2001], Blake et al. [ECCV 2004]

Min-Marginals in Image segmentation Unary likelihoodContrast Term Uniform Prior (Potts Model) Image Segmentation Energy Boykov and Jolly [ICCV 2001], Blake et al. [ECCV 2004] How smoothness effects solutions? encourage smoothness

Min-Marginals in Image segmentation Effect of increasing pair-wise terms of the energy function Image

Min-Marginals in Image segmentation Effect of increasing pair-wise terms of the energy function MAP Segmentation Foreground Confidence Map

Concluding Remarks Efficient method for computing exact min-marginals for certain labelling problems Relationship between Flow-potentials and min-marginals Applications Computing M Most Probable Solutions Min-marginals for parameter learning Hierarchical Segmentation Future Work Efficient min-marginal computation for general problems

Thank You