Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor Ladicky Philip Torr Oxford Brookes University, Oxford CVPR 2008

Image labelling Problems Assign a label to each image pixel Geometry Estimation Image Denoising Object Segmentation Sky Building Tree Grass

Object Segmentation using CRFs (Shotton et al. ECCV 2006) CRF Energy Unary potentials based on Colour, Location and Texture features Encourages label consistency in adjacent pixels

Limitations of Pairwise CRFs Encourages short boundaries (Shrinkage bias) Can only enforce label consistency in adjacent pixels Inability to incorporate region based features Image Unary Potential MAP-CRF Solution

Label Consistency in Image Regions Pixels constituting some regions belong to Same plane (Orientation) (Hoiem, Efros, & Herbert, ICCV’05) Same object (Russel, Efros, Sivic, Freeman, & Zisserman, CVPR06) Image (MSRC) Segmentation (Mean shift)

Image labelling using segments Unsupervised Segmentation Object Labelling Geometric Context [Hoiem et al, ICCV05] Object Segmentation [He et al. ECCV06, Yang et al. CVPR07, Rabinovich et al. ICCV07, Batra et al. CVPR08] Interactive Video Segmentation [Wang, SIGGRAPH 2005 ] Not robust to Inconsistent Segments!

Our Higher Order CRF Model Encourages label consistency in regions Multiple Segmentations c

Label Consistency in Segments Encourages consistency within super-pixels Takes the form of a PN Potts model [Kohli et al. CVPR 2007] c

Label Consistency in Segments Encourages consistency within super-pixels Takes the form of a PN Potts model [Kohli et al. CVPR 2007] c Cost: 0

Label Consistency in Segments Encourages consistency within super-pixels Takes the form of a PN Potts model [Kohli et al. CVPR 2007] c Cost: f (|c|)

Label Consistency in Segments Encourages consistency within super-pixels Takes the form of a PN Potts model [Kohli et al. CVPR 2007] Does not distinguish between Good/Bad Segments ! c Cost: f (|c|)

Quality based Label Consistency Label inconsistency cost depends on segment quality How to measure quality G(c)? [Ren and Malik ICCV03, Rabinovich et al. ICCV07, many others] Colour and Texture Similarity Contour Energy Measure quality from variance in feature responses Higher order generalization of contrast-sensitive pairwise potential

Quality based Label Consistency Mean shift segmentation Segment Quality (darker is better) MSRC image

Robust Consistency Potentials gmax PN Potts 1 Too Rigid! Inconsistent Pixels gmax 1 T Robust Inconsistent Pixels

Robust Consistency Potentials Maximum Inconsistency Cost Number of Inconsistent Pixels Slope gmax 1 T Robust Inconsistent Pixels

Minimizing Higher order Energy Functions Message passing is computationally expensive High runtime and space complexity - O(LN) L = Number of Labels, N = Size of Clique Efficient BP for Higher Order MRFs [Lan et al. ECCV 06, Potetz CVPR 2007] 2x2 clique potentials for Image Denoising Take minutes per iteration (Hours to converge)

Minimizing Higher order Energy Functions Graph Cut based move making algorithm [Kohli et al. CVPR 2007] Can handle very high order energy functions Extremely efficient: computation time in the order of seconds Only applicable to some classes of functions (PN Potts) Cannot handle robust consistency potential This paper Can minimize a much larger class of higher order energy functions Same time complexity as [Kohli et al. CVPR 2007]

Move making algorithms Expansion and Swap move algorithms [Boykov Veksler and Zabih, PAMI 2001] Makes a series of changes to the solution (moves) Each move results in a solution with smaller energy Current Solution How to minimize move functions? Move to new solution Generate pseudo-boolean move function Minimize move function to get optimal move

Minimizing Move Functions using Graph Cuts Most pairwise CRF models used in Computer Vision lead to submodular move functions Second order Pseudo-boolean Function Minimization (submodular) st-mincut (Positive weights) Optimal moves can be found extremely efficiently using graphs cuts

Minimizing Higher Order Energy Functions: Our results We show that a large class of higher order potentials lead to higher order submodular move functions Can be minimized in polynomial time Submodular Function Minimization Minimizing general submodular functions is computationally expensive Complexity O(n6) Cannot handle large problems! Details in Technical Report

Minimizing Higher Order Energy Functions: Our results Minimizing Higher order functions using Graph cuts Higher order functions can be transformed to second order functions by adding auxillary variables Exponential number of auxillary variables needed in general Result 2 Our higher order functions can be transformed to second-order functions using ≤2 auxillary variables per potential. Can be minimized extremely efficiently Complexity << O(n6) Details in Technical Report

Overview of our Method + + Higher Order Energy Segmentation Solution Unary Potentials [Shotton et al. ECCV 2006] + Energy Minimization Contrast Sensitive Pairwise Potentials + Segmentation Solution Higher Order Potentials (Multiple Segmentations)

Experimental results Datasets: MSRC (21), Sowerby (7) [Shotton et al. ECCV 2006] [He et al. CVPR 04]

Qualitative Results Image (MSRC-21) Pairwise CRF Higher order CRF Ground Truth Grass Sheep

Qualitative Results (Contd..) Image (MSRC-21) Pairwise CRF Higher order CRF Ground Truth Results can be improved using image specific colour models Rother et al. SIGGRAPH 2004 Shotton et al. ECCV 2006

Quantitative Results: Problems Rough ground truth segmentations Fine structures have small influence on overall pixel accuracy

Generating Accurate Segmentations Generated accurate segmentation of 27 images 30 minutes per image Image (MSRC-21) Original Segmentation New Segmentation

Relationship between Qualitative and Quantitative Results Pairwise CRF Higher order CRF Ground Truth Image (MSRC-21) Overall Pixel Accuracy 95.8% 98.7% Small changes in pixel accuracy can lead to large improvements in segmentation results.

Quantitative Accuracy Measure accuracy in labelling boundary pixels. Accuracy evaluated in boundary bands of variable width Hand-labelled Segmentation Trimap (8-pixels) Trimap (16-pixels) Image (MSRC-21)

Quantitative Accuracy Measure accuracy in labelling boundary pixels. Accuracy evaluated in boundary bands of variable width

Conclusions Method to enforce label consistency in image regions Generalization of the commonly used Pairwise CRF model Allows integration of pixel and region level features for image labelling problems

Thanks

Number of Segmentations Running Time Results Time (sec) Number of Segmentations Inconsistency Cost

Qualitative Results (Contd..) Image (MSRC-21) Pairwise CRF Higher order CRF Ground Truth

Transformation to second order functions Auxiliary variables