Associative Hierarchical CRFs for Object Class Image Segmentation Ľubor Ladický 1 1 Oxford Brookes University 2 Microsoft Research Cambridge Based on the work done with : Chris Russell 1, Pushmeet Kohli 1,2, Paul Sturgess 1, Karteek Alahari 1, Philip H.S. Torr 1

Overview Object-class Segmentation problem Datasets MSRC dataset VOC2009 dataset CamVid dataset Standard approaches Pairwise CRF over Pixels Pairwise CRF over Segments Detection-driven Segmentation Model Robust-P N model Associative Hierarchical CRF Detectors in AHCRF Label co-occurrence in CRFs Potential training Inference Results Discussion General overviewOur approach

Overview Object-class Segmentation problem Datasets MSRC dataset VOC2009 dataset CamVid dataset Standard approaches Pairwise CRF over Pixels Pairwise CRF over Segments Detection-driven Segmentation Model Robust-P N model Associative Hierarchical CRF Detectors in AHCRF Label co-occurrence in CRFs Potential training Inference Results Discussion General overviewOur approach

Object-class Segmentation problem MSRC ImageOur Result Aims to assign a label for each pixel of an image Classifier trained on the training set Performance evaluated on never seen test set

Overview Object-class Segmentation problem Datasets MSRC dataset VOC2009 dataset CamVid dataset Standard approaches Pairwise CRF over Pixels Pairwise CRF over Segments Detection-driven Segmentation Model Robust-P N model Associative Hierarchical CRF Detectors in AHCRF Label co-occurrence in CRFs Potential training Inference Results Discussion General overviewOur approach

Datasets MSRC Dataset 591 images 320 x classes MSRC Image Ground Truth

Datasets VOC2009 Dataset 1499 training test images 500 x foreground classes + 1 background class VOC Image Ground Truth

Datasets CamVid Dataset 367 training test video frames 960 x classes CamVid Image Ground Truth

Overview Object-class Segmentation problem Datasets MSRC dataset VOC2009 dataset CamVid dataset Standard approaches Pairwise CRF over Pixels Pairwise CRF over Segments Detection-driven Segmentation Model Robust-P N model Associative Hierarchical CRF Detectors in AHCRF Label co-occurrence in CRFs Potential training Inference Results Discussion General overviewOur approach

Pairwise CRF over Pixels CRF Shotton ECCV06

Pairwise CRF over Pixels No quantization errors Lacks long range interactions Results oversmoothed Better performance on CamVid & MSRC datasets

Pairwise CRF over Segments Yang et al. CVPR07, Batra et al. CVPR08, Shi, Malik PAMI2000, Comaniciu, Meer PAMI2002, Felzenschwalb, Huttenlocher, IJCV2004

Pairwise CRF over Segments Allows long range interactions Better performance for VOC dataset Can not recover from incorrect segmentation Impossible to obtain perfect unsupervised segmentation

Pairwise CRF over Segments Allows long range interactions Better performance for VOC dataset Can not recover from incorrect segmentation Impossible to obtain perfect unsupervised segmentation

Detection-driven Segmentation Larlus et al. CVPR08, Felzenszwalb et al. CVPR08, Vedaldi et al. ICCV09 GrabCut Detector

Detection-driven Segmentation The best performance for VOC dataset Can not be used for background classes Can not recover from incorrect detection

Overview Object-class Segmentation problem Datasets MSRC dataset VOC2009 dataset CamVid dataset Standard approaches Pairwise CRF over Pixels Pairwise CRF over Segments Detection-driven Segmentation Model Robust-P N model Associative Hierarchical CRF Detectors in AHCRF Label co-occurrence in CRFs Potential training Inference Results Discussion General overviewOur approach

Robust P N approach Kohli, Ladický, Torr CVPR08

Robust P N approach Segment consistency as a weak constraint Robust to misleading segmentations Allows multiple overlapping segmentations General formulation not used in application Unary and pairwise potentials only at the pixel level

Robust P N reformulation Ladický, Russell, Kohli, Torr ICCV09

Old formulation of Robust P N potential is equivalent to pairwise formulation where Robust P N reformulation Ladický, Russell, Kohli, Torr ICCV09

Associative Hierarchical CRF Allows unary potentials for region variables Allows pairwise potentials for region variables Allows multiple layers and multiple hierarchies Zhu et al. NIPS2008, Lim et al. ICCV2009, Hinton 2002

Analysis of the new model Ladický, Russell, Kohli, Torr ICCV09 Let's have one segmentation and potentials only over segment level

Analysis of the new model Let's have one segmentation and potentials only over segment level Energy of two pixels from the same clique is symmetric and semi-metric Minimum will be segment-consistent The cost of every segment consistent labelling is the same as the cost of the pairwise CRF labelling over segments Equivalent to pairwise CRF over segments Ladický, Russell, Kohli, Torr ICCV09

Associative Hierarchical CRF Merges information over multiple scales Allows multiple hierarchies Allows long range interactions Easy to learn weights Interlayer connection limited(?) to associative relationship Ladický, Russell, Kohli, Torr ICCV09

Comparison to other methods Ladický, Russell, Kohli, Torr ICCV09 Robust P N Segment CRFHierarchical CRF

Detectors in Associative Hierarchical CRFs Ladický, Alahari, Sturgess, Russell, Torr CVPR10 (submitted) GrabCutDetector Detector potential takes the form where

Detectors in Associative Hierarchical CRFs Ladický, Alahari, Sturgess, Russell, Torr CVPR10 (submitted) Special 2-label case of Robust P N Allows multiple overlapping detections Can recover from incorrect detection Same inference methods may be applied as for Associative Hierarchical CRFs We can distinguish between different instances of objects in the final result

Label co-occurrence in CRFs Ladický, Russell, Kohli, Torr CVPR10 (submitted) Possible labelling ? Why not ?

Label co-occurrence in CRFs Ladický, Russell, Kohli, Torr CVPR10 (submitted) Our requirements Global Energy function – no hard decisions Invariance to number of pixels taking given label Efficiency – should be tractable Standard approaches Hard decisions in the preprocessing step (Csurka et al. BMVC08) Unary potential (Torralba et al. ICCV03) Pairwise potential (Rabinovich et al. ICCV07, CVPR08) Only solution : E C = C(L(x))

Overview Object-class Segmentation problem Datasets MSRC dataset VOC2009 dataset CamVid dataset Standard approaches Pairwise CRF over Pixels Pairwise CRF over Segments Detection-driven Segmentation Model Robust-P N model Associative Hierarchical CRF Detectors in AHCRF Label co-occurrence in CRFs Potential training Inference Results Discussion General overviewOur approach

Multi-feature Boosted Unary Pixel Potential Ladický, Russell, Kohli, Torr ICCV09 Sturgess, Alahari, Ladický, Torr BMVC09 Unary likelihoods based on spatial configuration (Shotton et al. ECCV06) Classifier trained using boosting

Multi-feature Boosted Unary Segment Potential Ladický, Russell, Kohli, Torr ICCV09 Classifier trained using boosting

Other Potentials Ladický, Russell, Kohli, Torr ICCV09 Ladický, Russell, Kohli, Torr CVPR10 (submitted) Ladický, Alahari, Sturgess, Russell, Torr CVPR10 (submitted) Intensity dependent pairwise pixel potential Colour EMD-distance dependent pairwise segment potential Detector potentials based on off-the-shelf detectors (Felzenszwalb et al. CVPR08, Vedaldi et al. ICCV09) Generatively trained Co-occurence potential

Overview Object-class Segmentation problem Datasets MSRC dataset VOC2009 dataset CamVid dataset Standard approaches Pairwise CRF over Pixels Pairwise CRF over Segments Detection-driven Segmentation Model Robust-P N model Associative Hierarchical CRF Detectors in AHCRF Label co-occurrence in CRFs Potential training Inference Results Discussion General overviewOur approach

Inference over Hierarchical CRF Russell, Ladický, Kohli, Torr AISTATS10 (submitted)

Inference over Hierarchical CRF Problem is NP-hard Any message passing algorithm (TRW-S, BP,..) or ICM can be applied to pairwise model αβ-swap (potentials must be semi-metric) Ishikawa construction over (α-F-β transition) αexpansion (potentials must be metric) Reparametrization of interlayer connection to metric potential Ishikawa construction over (α-F-old transition) For more details read our technical report Russell, Ladický, Kohli, Torr AISTATS10 (submitted)

Inference over Hierarchical CRF Problem is NP-hard Any message passing algorithm (TRW-S, BP,..), ICM or LP relaxation can be applied to pairwise model αβ-swap (potentials must be semi-metric) Ishikawa construction over (α-F-β transition) αexpansion (potentials must be metric) Reparametrization of interlayer connection to metric potential Ishikawa construction over (α-F-old transition) For more details read our technical report Russell, Ladický, Kohli, Torr AISTATS10 (submitted)

Inference over Hierarchical CRF Problem is NP-hard Any message passing algorithm (TRW-S, BP,..), ICM or LP relaxation can be applied to pairwise model αβ-swap (potentials must be semi-metric) Ishikawa construction over (α-F-β transition) αexpansion (potentials must be metric) Reparametrization of interlayer connection to metric potential Ishikawa construction over (α-F-old transition) For more details read our technical report Russell, Ladický, Kohli, Torr AISTATS10 (submitted)

Inference over Hierarchical CRF Problem is NP-hard Any message passing algorithm (TRW-S, BP,..), ICM or LP relaxation can be applied to pairwise model αβ-swap (potentials must be semi-metric) Ishikawa construction over (α-F-β transition) α-expansion (potentials must be metric) Reparametrization of interlayer connection to metric potential Ishikawa construction over (α-F-old transition) Russell, Ladický, Kohli, Torr AISTATS10 (submitted)

Inference over Hierarchical CRF Russell, Ladický, Kohli, Torr AISTATS10 (submitted)

Inference for Co-occurence Integer programming formulation possible using one variable for each subset of the label set LP relaxation applicable by relaxing IP program αβ-swap possible if the cooccurence cost C(L(x)) is monotonically increasing with respect to L(x) α-expansion approximate move possible if the cooccurence cost C(L(x)) is monotonically increasing with respect to L(x) Ladický, Russell, Kohli, Torr CVPR10 (submitted)

Overview Object-class Segmentation problem Datasets MSRC dataset VOC2009 dataset CamVid dataset Standard approaches Pairwise CRF over Pixels Pairwise CRF over Segments Detection-driven Segmentation Model Robust-P N model Associative Hierarchical CRF Detectors in AHCRF Label co-occurrence in CRFs Potential training Inference Results Discussion General overviewOur approach

VOC2009 Results

MSRC Results

Qualitative comparison of results without and with co-occurence Ladický, Russell, Kohli, Torr CVPR10 (submitted) MSRC Image Without COWith CO Without CO

CamVid Results Sturgess, Alahari Ladický, Torr BMVC09 Ladický, Alahari, Sturgess, Russell, Torr CVPR10 (submitted) Our Result CamVid Image Ground Truth Brostow et al.

Qualitative comparisons Quantitative comparison of methods on VOC2009 dataset Quantitative comparison of methods on MSRC dataset Quantitative comparison of methods on CamVid dataset

Take home message

Use our model

Take home message Use our model Use your favourite potentials

Take home message Use our model Use your favourite potentials Use your friend's favourite potentials

Take home message Use our model Use your favourite potentials Use your friend's favourite potentials Use your friend's friend's favourite potentials

Take home message Use our model Use your favourite potentials Use your friend's favourite potentials Use your friend's friend's favourite potentials Vision solved

Take home message Use our model Use your favourite potentials Use your friend's favourite potentials Use your friend's friend's favourite potentials Vision solved (..almost)

Thank you Questions?