Contour Based Approaches for Visual Object Recognition Jamie Shotton University of Cambridge Joint work with Roberto Cipolla, Andrew Blake

Recent Related Work Recognition Recognition Constellation model – e.g. Fergus et al. [CVPR 03] and Li et al. [ICCV 03] Constellation model – e.g. Fergus et al. [CVPR 03] and Li et al. [ICCV 03] Deformable shape models – Berg et al. [CVPR 05] Deformable shape models – Berg et al. [CVPR 05] Segmentation Segmentation Top-down/bottom-up segmentation – Borenstein et al. [POCV 04] Top-down/bottom-up segmentation – Borenstein et al. [POCV 04] Joint recognition and segmentation Joint recognition and segmentation Implicit Shape Model – Leibe et al. [CVPR 05] Implicit Shape Model – Leibe et al. [CVPR 05] LOCUS – Winn & Jojic [ICCV 05] LOCUS – Winn & Jojic [ICCV 05] Obj Cut – Kumar et al. [ICCV 05] Obj Cut – Kumar et al. [ICCV 05] Multi-Class Recognition and Segmentation Multi-Class Recognition and Segmentation Multiscale CRFs – He et al. [CVPR 04] Multiscale CRFs – He et al. [CVPR 04] pLSA-based models – e.g. Sivic et al. and Fergus et al. [ICCV 05] pLSA-based models – e.g. Sivic et al. and Fergus et al. [ICCV 05] Hierarchical field framework – Kumar & Hebert [ICCV 05] Hierarchical field framework – Kumar & Hebert [ICCV 05]

Contour-Based Learning Goal – single class categorical recognition Goal – single class categorical recognition learn to detect and localise objects learn to detect and localise objects “find the car, face or horse” “find the car, face or horse” How can we exploit object contour? How can we exploit object contour? Desired detection results Our contribution

Contour Features Features Features contour fragments contour fragments and their parameters and their parameters Local features not whole contour Local features not whole contour account for variability separately account for variability separately increase generalisation increase generalisation decrease training requirements decrease training requirements

Object Model p σ T Model is set of M features Model is set of M features star constellation star constellation Each feature Each feature contour fragment expected offsetmodel parametersclassifier parameters

Matching Features Canny Edge Detector Distance Transform Gaussian weighted oriented chamfer matching Gaussian weighted oriented chamfer matching aligns features to image aligns features to image

Matching Features Gaussian weighted oriented chamfer matching Gaussian weighted oriented chamfer matching aligns features to image aligns features to image Chamfer Matching feature match score at optimal position optimal position

confidence weighted weak learner Location Sensitive Classification Feature match scores make detection simple Feature match scores make detection simple Detection uses a boosted classification function K(c): Detection uses a boosted classification function K(c): M number of features FmFmFmFm feature m E canny edge map c object centroid match scorethresholded match score  m weak learner threshold amamamam weak learner confidence bmbmbmbm  0-1 indicator function

Evaluate K(c) for all c gives a classification map Evaluate K(c) for all c gives a classification map confidence as function of position confidence as function of position Globally thresholded local maxima give final detections Globally thresholded local maxima give final detections Object Detection test image classification map contours object no object

Learning System Detection Boosting Algorithm K(c)K(c) Segmented Training Data Test Data Object Detections Background Training Data

Training Data

Boot-Strapping Learn detector K 1 (c) Learn detector K 1 (c) segmented training data segmented training data Evaluate detector K 1 (c) on Evaluate detector K 1 (c) on unsegmented class images unsegmented class images locates object centroids locates object centroids background images background images locates clutter locates clutter +

Learning System Detection Boosting Algorithm K1(c) K1(c) Segmented Training Data Unsegmented Training Data Detection Object Detections Boosting Algorithm K 2 (c) Test Data Background Training Data

Building a Fragment Dictionary …… Masks (~10 images) Contour Fragments T n (~1000 fragments) ……

Training Examples Learn classifier K(c) by boosting from Learn classifier K(c) by boosting from feature vectors x feature vectors x target values y (object/background) target values y (object/background) Encourage ‘good’ classification map: Encourage ‘good’ classification map: Take training examples at: Take training examples at: object no object + -

Boosting as Feature Selection Feature vectors Feature vectors 1000 random fragments 50 discriminative fragments Fragment Selection 1. Fragment Selection Model Parameter Estimation 2. Model Parameter Estimation Select ,  for each feature Weak-Learner Estimation 3. Weak-Learner Estimation Select  a, b for each feature FkFkFkFk candidate feature (fragment T 2 T, parameters  2 , 2  ) N number of candidate features = |T| x |  | x |  | EiEiEiEi canny edge map I cjcjcjcj example centroid j in image i

Learning System Detection Boosting Algorithm K 1 (c) Segmented Training Data Unsegmented Training Data Detection Object Detections Boosting Algorithm K 2 (c) Test Data Background Training Data

Contour Experiments Datasets: Datasets: Weizmann Horses Weizmann Horses UIUC Cars UIUC Cars Caltech Faces Caltech Faces Caltech Motorbikes Caltech Motorbikes Caltech Background Caltech Background Each category evaluated in turn Each category evaluated in turn 10 segmented training images 10 segmented training images 40 unsegmented training images 40 unsegmented training images 50 background images 50 background images single scale evaluation single scale evaluation

Contour Results

Recall Precision equal error rates Recall Precision equal error rates Weizmann Horses: 92.1% Weizmann Horses: 92.1% UIUC Cars: 92.8% UIUC Cars: 92.8% Caltech Faces: 94.0% Caltech Faces: 94.0% Caltech Motorbikes: 92.4% Caltech Motorbikes: 92.4% HorsesCars

Contour Results Occlusion Performance (horses)Performance of K 1 vs. K 2 (faces) No. Segmented Training Images

Conclusions Contour is very powerful cue Contour is very powerful cue Boot-strapping improves results Boot-strapping improves results Future directions Future directions extend to multiple classes, scales, views extend to multiple classes, scales, views segmentation segmentation