Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton.

Similar presentations


Presentation on theme: "The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton."— Presentation transcript:

1 The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton

2 LayoutCRF contributions Detection and segmentation Handles occlusion and deformation Multiple objects simultaneously Multiple classes

3 Related work Layout consistency Layout Consistent Random Field Results Roadmap

4 Related work: constellation models X [Crandall et al. ECCV 2006] [Fergus et al. CVPR 2003][Leibe et al. ECCV 2004] [Kumar et al. CVPR 2005] …

5 Related work: constellation models [Crandall et al. ECCV 2006] [Fergus et al. CVPR 2003][Leibe et al. ECCV 2004] [Kumar et al. CVPR 2005] … X X X X

6 Related work: windowed detectors [Viola and Jones CVPR 2001] [Shotton et al. ICCV 2005]… Localised features Classifier Car Sliding window

7 Related work: windowed detectors Localised features Classifier Car? Wall? [Viola and Jones CVPR 2001] [Shotton et al. ICCV 2005]… Sliding window

8 Related work: multiclass segmentation [Tu et al. CVPR 2003] [He et al. CVPR 2004] tree car road building Doesnt exploit layout of parts – cant identify object instances TextonBoost [Shotton et al. ECCV 2006]

9 Related work Layout consistency Layout Consistent Random Field Results Roadmap

10 Dense part labelling Automatic per-pixel labelling based on a grid of parts Part labels (color-coded)

11 Dense part labelling Background label Part labels (color-coded)

12 Patch-based part detector [Lepetit et al. CVPR 2005] Decision forest classifier Features are differences of pixel intensities Classifier

13 Decision trees Extremely efficient at both training and test time. e.g. takes 2ms to apply to 160x120 image using difference of pixel intensities. Improved performance with multiple decision trees (random forest). Performs as well as boosting with shared features, but can process much more data in the same time.

14 Patch-based part detector Colors show posterior over part labels – part detectors are noisy! Part color key

15 Layout consistency

16

17 (8,3)(9,3)(7,3) (8,2)(9,2)(7,2) (8,4)(9,4)(7,4) Neighboring pixels (p,q) ?

18 Layout consistency (8,3)(9,3)(7,3) (8,2)(9,2)(7,2) (8,4)(9,4)(7,4) Neighboring pixels (p,q) (p+1,q) (p,q) (p+1,q+1) (p+1,q-1) Allows for deformation /rotation Layout consistent

19 Layout consistency (8,3)(9,3)(7,3) (8,2)(9,2)(7,2) (8,4)(9,4)(7,4) Neighboring pixels (p,q) ? (p,q+1) (p,q) (p+1,q+1) (p-1,q+1) Layout consistent

20 Occlusions One object instance occludes another Background occludes object Object occludes background (object edge) Not layout consistent = occlusion (or invalid)

21 Effect of layout consistency Input image With layout consistency Part detector output Layout consistent regions

22 Related work Layout consistency Layout Consistent Random Field Results Roadmap

23 Layout Consistent Random Field Part detector Part labels h Image I

24 Layout Consistent Random Field Part labels h Layout consistency Image I Part detector

25 Layout Consistent Random Field Parameters θ = { β bg, β oe, β co, β iif, e 0, γ } (set by hand) Layout consistency Part detector Edge weight

26 Proposed labelling Inference of MAP labelling Graph cuts with customised alpha-expansion move [Boykov and Jolly, ICCV 2001] Part labels h

27 Inference of MAP labelling [Boykov and Jolly, ICCV 2001] Graph cuts with customised alpha-expansion move Proposed labelling Part labels h

28 Inference of MAP labelling [Boykov and Jolly, ICCV 2001] Expansion move not accepted Graph cuts with customised alpha-expansion move Proposed labelling Part labels h

29 Inference of MAP labelling [Boykov and Jolly, ICCV 2001] Graph cuts with customised alpha-expansion move Proposed labelling Part labels h

30 Example inference

31 Decision tree re-learning Part-labels are inferred (constrained by known mask) and decision forest re-trained

32 Limitation of layout consistency Allows arbitrary stretching/scaling

33 Part labels h Global layout Instance T 1 Instance T 2 Global layout constraint is (weak) star-shaped constellation model Constrains part locations relative to centroid Allows competition between different object instances Image I

34 Example with global consistency Input image Layout consistent regionsInstance labelling T1T1 T2T2 T3T3 T1T1 T2T2

35 Related work Layout consistency Layout Consistent Random Field Results Roadmap

36 UIUC car database Segmentation accuracy: 96.5% pixels correct (assessed on 20 randomly selected, hand-labelled images)

37 UIUC car database Segmentation accuracy: 96.5% pixels correct (assessed on 20 randomly selected, hand-labelled images)

38 UIUC car database: detection Results refer to detection of unoccluded cars only.

39 Detecting heavily occluded faces Caltech face database with artificial occlusions AR face database with real occlusions

40 Stability of part labelling Part color key

41 Multi-class detection Can extend to multiple classes with different numbers of part labels for each class Example: building has multiple parts, other classes have one

42 Summary + future directions Summary: LayoutCRF achieves multi-class detection and segmentation of occluded, deformable objects Future directions: Extend to multiple viewpoints and multiple scales Share parts between classes Incorporate object context (car above road) Incorporate geometric cues

43 Thank you

44


Download ppt "The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton."

Similar presentations


Ads by Google