Download presentation

Presentation is loading. Please wait.

Published byRory Burruss Modified about 1 year ago

1
Joint Optimisation for Object Class Segmentation and Dense Stereo Reconstruction Ľubor Ladický, Paul Sturgess, Christopher Russell, Sunando Sengupta, Yalin Bastanlar, William Clocksin, Philip H.S. Torr Oxford Brookes University

2
Black Box Solver Left Camera Image Right Camera Image Object Class Segmentation Dense Stereo Reconstruction Joint Object Class Segmentation and Dense Stereo Reconstruction

3
Black Box Solver Left Camera Image Right Camera Image Object Class Segmentation Dense Stereo Reconstruction Joint Object Class Segmentation and Dense Stereo Reconstruction Objective : Joint Estimation

4
Dense Stereo Reconstruction Left Camera ImageRight Camera ImageDense Stereo Result For each pixel assigns a disparity label : y Disparities from the discrete set {0, 1,.. D}

5
Dense Stereo Reconstruction Unary Potential Unary Cost dependent on the similarity of patches, e.g.cross correlation Disparity = 0

6
Dense Stereo Reconstruction Unary Potential Unary Cost dependent on the similarity of patches, e.g.cross correlation Disparity = 5

7
Dense Stereo Reconstruction Unary Cost dependent on the similarity of patches, e.g.cross correlation Unary Potential Disparity = 10

8
Dense Stereo Reconstruction Unary Cost dependent on the similarity of patches, e.g.cross correlation Unary Potential Disparity = 15

9
Dense Stereo Reconstruction Pairwise Potential Encourages label consistency in adjacent pixels Cost based on the distance of labels Linear TruncatedQuadratic Truncated

10
Dense Stereo Reconstruction Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original ImageInitial Solution

11
Dense Stereo Reconstruction Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original ImageInitial SolutionAfter 1 st expansion Final solution

12
Dense Stereo Reconstruction Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original ImageInitial Solution After 2 nd expansion After 1 st expansion

13
Dense Stereo Reconstruction Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original ImageInitial Solution After 2 nd expansion After 1 st expansion After 3 rd expansion

14
Dense Stereo Reconstruction Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original ImageInitial Solution After 2 nd expansion After 1 st expansion After 3 rd expansionFinal solution

15
Dense Stereo Reconstruction Does not work for Road Scenes ! Original Image Dense Stereo Reconstruction

16
Does not work for Road Scenes ! Patches can be matched to any other patch for flat surfices Different brightness in cameras

17
Dense Stereo Reconstruction Does not work for Road Scenes ! Patches can be matched to any other patch for flat surfices Different brightness in cameras Could object recognition for road scenes help? Recognition of road scenes is relatively easy (Sturgess et al., BMVC09)

18
Object Class Segmentation Aims to assign a class label for each pixel of an image Classifier trained on the training set Evaluated on never seen test images

19
Object Class Segmentation Unary Potential Likelihood of a pixel taking a label (Shotton et al. ECCV06, He et al, CVPR04, Ladický et al. ICCV 09)

20
Object Class Segmentation Pairwise Potential Contrast sensitive Potts model Encourages label consistency in adjacent pixels

21
Object Class Segmentation Higher Order Potential Encouraging consistency in superpixels (Kohli et al. CVPR08) Merging information at different scales (Ladický et al. ICCV09)

22
Object Class Segmentation Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) Original ImageInitial solution grass

23
Object Class Segmentation Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) Original ImageInitial solutionBuilding expansion grass building

24
Object Class Segmentation Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) Original ImageInitial solutionBuilding expansion Sky expansion grass building sky

25
Object Class Segmentation Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) Original ImageInitial solutionBuilding expansion Sky expansion Tree expansion grass building sky tree

26
Object Class Segmentation Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) Original ImageInitial solutionBuilding expansion Sky expansion Tree expansionFinal Solution grass building sky tree aeroplane

27
Object Class Segmentation vs. Dense Stereo Reconstruction sky Object class and 3D location are mutually informative Sky always in infinity (disparity = 0)

28
Object Class Segmentation vs. Dense Stereo Reconstruction buildingcarsky Object class and 3D location are mutually informative Sky always in infinity (disparity = 0) Cars, buildings & pedestrians have their typical height

29
Object Class Segmentation vs. Dense Stereo Reconstruction roadbuildingcarsky Object class and 3D location are mutually informative Sky always in infinity (disparity = 0) Cars, buses & pedestrians have their typical height Road and pavement on the ground plane

30
Object Class Segmentation vs. Dense Stereo Reconstruction roadbuildingcarsky Object class and 3D location are mutually informative Sky always in infinity (disparity = 0) Cars, buses & pedestrians have their typical height Road and pavement on the ground plane Buildings and pavement on the sides

31
Object Class Segmentation vs. Dense Stereo Reconstruction roadbuildingcarsky Object class and 3D location are mutually informative Sky always in infinity (disparity = 0) Cars, buses & pedestrians have their typical height Road and pavement on the ground plane Buildings and pavement on the sides Both problems formulated as CRF Joint approach possible?

32
Joint Formulation Each pixels takes label z i = [ x i y i ] L 1 L 2 Dependency of x i and y i encoded as a unary and pairwise potential, e.g. strong correlation between x = road, y = near ground plane strong correlation between x = sky, y = 0 Correlation of edge in object class and disparity domain

33
Joint formulation Weighted sum of object class, depth and joint potential Joint unary potential based on histograms of height Unary Potential Object layer Disparity layer Joint unary links

34
Joint Formulation Object class and depth edges correlated Transitions in depth occur often at the object boundaries Pairwise Potential Object layer Disparity layer Joint pairwise links

35
Joint Formulation

36
Standard α-expansion Each node in each expansion move keeps its old label or takes a new label [x L1, y L2 ], Possible in case of metric pairwise potentials Inference

37
Standard α-expansion Each node in each expansion move keeps its old label or takes a new label [x L1, y L2 ], Possible in case of metric pairwise potentials Inference Too many moves! ( |L1| |L2| ) Impractical !

38
Projected move for product label space One / Some of the label components remain(s) constant after the move Set of projected moves α-expansion in the object class projection Range-expansion in the depth projection Inference

39
Leuven Road Scene dataset Contained 3 sequences 643 pairs of images We labelled 50 training + 20 test images Object class (7 labels) Disparity (100 labels) Available on our website Dataset Left cameraRight cameraObject GTDisparity GT

40
Qualitative results Large improvement for dense stereo estimation Minor improvement in object class segmentation Original ImageObject GTObject ResultDisparity GTDisparity AloneDisparity Jointly

41
Quantitative disparity results Dependency of the ratio of correctly labelled pixels within the maximum allowed error delta

42
Application to monocular sequences Making method (close to) real time Application to multi-view problems Optical flow / motion estimation On-going and Future Work

43
First dataset with both object class and disparity labels Joint estimation improves significantly disparity results Projected moves make inference much faster Questions ? Summary

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google