1. Joint Optimisation for Object Class Segmentation and Dense Stereo Reconstruction Ľubor Ladický, Paul Sturgess, Christopher Russell, Sunando Sengupta, Yalin Bastanlar, William Clocksin, Philip H.S. Torr Oxford Brookes University http://cms.brookes.ac.uk/research/visiongroup/

2. Black Box Solver Left Camera Image Right Camera Image Object Class Segmentation Dense Stereo Reconstruction Joint Object Class Segmentation and Dense Stereo Reconstruction

3. Black Box Solver Left Camera Image Right Camera Image Object Class Segmentation Dense Stereo Reconstruction Joint Object Class Segmentation and Dense Stereo Reconstruction Objective : Joint Estimation

4. Dense Stereo Reconstruction Left Camera ImageRight Camera ImageDense Stereo Result For each pixel assigns a disparity label : y Disparities from the discrete set {0, 1,.. D}

5. Dense Stereo Reconstruction Unary Potential Unary Cost dependent on the similarity of patches, e.g.cross correlation Disparity = 0

6. Dense Stereo Reconstruction Unary Potential Unary Cost dependent on the similarity of patches, e.g.cross correlation Disparity = 5

7. Dense Stereo Reconstruction Unary Cost dependent on the similarity of patches, e.g.cross correlation Unary Potential Disparity = 10

8. Dense Stereo Reconstruction Unary Cost dependent on the similarity of patches, e.g.cross correlation Unary Potential Disparity = 15

9. Dense Stereo Reconstruction Pairwise Potential Encourages label consistency in adjacent pixels Cost based on the distance of labels Linear TruncatedQuadratic Truncated

10. Dense Stereo Reconstruction Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original ImageInitial Solution

11. Dense Stereo Reconstruction Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original ImageInitial SolutionAfter 1 st expansion Final solution

12. Dense Stereo Reconstruction Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original ImageInitial Solution After 2 nd expansion After 1 st expansion

13. Dense Stereo Reconstruction Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original ImageInitial Solution After 2 nd expansion After 1 st expansion After 3 rd expansion

14. Dense Stereo Reconstruction Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original ImageInitial Solution After 2 nd expansion After 1 st expansion After 3 rd expansionFinal solution

15. Dense Stereo Reconstruction Does not work for Road Scenes ! Original Image Dense Stereo Reconstruction

16. Does not work for Road Scenes ! Patches can be matched to any other patch for flat surfices Different brightness in cameras

17. Dense Stereo Reconstruction Does not work for Road Scenes ! Patches can be matched to any other patch for flat surfices Different brightness in cameras Could object recognition for road scenes help? Recognition of road scenes is relatively easy (Sturgess et al., BMVC09)

18. Object Class Segmentation Aims to assign a class label for each pixel of an image Classifier trained on the training set Evaluated on never seen test images

19. Object Class Segmentation Unary Potential Likelihood of a pixel taking a label (Shotton et al. ECCV06, He et al, CVPR04, Ladický et al. ICCV 09)

20. Object Class Segmentation Pairwise Potential Contrast sensitive Potts model Encourages label consistency in adjacent pixels

21. Object Class Segmentation Higher Order Potential Encouraging consistency in superpixels (Kohli et al. CVPR08) Merging information at different scales (Ladický et al. ICCV09)

22. Object Class Segmentation Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) Original ImageInitial solution grass

23. Object Class Segmentation Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) Original ImageInitial solutionBuilding expansion grass building

24. Object Class Segmentation Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) Original ImageInitial solutionBuilding expansion Sky expansion grass building sky

25. Object Class Segmentation Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) Original ImageInitial solutionBuilding expansion Sky expansion Tree expansion grass building sky tree

26. Object Class Segmentation Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) Original ImageInitial solutionBuilding expansion Sky expansion Tree expansionFinal Solution grass building sky tree aeroplane

27. Object Class Segmentation vs. Dense Stereo Reconstruction sky Object class and 3D location are mutually informative Sky always in infinity (disparity = 0) 

28. Object Class Segmentation vs. Dense Stereo Reconstruction buildingcarsky Object class and 3D location are mutually informative Sky always in infinity (disparity = 0) Cars, buildings & pedestrians have their typical height

29. Object Class Segmentation vs. Dense Stereo Reconstruction roadbuildingcarsky Object class and 3D location are mutually informative Sky always in infinity (disparity = 0) Cars, buses & pedestrians have their typical height Road and pavement on the ground plane

30. Object Class Segmentation vs. Dense Stereo Reconstruction roadbuildingcarsky Object class and 3D location are mutually informative Sky always in infinity (disparity = 0) Cars, buses & pedestrians have their typical height Road and pavement on the ground plane Buildings and pavement on the sides

31. Object Class Segmentation vs. Dense Stereo Reconstruction roadbuildingcarsky Object class and 3D location are mutually informative Sky always in infinity (disparity = 0) Cars, buses & pedestrians have their typical height Road and pavement on the ground plane Buildings and pavement on the sides Both problems formulated as CRF Joint approach possible?

32. Joint Formulation Each pixels takes label z i = [ x i y i ]  L 1  L 2 Dependency of x i and y i encoded as a unary and pairwise potential, e.g. strong correlation between x = road, y = near ground plane strong correlation between x = sky, y = 0 Correlation of edge in object class and disparity domain

33. Joint formulation Weighted sum of object class, depth and joint potential Joint unary potential based on histograms of height Unary Potential Object layer Disparity layer Joint unary links

34. Joint Formulation Object class and depth edges correlated Transitions in depth occur often at the object boundaries Pairwise Potential Object layer Disparity layer Joint pairwise links

35. Joint Formulation

36.  Standard α-expansion Each node in each expansion move keeps its old label or takes a new label [x L1, y L2 ], Possible in case of metric pairwise potentials Inference

37. Standard α-expansion Each node in each expansion move keeps its old label or takes a new label [x L1, y L2 ], Possible in case of metric pairwise potentials Inference Too many moves! ( |L1| |L2| ) Impractical !

38. Projected move for product label space One / Some of the label components remain(s) constant after the move Set of projected moves α-expansion in the object class projection Range-expansion in the depth projection Inference

39. Leuven Road Scene dataset Contained 3 sequences 643 pairs of images We labelled 50 training + 20 test images Object class (7 labels) Disparity (100 labels) Available on our website http://cms.brookes.ac.uk/research/visiongroup/files/Leuven.zip Dataset Left cameraRight cameraObject GTDisparity GT

40. Qualitative results Large improvement for dense stereo estimation Minor improvement in object class segmentation Original ImageObject GTObject ResultDisparity GTDisparity AloneDisparity Jointly

41. Quantitative disparity results Dependency of the ratio of correctly labelled pixels within the maximum allowed error delta

42. Application to monocular sequences Making method (close to) real time Application to multi-view problems Optical flow / motion estimation On-going and Future Work

43. First dataset with both object class and disparity labels Joint estimation improves significantly disparity results Projected moves make inference much faster Questions ? Summary