1. A Local Adaptive Approach for Dense Stereo Matching in Architectural Scene Reconstruction C. Stentoumis 1, L. Grammatikopoulos 2, I. Kalisperakis 2, E. Petsa 2, G. Karras 1 1. Laboratory of Photogrammetry, Department of Surveying, National Technical University of Athens, GR-15780 Athens, Greece 2. Laboratory of Photogrammetry, Department of Surveying, Technological Educational Institute of Athens, GR-12210 Athens, Greece 5th International Workshop on 3D Virtual Reconstruction and Visualization of Complex Architectures (3D-ARCH '2013), 25-26 February 2013 1

2. Outline Introduction Related Work Proposed Algorithm Experimental Results Conclusion 2

3. Introduction 3

4. Objective Present a method Combines pre-existing algorithms and novel considerations With good sub-pixel accuracy 4

5. Related work 5

6. Related Work 6 Stereo Matching : 14 5 Range: 0 - 16 (X,Y)(X-d,Y)

7. Related Work 7 Matching cost computation For every individual pixel Cost is assigned to all possible disparities Cost aggragation Neighboring share the same disparity A summation of pixel-wise costs (over a support region) Disparity optimization The disparity with the lowest aggregated cost is chosen. Refinement Correcting inaccurate disparity values

8. Proposed Algorithm 8

9. Census on intensity principal derivatives Census transformation based on gradients: Less sensitive to radiometric differences and repetitive patterns 9 Intensity Gradient

10. Census on Gradients 10 11000 11000 11X00 00011 11111 0.70.80.30.20.3 0.6 0.40.30.2 0.60.70.50.40.1 0.4 0.30.6 0.70.60.7 110001100011000001111111 Census transform window :

11. Hamming Distance 11 Left image Right image Hamming Distance = 3 110001100011000001111111 111001100101000001111111 XOR 001000000110000000000000

12. Comparison After aggregation step: Default census Census on gradients 2.5% less erroneous pixels 12

13. Comparison After aggregation step [13] : 13 [13]Mei X., Sun X., Zhou M., Jiao S., Wang H., Zhang X., 2011. On building an accurate stereo matching system on graphics hardware. ICCV Workshop on GPU in Computer Vision Applications.

14. Absolute Difference on Image Color and Gradients 14 AD ( color ) : AD ( Gradient ) :

15. Total Matching Cost normalized by λ 15 AD (color)Census (gradient) AD (gradient)

16. 16 Census (gradient) AD (color) AD (gradient) Combined

17. Support Region 17 Cross-based support region [25] : Threshold of cross-skeleton expansion: [25]Zhang K., Lu J., Lafruit G., 2009. Cross-based local stereo matching using orthogonal integral images. IEEE Transactions on Circuits & Systems for Video Technology.

18. Support Region 18 Threshold of cross-skeleton expansion [16] : L max : largest semi-dimension of the window size τ max : largest color dissimilarity between p and q 3X3 median filter [16]Stentoumis C., Grammatikopoulos L., Kalisperakis I., Karras G., 2012. Implementing an adaptive approach for dense stereo matching. International Archives of Photogrammetry, Remote Sensing & Spatial Information Sciences, Pq lqlq

19. Support Region 19

20. Aggregation Step 20 A. Normalized by the number of pixels in the support region B. 3D Gaussian function is applied for smoothing the aggregated costs. C. Winner-take-all

21. Comparison After aggregation step [13] : 21 Run this step for 4 iterations to get stable cost values. For iteration 1 and 3, aggregated horizontally and then vertically. For iteration 2 and 4, aggregated vertically and then horizontally.

22. 22 [13] method Proposed Method

23. Refinement 23 Left-right consistency check Pixel p is characterized as valid (inlier) if the following constraint holds:

24. Refinement 24 Outlier cross-based filtering The cross-based support regions provide a robust description of pixel neighborhoods The median value of inliers in the support region is selected and attributed to the mismatched pixel.

25. Refinement 25 Occlusion / mismatch labeling Remaining outliers are re-estimated Mismatches: The epipolar line of the mismatch pixel intersects with disparity function Use median interpolation in a small patch around them Occlusions Use the second lowest disparity value in the neighborhood

26. Refinement 26 Epipolar Line BeforeAfter

27. Refinement 27 Sub-pixel estimation Estimation at the sub-pixel level is made by interpolating a 2 nd order curve to the cost volume C(d). This curve is defined by the disparities of the preceding and following pixels and their corresponding cost values Choose minimum cost position through a closed form solution for the 3 curve points.

28. Refinement 28 Disparity map smoothing Median filter is applied. The effect of overall post-processing refinement

29. Experimental Results 29

30. Experimental Results Evaluated on the Middlebury and EPFL multi-view datasets Parameter values were kept constant for all tests. 30

31. 31

32. Experimental Results Middlebury evaluation 32 Error Threshold = 1 Error Threshold = 0.75

33. 33 [13] method Proposed Method

34. Experimental Results 34 AInitial disparity mapD Remaining occlusion/ mismatch handling BCost smoothingESub-pixel estimation COutlier filteringFMedian smoothing Threshold = 0.75, % of wrong pixels

35. Experimental Results 35 Herz-Jesu-K7 stereo pair

36. Experimental Results 36 Herz-Jesu-K7 stereo pair

37. Conclusion 37

38. Conclusion 38 Initial Cost AD (color / gradient) Census (color / gradient) Aggregation Aggregation more times Normalize → Smoothing Refinement Mismatch : Epipolar Line / Interpolation Sub-pixel Estimation Occlusion / Parameter adjustment