2. Lecture 12 Stereo Reconstruction II Lecture 12 Stereo Reconstruction II Mata kuliah: T0283 - Computer Vision Tahun: 2010

3. January 20, 2010T0283 - Computer Vision3 Learning Objectives After carefullylistening this lecture, students will be able to do the following : After carefully listening this lecture, students will be able to do the following : demonstrate 3D stereo computation by solving point- correspondence problems and fundamentalmatrix. demonstrate 3D stereo computation by solving point- correspondence problems and fundamental matrix. Calculate object-depth information using disparity and triangulation techniques Calculate object-depth information using disparity and triangulation techniques

4. January 20, 2010T0283 - Computer Vision4 An algorithm for stereo reconstruction 1.For each point in the first image determine the corresponding point in the second image (this is a search problem) 2.For each pair of matched points determine the 3D point by triangulation (this is an estimation problem)

5. January 20, 2010T0283 - Computer Vision5 Epipolar line Epipolar constraint Reduces correspondence problem to 1D search along an epipolar line

6. January 20, 2010T0283 - Computer Vision6 Algebraic representation of epipolar geometry We know that the epipolar geometry defines a mapping x l / point in first image epipolar line in second image the map only depends on the cameras P, P / (not on structure) it will be shown that the map is linear and can be written as

7. January 20, 2010T0283 - Computer Vision7 Stereo correspondence algorithms

8. January 20, 2010T0283 - Computer Vision8 Problem statement Given: two images and their associated cameras compute corresponding image points. Algorithms may be classified into two types: 1.Dense: compute a correspondence at every pixel 2.Sparse: compute correspondences only for features The methods may be top down or bottom up

9. January 20, 2010T0283 - Computer Vision9 Top down matching 1.Group model (house, windows, etc) independently in each image 2.Match points (vertices) between images

10. January 20, 2010T0283 - Computer Vision10 Bottom up matching epipolar geometry reduces the correspondence search from 2D to a 1D search on corresponding epipolar lines 1D correspondence problem b/b/ a/a/ b c a C B A c/c/

11. January 20, 2010T0283 - Computer Vision11 Example image pair – parallel cameras Example image pair – parallel cameras

12. January 20, 2010T0283 - Computer Vision12 First image

13. January 20, 2010T0283 - Computer Vision13 Second image

14. January 20, 2010T0283 - Computer Vision14 Dense correspondence algorithm Search problem (geometric constraint): for each point in the left image, the corresponding point in the right image lies on the epipolar line (1D ambiguity) Disambiguating assumption (photometric constraint): the intensity neighbourhood of corresponding points are similar across images Measure similarity of neighbourhood intensity by cross-correlation Parallel camera example – epipolar lines are corresponding rasters epipolar line

15. January 20, 2010T0283 - Computer Vision15 Intensity profiles Clear correspondence between intensities, but also noise and ambiguity

16. January 20, 2010T0283 - Computer Vision16 Normalized Cross Correlation region Aregion B vector a vector b write regions as vectors a b

17. January 20, 2010T0283 - Computer Vision17 Cross-correlation of neighbourhood regions epipolar line

18. January 20, 2010T0283 - Computer Vision18 left image band right image band cross correlation 1 0 0.5 x

19. January 20, 2010T0283 - Computer Vision19 left image band right image band cross correlation 1 0 x 0.5 target region

20. January 20, 2010T0283 - Computer Vision20 Why is cross-correlation such a poor measure in the second case? 1.The neighborhood region does not have a “distinctive” spatial intensity distribution 2.Foreshortening effects front-parallel surface imaged length the same slanting surface imaged lengths differ

21. January 20, 2010T0283 - Computer Vision21 Limitations of similarity constraint Textureless surfaces Occlusions, repetition Non-Lambertian surfaces, specularities

22. January 20, 2010T0283 - Computer Vision22 Results with window search Window-based matchingGround truth Data

23. January 20, 2010T0283 - Computer Vision23 Sketch of a dense correspondence algorithm For each pixel in the left image compute the neighbourhood cross correlation along the corresponding epipolar line in the right image the corresponding pixel is the one with the highest cross correlation Parameters size (scale) of neighbourhood search disparity Other constraints uniquenessordering smoothness of disparity field Applicability textured scene, largely fronto-parallel

24. January 20, 2010T0283 - Computer Vision24 Example dense correspondence algorithm left imageright image

25. January 20, 2010T0283 - Computer Vision25 3D Reconstruction intensity = depth right image depth map

26. January 20, 2010T0283 - Computer Vision26 range map Pentagon example left imageright image

27. January 20, 2010T0283 - Computer Vision27 Rectification e e / For converging cameras epipolar lines are not paralle

28. January 20, 2010T0283 - Computer Vision28 Project images onto plane parallel to baseline epipolar plane

29. January 20, 2010T0283 - Computer Vision29 Rectification continued Convert converging cameras to parallel camera geometry by an image mapping Image mapping is a 2D homography (projective transformation)

30. January 20, 2010T0283 - Computer Vision30 Example original stereo pair rectified stereo pair

31. January 20, 2010T0283 - Computer Vision31 Note image movement (disparity) is inversely proportional to depth Z depth is inversely proportional to disparity Example : depth and disparity for a parallel camera stereo rig Then, y / = y, and the disparity Derivation x x / d

32. January 20, 2010T0283 - Computer Vision32 Triangulation

33. January 20, 2010T0283 - Computer Vision33 Problem statement Given: corresponding measured (i.e. noisy) points x and x /, and cameras (exact) P and P /, compute the 3D point X Problem: in the presence of noise, back projected rays do not intersect rays are skew in space Measured points do not lie on corresponding epipolar lines C C / x x /

34. January 20, 2010T0283 - Computer Vision34 1. Vector solution C C / Compute the mid-point of the shortest line between the two rays

35. January 20, 2010T0283 - Computer Vision35 2. Linear triangulation (algebraic solution)

36. January 20, 2010T0283 - Computer Vision36

37. January 20, 2010T0283 - Computer Vision37 3. Minimizing a geometric/statistical error