1. Raquel A. Romano 1 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Projective Geometry for Computer Vision Raquel A. Romano MIT Artificial Intelligence Laboratory romano@ai.mit.edu

2. Raquel A. Romano 2 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision 3D Computer Vision Classical Problem: Given a collection of 2D images, build a model of the 3D world. Example Applications: virtual/immersive environments robotics & autonomous vehicles minimally invasive surgery

3. Raquel A. Romano 3 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Outline 1.Projective Geometry Overview 2.Minimal Projective Parameters 3.Projective Parameter Estimation 4.Motion Boundary Detection 5.Conclusion

4. Raquel A. Romano 4 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision imaging Image Formation 3D scene 2D images

5. Raquel A. Romano 5 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision measurement Computer Vision 3D scene model 2D images analysis data

6. Raquel A. Romano 6 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision scene point optical center image point optical ray optical axis Camera Geometry: Single View pinhole model of perspective projection unknown internal camera parameters unknown depth at each point

7. Raquel A. Romano 7 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Camera Geometry: Multiple Views unknown rotations and translations

8. Raquel A. Romano 8 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Measured Data: Image Points and Lines geometric constraint: optical rays intersect in 3D projective geometry: express constraint in terms of measured 2D image features

9. Raquel A. Romano 9 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Projective Camera Model linear model of image formation depth-independent expression for optical ray intersections multilinear relations among point and line matches

10. Raquel A. Romano 10 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Bilinear Constraints fundamental matrix (Longuet-Higgins,1981, Faugeras, 1992; Hartley, 1992)

11. Raquel A. Romano 11 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Fundamental Matrix Maps a point in one image to a line in the other image that contains its match Given matching points in two views, predict the matching point in a third image.

12. Raquel A. Romano 12 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Projective Models in Practice View synthesis and interpolation: point transfer function for dense point correspondences Self-calibration: automatic recovery of internal camera parameters from fundamental matrices Bundle adjustment initialization: initial rotation and translation for nonlinear Euclidean optimization

13. Raquel A. Romano 13 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Outline 1.Projective Geometry Overview 2.Minimal Projective Parameters 3.Projective Parameter Estimation 4.Motion Boundary Detection 5.Conclusion

14. Raquel A. Romano 14 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Practical Problem Few point matches between some views. Unstable for estimating geometric relationships.

15. Raquel A. Romano 15 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Geometric Consistency Pairwise geometric relations may be inconsistent. ?

16. Raquel A. Romano 16 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Goals Impose algebraic geometric constraints on stationary points seen in arbitrarily many views. Avoid estimating too many parameters: depths, rotations, translations

17. Raquel A. Romano 17 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Geometric Dependencies Pairwise projective geometric relations are interdependent. Approach: define projective dependencies and restrict solutions to be globally consistent

18. Raquel A. Romano 18 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Projective Bilinear Parameters

19. Raquel A. Romano 19 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision imaged 3D translation & rotation Projective Bilinear Parameters epipoles epipolar collineation (Csurka, et.al., 1997)

20. Raquel A. Romano 20 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Projective Parameters provide a complete projective model of camera configuration But... set of all pairwise parameters are still redundant not all images have sufficient overlap

21. Raquel A. Romano 21 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Trifocal Dependencies derive dependencies among three fundamental matrices correctly models degrees of freedom in camera configuration geometrically consistent parameterized model of view triplets

22. Raquel A. Romano 22 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Trifocal Dependencies trifocal lines available from two fundamental matrices derive dependencies among three fundamental matrices correctly models degrees of freedom in camera configuration geometrically consistent parameterized model of view triplets

23. Raquel A. Romano 23 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Outline 1.Projective Geometry Overview 2.Minimal Projective Parameters 3.Projective Parameter Estimation 4.Motion Boundary Detection 5.Conclusion

24. Raquel A. Romano 24 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Recovering Camera Geometry view i view j view k few correspondences

25. Raquel A. Romano 25 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Linear Initialization 8-point Algorithm (Hartley, 1995) Rewrite bilinear constraints as where and solve linear system Minimize over all matching point pairs.

26. Raquel A. Romano 26 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Projection to Parameter Space Map linear estimate of fundamental matrix to projective parameter space: parameterization requires choice of projective basis basis affects shape of error surface for nonlinear optimization

27. Raquel A. Romano 27 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Geometric Objective Function point-to-epipolar-line distance ~ image reprojection error weighted residual of bilinear constraint

28. Raquel A. Romano 28 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision gamma(i,j) (h1,h2) Error Surface Depends on Basis (h1,h2) (h2,h3) (h1,h3) (e1,e2) (e3,e4) (h1,h2) (h2,h3) (h1,h3) gamma(i,j) (h1,h2) (e1,e2) (e3,e4) canonical basisgeometrically defined basis

29. Raquel A. Romano 29 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Nonlinear Trifocal Estimation 1. Initialize epipolar geometry 8-point algorithm: linear solution to fundamental matrix for all view pairs extract epipoles and epipolar collineations 2.7D nonlinear minimization: bifocal parameters for view pairs (i,k) (j,k) 3.Trifocally constrained estimation for view pair (i,j) compute trifocal lines project parameters to trifocally constrained space 4D nonlinear minimization for bifocal parameters i j k

30. Raquel A. Romano 30 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Ground Truth 8-point Algorithm 7-Parameter Search Trifocal Projection 4-Parameter Search Convergence eijerroreji -4000 -2000 0

31. Raquel A. Romano 31 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Ground Truth 8-Point Algorithm 7-Parameter Algorithm 4-Parameter Algorithm Results

32. Raquel A. Romano 32 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision knossos sequence view i view k view j few correspondences

33. Raquel A. Romano 33 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision 4-Parameter Algorithm Ground Truth 7-Parameter Algorithm 8-Point Algorithm Results

34. Raquel A. Romano 34 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Summary Imposing projective constraints on camera geometry corrects the estimation of epipolar geometry Resulting camera configuration for multiple cameras is globally consistent

35. Raquel A. Romano 35 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Outline 1.Projective Geometry Overview 2.Minimal Projective Parameters 3.Projective Parameter Estimation 4.Motion Boundary Detection 5.Conclusion

36. Raquel A. Romano 36 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Camera and Scene Motion

37. Raquel A. Romano 37 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Combining Intensity and Geometry trifocal tensor projective linear form relating a point-line-line (Spetsakis & Aloimonos, 1990; Shashua, 1994)

38. Raquel A. Romano 38 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Tensor Brightness Constraint (Shashua & Hannah, 1995; Shashua & Stein, 1997) Horn-Schunk brightness constraint is linear in point coordinates Defines line in each image containing matching point Spatiotemporal gradient at every pixel provides test of rigid motion u I x + v I y + I t = 0 ax + by + c = 0 (a,b,c) T  IxIx       IyIy I t - x 0 I x – y 0 I y u = x - x 0 v = y - y 0

39. Raquel A. Romano 39 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Motion Boundary Detection Partition image into windows and solve for trifocal tensor coefficients. Sum residual error of tensor solution. Only regions with rigid 3D motion have a good fit. High residuals indicate regions that cross a motion boundary.

40. Raquel A. Romano 40 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Multiple Frame Flow Multi-frame tracks fall into separable classes Track points over many frames Robustly fit tracks to linear approximation of instantaneous planar motion x(t) = x 0 + t [Ax 0 + b]

41. Raquel A. Romano 41 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Detecting Independent Motions Residual error of estimated motion model on all point tracks

42. Raquel A. Romano 42 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Complexity of Motion Model

43. Raquel A. Romano 43 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Conclusions When possible, use domain and task knowledge to choose model: What type of information is needed What aspects of the imaging conditions are known or controlled What types of uncertainty can be modeled and compensated for

44. Raquel A. Romano 44 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Future Needs Role of learning in motion analysis: Supervised learning of geometric motion classes Data-driven model selection by flow classification Robust estimation of appropriate motion model Adaptive, time-varying estimation

45. Raquel A. Romano 45 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision END