1. Computer Vision Projective structure from motion Marc Pollefeys COMP 256 Some slides and illustrations from J. Ponce, …

2. Computer Vision Last class: Affine camera The affine projection equations are how to find the origin? or for that matter a 3D reference point? affine projection preserves center of gravity

3. Computer Vision Orthographic factorization The orthographic projection equations are where All equations can be collected for all i and j where Note that P and X are resp. 2 m x3 and 3x n matrices and therefore the rank of x is at most 3

4. Computer Vision Orthographic factorization Factorize m through singular value decomposition An affine reconstruction is obtained as follows Closest rank-3 approximation yields MLE!

5. Computer Vision Aug 26/28-Introduction Sep 2/4CamerasRadiometry Sep 9/11Sources & ShadowsColor Sep 16/18Linear filters & edges(hurricane Isabel) Sep 23/25Pyramids & TextureMulti-View Geometry Sep30/Oct2StereoProject proposals Oct 7/9Tracking (Welch)Optical flow Oct 14/16-- Oct 21/23Silhouettes/carving(Fall break) Oct 28/30-Structure from motion Nov 4/6Project updateProj. SfM Nov 11/13Camera calibrationSegmentation Nov 18/20FittingProb. segm.&fit. Nov 25/27Matching templates(Thanksgiving) Dec 2/4Matching relationsRange data Dec ?Final project Tentative class schedule

6. Computer Vision Further Factorization work Factorization with uncertainty Factorization for indep. moving objects Factorization for dynamic objects Perspective factorization Factorization with outliers and missing pts. (Irani & Anandan, IJCV’02) (Costeira and Kanade ‘94) (Bregler et al. 2000, Brand 2001) (Jacobs 1997 (affine), Martinek and Pajdla 2001, Aanaes 2002 (perspective)) (Sturm & Triggs 1996, …)

7. Computer Vision Structure from motion of multiple moving objects one object: multiple objects:

8. Computer Vision SfM of multiple moving objects

9. Computer Vision Structure from motion of deforming objects Extend factorization approaches to deal with dynamic shapes (Bregler et al ’00; Brand ‘01)

10. Computer Vision Representing dynamic shapes represent dynamic shape as varying linear combination of basis shapes (fig. M.Brand)

11. Computer Vision Projecting dynamic shapes (figs. M.Brand) Rewrite:

12. Computer Vision Dynamic image sequences One image: Multiple images (figs. M.Brand)

13. Computer Vision Dynamic SfM factorization? Problem: find J so that M has proper structure

14. Computer Vision Dynamic SfM factorization (Bregler et al ’00) Assumption: SVD preserves order and orientation of basis shape components

15. Computer Vision Results (Bregler et al ’00)

16. Computer Vision Dynamic SfM factorization (Brand ’01) constraints to be satisfied for M constraints to be satisfied for M, use to compute J hard! (different methods are possible, not so simple and also not optimal)

17. Computer Vision Non-rigid 3D subspace flow Same is also possible using optical flow in stead of features, also takes uncertainty into account (Brand ’01)

18. Computer Vision Results (Brand ’01)

19. Computer Vision (Brand ’01) Results

20. Computer Vision Results (Bregler et al ’01)

21. Computer Vision PROJECTIVE STRUCTURE FROM MOTION Reading: Chapter 13. The Projective Structure from Motion Problem Elements of Projective Geometry Projective Structure and Motion from Two Images Projective Motion from Fundamental Matrices Projective Structure and Motion from Multiple Images

22. Computer Vision The Projective Structure-from-Motion Problem Given m perspective images of n fixed points P we can write Problem: estimate the m 3x4 matrices M and the n positions P from the mn correspondences p. i j ij 2mn equations in 11m+3n unknowns Overconstrained problem, that can be solved using (non-linear) least squares! j

23. Computer Vision The Projective Ambiguity of Projective SFM If M and P are solutions, i j So are M’ and P’ where i j and Q is an arbitrary non-singular 4x4 matrix. When the intrinsic and extrinsic parameters are unknown Q is a projective transformation.

24. Computer Vision Projective Spaces: (Semi-Formal) Definition

25. Computer Vision A Model of P( R ) 3

26. Computer Vision Projective Subspaces and Projective Coordinates

27. Computer Vision Projective Subspaces and Projective Coordinates Projective coordinates P

28. Computer Vision Projective Subspaces Given a choice of coordinate frame Line:Plane:

29. Computer Vision Affine and Projective Spaces

30. Computer Vision Affine and Projective Coordinates

31. Computer Vision Cross-Ratios Collinear points Pencil of coplanar lines Pencil of planes {A,B;C,D}= sin(  +  )sin(  +  ) sin(  +  +  )sin 

32. Computer Vision Cross-Ratios and Projective Coordinates Along a line equipped with the basis In a plane equipped with the basis In 3-space equipped with the basis * *

33. Computer Vision Projective Transformations Bijective linear map: Projective transformation: ( = homography ) Projective transformations map projective subspaces onto projective subspaces and preserve projective coordinates. Projective transformations map lines onto lines and preserve cross-ratios.

34. Computer Vision Perspective Projections induce projective transformations between planes.

35. Computer Vision Geometric Scene Reconstruction Idea: use (A,O”,O’,B,C) as a projective basis.

36. Computer Vision Reprinted from “Relative Stereo and Motion Reconstruction,” by J. Ponce, T.A. Cass and D.H. Marimont, Tech. Report UIUC-BI-AI-RCV-93-07, Beckman Institute, Univ. of Illinois (1993).

37. Computer Vision Motion estimation from fundamental matrices Q Facts: b’ can be found using LLS. Once M and M’ are known, P can be computed with LLS.

38. Computer Vision Projective Structure from Motion and Factorization Factorization?? Algorithm (Sturm and Triggs, 1996) Guess the depths; Factorize D ; Iterate. Does it converge? (Mahamud and Hebert, 2000)

39. Computer Vision Bundle adjustment - refining structure and motion Minimize reprojection error –Maximum Likelyhood Estimation (if error zero-mean Gaussian noise) –Huge problem but can be solved efficiently (exploit sparseness)

40. Computer Vision Developed in photogrammetry in 50´s Bundle adjustment

41. Computer Vision Non-linear least squares Linear approximation of residual allows quadratic approximation of sum- of-squares N (extra term = descent term) Minimization corresponds to finding zeros of derivative Levenberg-Marquardt: extra term to deal with singular N (decrease/increase if success/failure to descent)

42. Computer Vision Bundle adjustment U1U1 U2U2 U3U3 WTWT W V P1P1 P2P2 P3P3 M Jacobian of has sparse block structure –cameras independent of other cameras, –points independent of other points 12xm 3xn (in general much larger) im.pts. view 1 Needed for non-linear minimization

43. Computer Vision Bundle adjustment Eliminate dependence of camera/motion parameters on structure parameters Note in general 3n >> 11m WTWT V U-WV -1 W T 11xm 3xn Allows much more efficient computations e.g. 100 views,10000 points, solve  1000x1000, not  30000x30000 Often still band diagonal use sparse linear algebra algorithms

44. Computer Vision Sequential SfM Initialize motion from two images Initialize structure For each additional view –Determine pose of camera –Refine and extend structure Refine structure and motion

45. Computer Vision Initial projective camera motion Choose P and P´compatible with F Reconstruction up to projective ambiguity (reference plane;arbitrary) Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion Same for more views? different projective basis

46. Computer Vision Initializing projective structure Reconstruct matches in projective frame by minimizing the reprojection error Non-iterative optimal solution Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion

47. Computer Vision Projective pose estimation Infere 2D-3D matches from 2D-2D matches Compute pose from (RANSAC,6pts) F X x Inliers: Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion

48. Computer Vision Refining structure Extending structure 2-view triangulation (Iterative linear) Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion Refining and extending structure

49. Computer Vision Refining structure and motion use bundle adjustment Also model radial distortion to avoid bias!

50. Computer Vision Hierarchical structure and motion recovery Compute 2-view Compute 3-view Stitch 3-view reconstructions Merge and refine reconstruction F T H PM

51. Computer Vision Metric structure and motion Note that a fundamental problem of the uncalibrated approach is that it fails if a purely planar scene is observed (in one or more views) (solution possible based on model selection) use self-calibration (see next class)

52. Computer Vision Dealing with dominant planes

53. Computer Vision PPPgric HHgric

54. Computer Vision Farmhouse 3D models (note: reconstruction much larger than camera field-of-view)

55. Computer Vision Application: video augmentation

56. Computer Vision Next class: Camera calibration (and self-calibration) Reading: Chapter 2 and 3