1. Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches : –http://www.di.ens.fr/~ponce/geomvis/lect4.ppthttp://www.di.ens.fr/~ponce/geomvis/lect4.ppt –http://www.di.ens.fr/~ponce/geomvis/lect4.pdfhttp://www.di.ens.fr/~ponce/geomvis/lect4.pdf

2. Epipolar Constraint Potential matches for p have to lie on the corresponding epipolar line l’. Potential matches for p’ have to lie on the corresponding epipolar line l.

3. Epipolar Constraint: Calibrated Case Essential Matrix (Longuet-Higgins, 1981)

4. Properties of the Essential Matrix E p’ is the epipolar line associated with p’. E p is the epipolar line associated with p. E e’=0 and E e=0. E is singular. E has two equal non-zero singular values (Huang and Faugeras, 1989). T T

5. Epipolar Constraint: Small Motions To First-Order: Pure translation: Focus of Expansion

6. Epipolar Constraint: Uncalibrated Case Fundamental Matrix (Faugeras and Luong, 1992)

7. Properties of the Fundamental Matrix F p’ is the epipolar line associated with p’. F p is the epipolar line associated with p. F e’=0 and F e=0. F is singular. T T

8. The Eight-Point Algorithm (Longuet-Higgins, 1981) | F | =1. Minimize: under the constraint 2

9. Non-Linear Least-Squares Approach (Luong et al., 1993) Minimize with respect to the coefficients of F, using an appropriate rank-2 parameterization.

10. The Normalized Eight-Point Algorithm (Hartley, 1995) Center the image data at the origin, and scale it so the mean squared distance between the origin and the data points is 2 pixels: q = T p, q’ = T’ p’. Use the eight-point algorithm to compute F from the points q and q’. Enforce the rank-2 constraint. Output T F T’. T iiii ii

11. Data courtesy of R. Mohr and B. Boufama.

12. Without normalization With normalization Mean errors: 10.0pixel 9.1pixel Mean errors: 1.0pixel 0.9pixel

13. Trinocular Epipolar Constraints These constraints are not independent!

14. Trinocular Epipolar Constraints: Transfer Given p and p, p can be computed as the solution of linear equations. 123

15. Trifocal Constraints

16. All 3x3 minors must be zero! Calibrated Case Trifocal Tensor

17. Trifocal Constraints Uncalibrated Case Trifocal Tensor

18. Trifocal Constraints: 3 Points Pick any two lines l and l through p and p. Do it again. 23 23 T( p, p, p )=0 12 3

19. Properties of the Trifocal Tensor Estimating the Trifocal Tensor Ignore the non-linear constraints and use linear least-squares a posteriori. Impose the constraints a posteriori. For any matching epipolar lines, l G l = 0. The matrices G are singular. They satisfy 8 independent constraints in the uncalibrated case (Faugeras and Mourrain, 1995). 213 Ti 1 i

20. For any matching epipolar lines, l G l = 0. 213Ti The backprojections of the two lines do not define a line!

21. Multiple Views (Faugeras and Mourrain, 1995)

22. Two Views Epipolar Constraint

23. Three Views Trifocal Constraint

24. Four Views Quadrifocal Constraint (Triggs, 1995)

25. Geometrically, the four rays must intersect in P..

26. Quadrifocal Tensor and Lines

27. Scale-Restraint Condition from Photogrammetry

28. The Euclidean (perspective) Structure-from-Motion Problem Given m calibrated perspective images of n fixed points P j we can write Problem: estimate the m 3x4 matrices M i = [R i t i ] and the n positions P j from the mn correspondences p ij. 2mn equations in 11m+3n unknowns Overconstrained problem, that can be solved using (non-linear) least squares!

29. The Euclidean Ambiguity of Euclidean SFM If R i, t i, and P j are solutions, So are R i ’, t i ’, and P j ’, where In fact, the absolute scale cannot be recovered since: When the intrinsic and extrinsic parameters are known Euclidean ambiguity up to a similarity transformation.

30. The Affine Structure-from-Motion Problem Given m images of n fixed points P we can write Problem: estimate the m 2x4 matrices M and the n positions P from the mn correspondences p. i j ij 2mn equations in 8m+3n unknowns Overconstrained problem, that can be solved using (non-linear) least squares! j

31. The Affine Ambiguity of Affine SFM If M and P are solutions, i j So are M’ and P’ where i j and Q is an affine transformation. When the intrinsic and extrinsic parameters are unknown

32. The Affine Epipolar Constraint Note: the epipolar lines are parallel.

33. Affine Epipolar Geometry

34. The Affine Fundamental Matrix where

35. Without normalization With normalization Mean errors: 10.0pixel 9.1pixel Mean errors: 1.0pixel 0.9pixel Perspective case..

36. Mean errors: 3.24 and 3.15pixel (without normalization 160.92 and 158.54pixel). Affine case..

37. An Affine Trick..

38. The Affine Epipolar Constraint Note: the epipolar lines are parallel.

39. An Affine Trick..Algebraic Scene Reconstruction Method

40. Affine reconstruction. Mean relative error: 3.2%

41. The Affine Structure of Affine Images Suppose we observe a static scene with m fixed cameras.. The set m-tuples of all image points in a scene is a 3D affine space!

42. has rank 4!

43. From Affine to Vectorial Structure Idea: pick one of the points (or their center of mass) as the origin.

44. Singular Value Decomposition

45. square roots of

46. Singular Value Decomposition

48. What if we could factorize D? (Tomasi and Kanade, 1992) Affine SFM is solved! Singular Value Decomposition We can take

49. Affine reconstruction. Mean relative error: 2.8%

50. Back to perspective: Euclidean motion from E (Longuet-Higgins, 1981) Given F computed from n > 7 point correspondences, and its SVD F= UWV T, compute E=U diag(1,1,0) V T. There are two solutions t’ = u 3 and t’’ = -t’ to E T t=0. Define R’ = UWV T and R” = UW T V T where (It is easy to check R’ and R” are rotations.) Then [t x ’]R’ = -E and [t x ’]R” = E. Similar reasoning for t”. Four solutions. Only two of them place the reconstructed points in front of the cameras.

51. Euclidean reconstruction. Mean relative error: 3.1%

52. A different view of the fundamental matrix Projective ambiguity ! M’Q=[Id 0] MQ=[A b]. Hence: zp = [A b] P and z’p’ = [Id 0] P, with P=(x,y,z,1) T. This can be rewritten as: zp = ( A [Id 0] + [0 b] ) P = z’Ap’ + b. Or: z (b x p) = z’ (b x Ap’). Finally: p T Fp’ = 0 with F = [b x ] A.

53. Projective motion from the fundamental matrix Given F computed from n > 7 point correspondences, compute b as the solution of F T b=0 with |b| 2 =1. Note that: [a x ] 2 = aa T - |a| 2 Id for any a. Thus, if A 0 = - [b x ] F, [b x ] A 0 = - [b x ] 2 F = - bb T F + |b| 2 F = F. The general solution is M = [A b] with A = A 0 + (  b | b |  b).

54. Two-view projective reconstruction. Mean relative error: 3.0%

55. Bundle adjustment Use nonlinear least-squares to minimize:

56. Bundle adjustment. Mean relative error: 0.2%

57. Projective SFM from multiple images z 11 p 11 … z 1n p 1n … … … z m1 p m1 … z mn p mn M1…MmM1…Mm P_1 … P_n =, D = MP If the z ij ’s are known, can be done via SVD. In principle the z ij ’s can be found pairwise from F (Triggs 96). Alternative, eliminate z ij from the minimization of E=|D-MP| 2 This reduces the problem to the minimization of E =  ij |p ij x M i P j | 2 under the constraints |M i | 2 =|P j | 2 =1 with |p ij | 2 =1. Bilinear problem.

58. Bilinear projective reconstruction. Mean relative error: 0.2%

59. From uncalibrated to calibrated cameras Weak-perspective camera: Calibrated camera: Problem: what is Q ? Note: Absolute scale cannot be recovered. The Euclidean shape (defined up to an arbitrary similitude) is recovered.

60. Reconstruction Results (Tomasi and Kanade, 1992) Reprinted from “Factoring Image Sequences into Shape and Motion,” by C. Tomasi and T. Kanade, Proc. IEEE Workshop on Visual Motion (1991).  1991 IEEE.

61. What is some parameters are known? Weak-perspective camera: Zero skew: Problem: what is Q ? 0 Self calibration!

62. П1П1 Chasles’ absolute conic: x 1 2 +x 2 2 +x 3 2 = 0, x 4 = 0. Kruppa (1913); Maybank & Faugeras (1992) Triggs (1997); Pollefeys et al. (1998,2002)  ,  u 0, v 0 The absolute quadric u 0 = v 0 = 0 The absolute quadratic complex  2 =  2,  = 0 u0u0 v0v0 k l f x’ ≈ P ( H H -1 ) x H = [ X y ]

63. Relation between K, , and  *