1. Stanford CS223B Computer Vision, Winter 2006 Lecture 8 Structure From Motion Professor Sebastian Thrun CAs: Dan Maynes-Aminzade, Mitul Saha, Greg Corrado Slides by: Gary Bradski, Intel Research and Stanford SAIL

2. Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion camera features Recover: structure (feature locations), motion (camera extrinsics)

3. Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion (1) [Tomasi & Kanade 92]

4. Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion (2) [Tomasi & Kanade 92]

5. Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion (3) [Tomasi & Kanade 92]

6. Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion (4a): Images Marc Pollefeys

7. Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion (4b) Marc Pollefeys

8. Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion n Problem 1: –Given n points p ij =(x ij, y ij ) in m images –Reconstruct structure: 3-D locations P j =(x j, y j, z j ) –Reconstruct camera positions (extrinsics) M i =(A j, b j ) n Problem 2: –Establish correspondence: c(p ij )

9. Sebastian Thrun Stanford University CS223B Computer Vision SFM: General Formulation fZ X O -x

10. Sebastian Thrun Stanford University CS223B Computer Vision SFM: Bundle Adjustment fZ X O -x

11. Sebastian Thrun Stanford University CS223B Computer Vision Bundle Adjustment n SFM = Nonlinear Least Squares problem n Minimize through –Gradient Descent –Conjugate Gradient –Gauss-Newton –Levenberg Marquardt (!) n Prone to local minima

12. Sebastian Thrun Stanford University CS223B Computer Vision Count # Constraints vs #Unknowns n m camera poses n n points n 2mn point constraints n 6m+3n unknowns n Suggests: need 2mn  6m + 3n n But: Can we really recover all parameters???

13. Sebastian Thrun Stanford University CS223B Computer Vision How Many Parameters Can’t We Recover? 036781012nmnm Place Your Bet! We can recover all but…

14. Sebastian Thrun Stanford University CS223B Computer Vision Count # Constraints vs #Unknowns n m camera poses n n points n 2mn point constraints n 6m+3n unknowns n Suggests: need 2mn  6m + 3n n But: Can we really recover all parameters??? –Can’t recover origin, orientation (6 params) –Can’t recover scale (1 param) n Thus, we need 2mn  6m + 3n - 7

15. Sebastian Thrun Stanford University CS223B Computer Vision Are done? n No, bundle adjustment has many local minima.

16. Sebastian Thrun Stanford University CS223B Computer Vision The “Trick Of The Day” n Replace Perspective by Orthographic Geometry n Replace Euclidean Geometry by Affine Geometry n Solve SFM linearly (“closed” form, globally optimal) n Post-Process to make solution Euclidean n Post-Process to make solution perspective By Tomasi and Kanade, 1992

17. Sebastian Thrun Stanford University CS223B Computer Vision Orthographic Camera Model Limit of Pinhole Model: Extrinsic Parameters Rotation Orthographic Projection

18. Sebastian Thrun Stanford University CS223B Computer Vision Orthographic Projection Limit of Pinhole Model: Orthographic Projection

19. Sebastian Thrun Stanford University CS223B Computer Vision The Orthographic SFM Problem subject to

20. Sebastian Thrun Stanford University CS223B Computer Vision The Affine SFM Problem subject to drop the constraints

21. Sebastian Thrun Stanford University CS223B Computer Vision Count # Constraints vs #Unknowns n m camera poses n n points n 2mn point constraints n 8m+3n unknowns n Suggests: need 2mn  8m + 3n n But: Can we really recover all parameters???

22. Sebastian Thrun Stanford University CS223B Computer Vision How Many Parameters Can’t We Recover? 036781012nmnm Place Your Bet! We can recover all but…

23. Sebastian Thrun Stanford University CS223B Computer Vision The Answer is (at least): 12

24. Sebastian Thrun Stanford University CS223B Computer Vision Points for Solving Affine SFM Problem n m camera poses n n points n Need to have: 2mn  8m + 3n-12

25. Sebastian Thrun Stanford University CS223B Computer Vision Affine SFM Fix coordinate system by making p 0 =origin Proof: Rank Theorem: Q has rank 3

26. Sebastian Thrun Stanford University CS223B Computer Vision The Rank Theorem n elements 2m elements

27. Sebastian Thrun Stanford University CS223B Computer Vision Singular Value Decomposition

28. Sebastian Thrun Stanford University CS223B Computer Vision Affine Solution to Orthographic SFM Gives also the optimal affine reconstruction under noise

29. Sebastian Thrun Stanford University CS223B Computer Vision Back To Orthographic Projection Find C and d for which constraints are met Search in 12-dim space (instead of 8m + 3n-12)

30. Sebastian Thrun Stanford University CS223B Computer Vision Back To Projective Geometry Orthographic (in the limit) Projective

31. Sebastian Thrun Stanford University CS223B Computer Vision Back To Projective Geometry fZ X O -x Optimize Using orthographic solution as starting point

32. Sebastian Thrun Stanford University CS223B Computer Vision The “Trick Of The Day” n Replace Perspective by Orthographic Geometry n Replace Euclidean Geometry by Affine Geometry n Solve SFM linearly (“closed” form, globally optimal) n Post-Process to make solution Euclidean n Post-Process to make solution perspective By Tomasi and Kanade, 1992

33. Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion n Problem 1: –Given n points p ij =(x ij, y ij ) in m images –Reconstruct structure: 3-D locations P j =(x j, y j, z j ) –Reconstruct camera positions (extrinsics) M i =(A j, b j ) n Problem 2: –Establish correspondence: c(p ij )

34. Sebastian Thrun Stanford University CS223B Computer Vision The Correspondence Problem View 1View 3View 2

35. Sebastian Thrun Stanford University CS223B Computer Vision Correspondence: Solution 1 n Track features (e.g., optical flow) n …but fails when images taken from widely different poses

36. Sebastian Thrun Stanford University CS223B Computer Vision Correspondence: Solution 2 n Start with random solution A, b, P n Compute soft correspondence: p(c|A,b,P) n Plug soft correspondence into SFM n Reiterate See Dellaert/Seitz/Thorpe/Thrun, Machine Learning Journal, 2003

37. Sebastian Thrun Stanford University CS223B Computer Vision Example

38. Sebastian Thrun Stanford University CS223B Computer Vision Results: Cube

39. Sebastian Thrun Stanford University CS223B Computer Vision Animation

40. Sebastian Thrun Stanford University CS223B Computer Vision Tomasi’s Benchmark Problem

41. Sebastian Thrun Stanford University CS223B Computer Vision Reconstruction with EM

42. Sebastian Thrun Stanford University CS223B Computer Vision 3-D Structure

43. Sebastian Thrun Stanford University CS223B Computer Vision Correspondence: Alternative Approach n Ransac [Fisher/Bolles] = Random sampling and consensus

44. Sebastian Thrun Stanford University CS223B Computer Vision Summary SFM n Problem –Determine feature locations (=structure) –Determine camera extrinsic (=motion) n Two Principal Solutions –Bundle adjustment (nonlinear least squares, local minima) –SVD (through orthographic approximation, affine geometry) n Correspondence –(RANSAC) –Expectation Maximization