Stanford CS223B Computer Vision, Winter 2006 Lecture 8 Structure From Motion Professor Sebastian Thrun CAs: Dan Maynes-Aminzade, Mitul Saha, Greg Corrado Slides by: Gary Bradski, Intel Research and Stanford SAIL

Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion camera features Recover: structure (feature locations), motion (camera extrinsics)

Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion (1) [Tomasi & Kanade 92]

Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion (2) [Tomasi & Kanade 92]

Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion (3) [Tomasi & Kanade 92]

Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion (4a): Images Marc Pollefeys

Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion (4b) Marc Pollefeys

Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion n Problem 1: –Given n points p ij =(x ij, y ij ) in m images –Reconstruct structure: 3-D locations P j =(x j, y j, z j ) –Reconstruct camera positions (extrinsics) M i =(A j, b j ) n Problem 2: –Establish correspondence: c(p ij )

Sebastian Thrun Stanford University CS223B Computer Vision SFM: General Formulation fZ X O -x

Sebastian Thrun Stanford University CS223B Computer Vision SFM: Bundle Adjustment fZ X O -x

Sebastian Thrun Stanford University CS223B Computer Vision Bundle Adjustment n SFM = Nonlinear Least Squares problem n Minimize through –Gradient Descent –Conjugate Gradient –Gauss-Newton –Levenberg Marquardt (!) n Prone to local minima

Sebastian Thrun Stanford University CS223B Computer Vision Count # Constraints vs #Unknowns n m camera poses n n points n 2mn point constraints n 6m+3n unknowns n Suggests: need 2mn  6m + 3n n But: Can we really recover all parameters???

Sebastian Thrun Stanford University CS223B Computer Vision How Many Parameters Can’t We Recover? nmnm Place Your Bet! We can recover all but…

Sebastian Thrun Stanford University CS223B Computer Vision Count # Constraints vs #Unknowns n m camera poses n n points n 2mn point constraints n 6m+3n unknowns n Suggests: need 2mn  6m + 3n n But: Can we really recover all parameters??? –Can’t recover origin, orientation (6 params) –Can’t recover scale (1 param) n Thus, we need 2mn  6m + 3n - 7

Sebastian Thrun Stanford University CS223B Computer Vision Are done? n No, bundle adjustment has many local minima.

Sebastian Thrun Stanford University CS223B Computer Vision The “Trick Of The Day” n Replace Perspective by Orthographic Geometry n Replace Euclidean Geometry by Affine Geometry n Solve SFM linearly (“closed” form, globally optimal) n Post-Process to make solution Euclidean n Post-Process to make solution perspective By Tomasi and Kanade, 1992

Sebastian Thrun Stanford University CS223B Computer Vision Orthographic Camera Model Limit of Pinhole Model: Extrinsic Parameters Rotation Orthographic Projection

Sebastian Thrun Stanford University CS223B Computer Vision Orthographic Projection Limit of Pinhole Model: Orthographic Projection

Sebastian Thrun Stanford University CS223B Computer Vision The Orthographic SFM Problem subject to

Sebastian Thrun Stanford University CS223B Computer Vision The Affine SFM Problem subject to drop the constraints

Sebastian Thrun Stanford University CS223B Computer Vision Count # Constraints vs #Unknowns n m camera poses n n points n 2mn point constraints n 8m+3n unknowns n Suggests: need 2mn  8m + 3n n But: Can we really recover all parameters???

Sebastian Thrun Stanford University CS223B Computer Vision How Many Parameters Can’t We Recover? nmnm Place Your Bet! We can recover all but…

Sebastian Thrun Stanford University CS223B Computer Vision The Answer is (at least): 12

Sebastian Thrun Stanford University CS223B Computer Vision Points for Solving Affine SFM Problem n m camera poses n n points n Need to have: 2mn  8m + 3n-12

Sebastian Thrun Stanford University CS223B Computer Vision Affine SFM Fix coordinate system by making p 0 =origin Proof: Rank Theorem: Q has rank 3

Sebastian Thrun Stanford University CS223B Computer Vision The Rank Theorem n elements 2m elements

Sebastian Thrun Stanford University CS223B Computer Vision Singular Value Decomposition

Sebastian Thrun Stanford University CS223B Computer Vision Affine Solution to Orthographic SFM Gives also the optimal affine reconstruction under noise

Sebastian Thrun Stanford University CS223B Computer Vision Back To Orthographic Projection Find C and d for which constraints are met Search in 12-dim space (instead of 8m + 3n-12)

Sebastian Thrun Stanford University CS223B Computer Vision Back To Projective Geometry Orthographic (in the limit) Projective

Sebastian Thrun Stanford University CS223B Computer Vision Back To Projective Geometry fZ X O -x Optimize Using orthographic solution as starting point

Sebastian Thrun Stanford University CS223B Computer Vision The “Trick Of The Day” n Replace Perspective by Orthographic Geometry n Replace Euclidean Geometry by Affine Geometry n Solve SFM linearly (“closed” form, globally optimal) n Post-Process to make solution Euclidean n Post-Process to make solution perspective By Tomasi and Kanade, 1992

Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion n Problem 1: –Given n points p ij =(x ij, y ij ) in m images –Reconstruct structure: 3-D locations P j =(x j, y j, z j ) –Reconstruct camera positions (extrinsics) M i =(A j, b j ) n Problem 2: –Establish correspondence: c(p ij )

Sebastian Thrun Stanford University CS223B Computer Vision The Correspondence Problem View 1View 3View 2

Sebastian Thrun Stanford University CS223B Computer Vision Correspondence: Solution 1 n Track features (e.g., optical flow) n …but fails when images taken from widely different poses

Sebastian Thrun Stanford University CS223B Computer Vision Correspondence: Solution 2 n Start with random solution A, b, P n Compute soft correspondence: p(c|A,b,P) n Plug soft correspondence into SFM n Reiterate See Dellaert/Seitz/Thorpe/Thrun, Machine Learning Journal, 2003

Sebastian Thrun Stanford University CS223B Computer Vision Example

Sebastian Thrun Stanford University CS223B Computer Vision Results: Cube

Sebastian Thrun Stanford University CS223B Computer Vision Animation

Sebastian Thrun Stanford University CS223B Computer Vision Tomasi’s Benchmark Problem

Sebastian Thrun Stanford University CS223B Computer Vision Reconstruction with EM

Sebastian Thrun Stanford University CS223B Computer Vision 3-D Structure

Sebastian Thrun Stanford University CS223B Computer Vision Correspondence: Alternative Approach n Ransac [Fisher/Bolles] = Random sampling and consensus

Sebastian Thrun Stanford University CS223B Computer Vision Summary SFM n Problem –Determine feature locations (=structure) –Determine camera extrinsic (=motion) n Two Principal Solutions –Bundle adjustment (nonlinear least squares, local minima) –SVD (through orthographic approximation, affine geometry) n Correspondence –(RANSAC) –Expectation Maximization