Computer Vision Projective structure from motion Marc Pollefeys COMP 256 Some slides and illustrations from J. Ponce, …

Computer Vision Last class: Affine camera The affine projection equations are how to find the origin? or for that matter a 3D reference point? affine projection preserves center of gravity

Computer Vision Orthographic factorization The orthographic projection equations are where All equations can be collected for all i and j where Note that P and X are resp. 2 m x3 and 3x n matrices and therefore the rank of x is at most 3

Computer Vision Orthographic factorization Factorize m through singular value decomposition An affine reconstruction is obtained as follows Closest rank-3 approximation yields MLE!

Computer Vision Aug 26/28-Introduction Sep 2/4CamerasRadiometry Sep 9/11Sources & ShadowsColor Sep 16/18Linear filters & edges(hurricane Isabel) Sep 23/25Pyramids & TextureMulti-View Geometry Sep30/Oct2StereoProject proposals Oct 7/9Tracking (Welch)Optical flow Oct 14/16-- Oct 21/23Silhouettes/carving(Fall break) Oct 28/30-Structure from motion Nov 4/6Project updateProj. SfM Nov 11/13Camera calibrationSegmentation Nov 18/20FittingProb. segm.&fit. Nov 25/27Matching templates(Thanksgiving) Dec 2/4Matching relationsRange data Dec ?Final project Tentative class schedule

Computer Vision Further Factorization work Factorization with uncertainty Factorization for indep. moving objects Factorization for dynamic objects Perspective factorization Factorization with outliers and missing pts. (Irani & Anandan, IJCV’02) (Costeira and Kanade ‘94) (Bregler et al. 2000, Brand 2001) (Jacobs 1997 (affine), Martinek and Pajdla 2001, Aanaes 2002 (perspective)) (Sturm & Triggs 1996, …)

Computer Vision Structure from motion of multiple moving objects one object: multiple objects:

Computer Vision SfM of multiple moving objects

Computer Vision Structure from motion of deforming objects Extend factorization approaches to deal with dynamic shapes (Bregler et al ’00; Brand ‘01)

Computer Vision Representing dynamic shapes represent dynamic shape as varying linear combination of basis shapes (fig. M.Brand)

Computer Vision Projecting dynamic shapes (figs. M.Brand) Rewrite:

Computer Vision Dynamic image sequences One image: Multiple images (figs. M.Brand)

Computer Vision Dynamic SfM factorization? Problem: find J so that M has proper structure

Computer Vision Dynamic SfM factorization (Bregler et al ’00) Assumption: SVD preserves order and orientation of basis shape components

Computer Vision Results (Bregler et al ’00)

Computer Vision Dynamic SfM factorization (Brand ’01) constraints to be satisfied for M constraints to be satisfied for M, use to compute J hard! (different methods are possible, not so simple and also not optimal)

Computer Vision Non-rigid 3D subspace flow Same is also possible using optical flow in stead of features, also takes uncertainty into account (Brand ’01)

Computer Vision Results (Brand ’01)

Computer Vision (Brand ’01) Results

Computer Vision Results (Bregler et al ’01)

Computer Vision PROJECTIVE STRUCTURE FROM MOTION Reading: Chapter 13. The Projective Structure from Motion Problem Elements of Projective Geometry Projective Structure and Motion from Two Images Projective Motion from Fundamental Matrices Projective Structure and Motion from Multiple Images

Computer Vision The Projective Structure-from-Motion Problem Given m perspective images of n fixed points P we can write Problem: estimate the m 3x4 matrices M and the n positions P from the mn correspondences p. i j ij 2mn equations in 11m+3n unknowns Overconstrained problem, that can be solved using (non-linear) least squares! j

Computer Vision The Projective Ambiguity of Projective SFM If M and P are solutions, i j So are M’ and P’ where i j and Q is an arbitrary non-singular 4x4 matrix. When the intrinsic and extrinsic parameters are unknown Q is a projective transformation.

Computer Vision Projective Spaces: (Semi-Formal) Definition

Computer Vision A Model of P( R ) 3

Computer Vision Projective Subspaces and Projective Coordinates

Computer Vision Projective Subspaces and Projective Coordinates Projective coordinates P

Computer Vision Projective Subspaces Given a choice of coordinate frame Line:Plane:

Computer Vision Affine and Projective Spaces

Computer Vision Affine and Projective Coordinates

Computer Vision Cross-Ratios Collinear points Pencil of coplanar lines Pencil of planes {A,B;C,D}= sin(  +  )sin(  +  ) sin(  +  +  )sin 

Computer Vision Cross-Ratios and Projective Coordinates Along a line equipped with the basis In a plane equipped with the basis In 3-space equipped with the basis * *

Computer Vision Projective Transformations Bijective linear map: Projective transformation: ( = homography ) Projective transformations map projective subspaces onto projective subspaces and preserve projective coordinates. Projective transformations map lines onto lines and preserve cross-ratios.

Computer Vision Perspective Projections induce projective transformations between planes.

Computer Vision Geometric Scene Reconstruction Idea: use (A,O”,O’,B,C) as a projective basis.

Computer Vision Reprinted from “Relative Stereo and Motion Reconstruction,” by J. Ponce, T.A. Cass and D.H. Marimont, Tech. Report UIUC-BI-AI-RCV-93-07, Beckman Institute, Univ. of Illinois (1993).

Computer Vision Motion estimation from fundamental matrices Q Facts: b’ can be found using LLS. Once M and M’ are known, P can be computed with LLS.

Computer Vision Projective Structure from Motion and Factorization Factorization?? Algorithm (Sturm and Triggs, 1996) Guess the depths; Factorize D ; Iterate. Does it converge? (Mahamud and Hebert, 2000)

Computer Vision Bundle adjustment - refining structure and motion Minimize reprojection error –Maximum Likelyhood Estimation (if error zero-mean Gaussian noise) –Huge problem but can be solved efficiently (exploit sparseness)

Computer Vision Developed in photogrammetry in 50´s Bundle adjustment

Computer Vision Non-linear least squares Linear approximation of residual allows quadratic approximation of sum- of-squares N (extra term = descent term) Minimization corresponds to finding zeros of derivative Levenberg-Marquardt: extra term to deal with singular N (decrease/increase if success/failure to descent)

Computer Vision Bundle adjustment U1U1 U2U2 U3U3 WTWT W V P1P1 P2P2 P3P3 M Jacobian of has sparse block structure –cameras independent of other cameras, –points independent of other points 12xm 3xn (in general much larger) im.pts. view 1 Needed for non-linear minimization

Computer Vision Bundle adjustment Eliminate dependence of camera/motion parameters on structure parameters Note in general 3n >> 11m WTWT V U-WV -1 W T 11xm 3xn Allows much more efficient computations e.g. 100 views,10000 points, solve  1000x1000, not  30000x30000 Often still band diagonal use sparse linear algebra algorithms

Computer Vision Sequential SfM Initialize motion from two images Initialize structure For each additional view –Determine pose of camera –Refine and extend structure Refine structure and motion

Computer Vision Initial projective camera motion Choose P and P´compatible with F Reconstruction up to projective ambiguity (reference plane;arbitrary) Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion Same for more views? different projective basis

Computer Vision Initializing projective structure Reconstruct matches in projective frame by minimizing the reprojection error Non-iterative optimal solution Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion

Computer Vision Projective pose estimation Infere 2D-3D matches from 2D-2D matches Compute pose from (RANSAC,6pts) F X x Inliers: Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion

Computer Vision Refining structure Extending structure 2-view triangulation (Iterative linear) Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion Refining and extending structure

Computer Vision Refining structure and motion use bundle adjustment Also model radial distortion to avoid bias!

Computer Vision Hierarchical structure and motion recovery Compute 2-view Compute 3-view Stitch 3-view reconstructions Merge and refine reconstruction F T H PM

Computer Vision Metric structure and motion Note that a fundamental problem of the uncalibrated approach is that it fails if a purely planar scene is observed (in one or more views) (solution possible based on model selection) use self-calibration (see next class)

Computer Vision Dealing with dominant planes

Computer Vision PPPgric HHgric

Computer Vision Farmhouse 3D models (note: reconstruction much larger than camera field-of-view)

Computer Vision Application: video augmentation

Computer Vision Next class: Camera calibration (and self-calibration) Reading: Chapter 2 and 3