1. Projective structure from motion
Marc Pollefeys
COMP 256
Some slides and illustrations from J. Ponce, …

2. Last class: Affine camera
The affine projection equations are
how to find the origin? or for that matter a 3D reference point?
affine projection preserves center of gravity

3. Orthographic factorization
The orthographic projection equations are
where
All equations can be collected for all i and j
where
Note that P and X are resp. 2mx3 and 3xn matrices and therefore the rank of x is at most 3

4. Orthographic factorization
Factorize m through singular value decomposition
An affine reconstruction is obtained as follows
Closest rank-3 approximation yields MLE!

5. Tentative class schedule
Jan 16/18 -
Introduction
Jan 23/25
Cameras
Radiometry
Jan 30/Feb1
Sources & Shadows
Color
Feb 6/8
Linear filters & edges
Texture
Feb 13/15
Multi-View Geometry
Stereo
Feb 20/22
Optical flow
Project proposals
Feb27/Mar1
Affine SfM
Projective SfM
Mar 6/8
Camera Calibration
Silhouettes and Photoconsistency
Mar 13/15
Springbreak
Mar 20/22
Segmentation
Fitting
Mar 27/29
Prob. Segmentation
Project Update
Apr 3/5
Tracking
Apr 10/12
Object Recognition
Apr 17/19
Range data
Apr 24/26
Final project

6. PROJECTIVE STRUCTURE FROM MOTION
The Projective Structure from Motion Problem
Elements of Projective Geometry
Projective Structure and Motion from Two Images
Projective Motion from Fundamental Matrices
Projective Structure and Motion from Multiple Images
Reading: Chapter 13.

7. The Projective Structure-from-Motion Problem
Given m perspective images of n fixed points P we can write j
Problem: estimate the m 3x4 matrices M and
the n positions P from the mn correspondences p . i j ij
2mn equations in 11m+3n unknowns
Overconstrained problem, that can be solved
using (non-linear) least squares!

8. The Projective Ambiguity of Projective SFM
When the intrinsic and extrinsic parameters are unknown
If M and P are solutions, i j
So are M’ and P’ where i j
and Q is an arbitrary non-singular 4x4 matrix.
Q is a projective transformation.

9. Projective Spaces: (Semi-Formal) Definition

10. A Model of P( R ) 3

11. Projective Subspaces and Projective Coordinates

12. Projective Subspaces and Projective Coordinates

13. Projective Subspaces
Given a choice of coordinate frame
Line:
Plane:

14. Affine and Projective Spaces

15. Affine and Projective
Coordinates

16. Cross-Ratios
Collinear points
Pencil of coplanar lines
Pencil of planes
{A,B;C,D}=
sin(+)sin(+)
sin(++)sin

17. Cross-Ratios and Projective Coordinates*
Along a line equipped with the basis *
In a plane equipped with the basis
In 3-space equipped with the basis

18. Projective Transformations
Bijective linear map:
Projective transformation:
( = homography )
Projective transformations map projective subspaces onto
projective subspaces and preserve projective coordinates.
Projective transformations map lines onto lines and
preserve cross-ratios.

19. Perspective Projections induce projective transformations
between planes.

20. Geometric Scene
Reconstruction
Idea: use (A,O”,O’,B,C)
as a projective basis.

21. Reprinted from “Relative Stereo and Motion Reconstruction,” by J
Reprinted from “Relative Stereo and Motion Reconstruction,” by J. Ponce, T.A. Cass and D.H. Marimont, Tech. Report
UIUC-BI-AI-RCV-93-07, Beckman Institute, Univ. of Illinois (1993).

22. Motion estimation from fundamental matricesQ
Once M and M’ are known,
P can be computed with LLS.
Facts:
b’ can be found using LLS.

23. Projective Structure from Motion and Factorization
Algorithm (Sturm and Triggs, 1996)
Guess the depths;
Factorize D;
Iterate.
Does it converge? (Mahamud and Hebert, 2000)

24. Bundle adjustment - refining structure and motion
Minimize reprojection error
Maximum Likelyhood Estimation (if error zero-mean Gaussian noise)
Huge problem but can be solved efficiently (exploit sparseness)

25. Bundle adjustment
Developed in photogrammetry in 50´s

26. Non-linear least squares
Linear approximation of residual
allows quadratic approximation of sum-
of-squares
Minimization corresponds to finding zeros of derivative N
Levenberg-Marquardt: extra term to deal with singular N
(decrease/increase l if success/failure to descent)
(extra term = descent term)

27. Bundle adjustment Jacobian of has sparse block structure
cameras independent of other cameras,
points independent of other points U1 U2 U3 WT W V P1 P2 P3 M
im.pts.
view 1
12xm
3xn
(in general
much larger)
Needed for non-linear minimization

28. Bundle adjustment Allows much more efficient computations
Eliminate dependence of camera/motion parameters on structure parameters
Note in general 3n >> 11m WT V
U-WV-1WT
11xm
3xn
Allows much more efficient computations
e.g. 100 views,10000 points,
solve 1000x1000, not 30000x30000
Often still band diagonal
use sparse linear algebra algorithms

29. Sequential SfM Initialize motion from two images Initialize structure
For each additional view
Determine pose of camera
Refine and extend structure
Refine structure and motion

30. Initial projective camera motion
Choose P and P´compatible with F
Reconstruction up to projective ambiguity
(reference plane;arbitrary)
Same for more views?
Initialize motion
Initialize structure
For each additional view
Determine pose of camera
Refine and extend structure
Refine structure and motion
different projective basis

31. Initializing projective structure
Reconstruct matches in projective frame
by minimizing the reprojection error
Non-iterative optimal solution
Initialize motion
Initialize structure
For each additional view
Determine pose of camera
Refine and extend structure
Refine structure and motion

32. Projective pose estimation
Infere 2D-3D matches from 2D-2D matches
Compute pose from (RANSAC,6pts) X F x
Initialize motion
Initialize structure
For each additional view
Determine pose of camera
Refine and extend structure
Refine structure and motion
Inliers:

33. Refining and extending structure
Refining structure
Extending structure
2-view triangulation
(Iterative linear)
Initialize motion
Initialize structure
For each additional view
Determine pose of camera
Refine and extend structure
Refine structure and motion

34. Refining structure and motion
use bundle adjustment
Also model radial distortion to avoid bias!

35. Hierarchical structure and motion recovery
Compute 2-view
Compute 3-view
Stitch 3-view reconstructions
Merge and refine reconstruction F T H PM

36. Metric structure and motion
use self-calibration (see next class)
Note that a fundamental problem of the uncalibrated approach is
that it fails if a purely planar scene is observed (in one or more views)
(solution possible based on model selection)

37. Dealing with dominant planes

38. PPPgric
HHgric

39. Farmhouse 3D models
(note: reconstruction much larger than camera field-of-view)

40. Application: video augmentation

41. Next class: Camera calibration (and self-calibration)
Reading: Chapter 2 and 3