1. Jan-Michael Frahm, Philippos Mordohai
3D Urban Modeling
Marc Pollefeys
Jan-Michael Frahm, Philippos Mordohai
Fall 2006 / Comp
Tue & Thu 14:00-15:15
SN252

2. 3D Urban Modeling Basics: sensor, techniques, …
Video, LIDAR, GPS, IMU, …
SfM, Stereo, …
Robot Mapping, GIS, …
Review state-of-the-art
Video and LIDAR-based systems
Projects to experiment
Virtual UNC-Chapel Hill
DARPA Urban Challenge …

3. 3D Urban Modeling course schedule (tentative)
Sep. 5, 7
Introduction
Video-based Urban 3D Capture
(Jan-Michael)
Sep. 12, 14
Cameras
Epi. Geom. and Triangulation
Sep. 19, 21
Feature Tracking & Matching
(Sudipta)
Stereo matching
(Philippos)
Sep. 26, 28
GPS/INS/LIDAR
(Brian)
Project proposals
Oct. 3, 5
Oct. 10, 12
Oct. 17, 19
(fall break)
Oct. 24, 26
Project Status Update
Oct. 31, Nov. 2
Nov. 7, 9
Nov. 14, 16
Nov. 21, 23
(Thanksgiving)
Nov. 28, 30
Dec. 5
Project demonstrations
(classes ended)
Note: Dec. 3 is CVPR deadline

4. Course material Slides, notes, papers and references
see course webpage/wiki (later)
On-line “shape-from-video” tutorial:

5. Epipolar geometry?
courtesy Frank Dellaert

6. Fundamental matrix (3x3 rank 2 matrix)
Epipolar geometry l2 C1 m1 L1 m2 L2 M C2 C1 C2 l2 p l1 e1 e2 m1 L1 m2 L2 M C1 C2 l2 p l1 e1 e2
Underlying structure in set of matches for rigid scenes m1 m2
lT1 l2
Fundamental matrix (3x3 rank 2 matrix)
Computable from corresponding points
Simplifies matching
Allows to detect wrong matches
Related to calibration
Canonical representation:

7. Fundamental matrix (3x3 rank 2 matrix)
Epipolar geometry C1 C2 l2 P l1 e1 e2 C1 C2 l2 P l1 e1 e2 x1 L1 x2 L2 M
lT1 l2
Fundamental matrix (3x3 rank 2 matrix)

8. The projective reconstruction theorem
If a set of point correspondences in two views determine the fundamental matrix uniquely, then the scene and cameras may be reconstructed from these correspondences alone, and any two such reconstructions from these correspondences are projectively equivalent
allows reconstruction from pair of uncalibrated images!

9. Computation of F Linear (8-point) Minimal (7-point) Robust (RANSAC)
Non-linear refinement (MLE, …)
Practical approach

10. separate known from unknown
Epipolar geometry: basic equation
separate known from unknown
(data)
(unknowns)
(linear)

11. ! the NOT normalized 8-point algorithm Orders of magnitude difference
~10000
~100 1 !
Orders of magnitude difference
between column of data matrix
 least-squares yields poor results

12. Transform image to ~[-1,1]x[-1,1]
the normalized 8-point algorithm
Transform image to ~[-1,1]x[-1,1]
(0,0)
(700,500)
(700,0)
(0,500)
(1,-1)
(0,0)
(1,1)
(-1,1)
(-1,-1)
normalized least squares yields good results (Hartley, PAMI´97)

13. the singularity constraint
SVD from linearly computed F matrix (rank 3)
Compute closest rank-2 approximation

15. the minimum case – 7 point correspondences
one parameter family of solutions
but F1+lF2 not automatically rank 2

16. the minimum case – impose rank 2F1 F2 F 3
F7pts
(obtain 1 or 3 solutions)
(cubic equation)
Compute possible l as eigenvalues of
(only real solutions are potential solutions)
Minimal solution for calibrated cameras: 5-point

17. Robust estimation What if set of matches contains gross outliers?
(to keep things simple let’s consider line fitting first)

18. RANSAC (RANdom Sampling Consensus)
Objective
Robust fit of model to data set S which contains outliers
Algorithm
Randomly select a sample of s data points from S and instantiate the model from this subset.
Determine the set of data points Si which are within a distance threshold t of the model. The set Si is the consensus set of samples and defines the inliers of S.
If the subset of Si is greater than some threshold T, re-estimate the model using all the points in Si and terminate
If the size of Si is less than T, select a new subset and repeat the above.
After N trials the largest consensus set Si is selected, and the model is re-estimated using all the points in the subset Si

19. (dimension+codimension=dimension space)
Distance threshold
Choose t so probability for inlier is α (e.g. 0.95)
Often empirically
Zero-mean Gaussian noise σ then follows
distribution with m=codimension of model
(dimension+codimension=dimension space)
Codimension
Model
t 2 1
line,F
3.84σ2 2
H,P
5.99σ2 3 T
7.81σ2

20. proportion of outliers e
How many samples?
Choose N so that, with probability p, at least one random sample is free from outliers. e.g. p=0.99
proportion of outliers e s 5%
10%
20%
25%
30%
40%
50% 2 3 5 6 7 11 17 4 9 19 35 13 34 72 12 26 57
146 16 24 37 97
293 8 20 33 54
163
588 44 78
272
1177
Note: Assumes that inliers allow to identify other inliers

21. Acceptable consensus set?
Typically, terminate when inlier ratio reaches expected ratio of inliers

22. Adaptively determining the number of samples
e is often unknown a priori, so pick worst case, i.e. 0, and adapt if more inliers are found, e.g. 80% would yield e=0.2
N=∞, sample_count =0
While N >sample_count repeat
Choose a sample and count the number of inliers
Set e=1-(number of inliers)/(total number of points)
Recompute N from e
Increment the sample_count by 1
Terminate

23. Other robust algorithms
RANSAC maximizes number of inliers
LMedS minimizes median error
Not recommended: case deletion, iterative least-squares, etc.
inlier percentile
100%
50%
residual
(pixels)
1.25

24. Non-linear refinment

25. Geometric distance
Gold standard
Symmetric epipolar distance

26. Gold standard Maximum Likelihood Estimation
(= least-squares for Gaussian noise)
Initialize: normalized 8-point, (P,P‘) from F, reconstruct Xi
Parameterize:
(overparametrized)
Minimize cost using Levenberg-Marquardt
(preferably sparse LM, e.g. see H&Z)

27. Gold standard Alternative, minimal parametrization (with a=1)
(note (x,y,1) and (x‘,y‘,1) are epipoles)
problems:
a=0
 pick largest of a,b,c,d to fix to 1
epipole at infinity
 pick largest of x,y,w and of x’,y’,w’
4x3x3=36 parametrizations!
reparametrize at every iteration, to be sure

28. Zhang&Loop’s approach CVIU’01

29. First-order geometric error (Sampson error)
(one eq./point
JJT scalar)
(problem if some x is located at epipole)
advantage: no subsidiary variables required

30. Symmetric epipolar error

31. Some experiments:

32. Some experiments:

33. Some experiments:

34. Some experiments:
Residual error:
(for all points!)

35. Recommendations: Do not use unnormalized algorithms
Quick and easy to implement: 8-point normalized
Better: enforce rank-2 constraint during minimization
Best: Maximum Likelihood Estimation (minimal parameterization, sparse implementation)

36. The envelope of epipolar lines
What happens to an epipolar line if there is noise?
Monte Carlo
n=10
n=15
n=25
n=50

37. Automatic computation of F
Step 1. Extract features
Step 2. Compute a set of potential matches
Step 3. do
Step 3.1 select minimal sample (i.e. 7 matches)
Step 3.2 compute solution(s) for F
Step 3.3 determine inliers
until (#inliers,#samples)<95%
(generate
hypothesis)
(verify hypothesis)
Step 4. Compute F based on all inliers
Step 5. Look for additional matches
Step 6. Refine F based on all correct matches
#inliers
90%
80%
70%
60%
50%
#samples 5 13 35
106
382

38. Abort verification early
O O O O O I O O I O O O O
I I I O I I I I O O I O I I I I I I O I I I I I I O I O O I I I I…
O O O O O O O
O O O O I O O O O
Assume percentage of inliers p, what is the probability P to pick n or more wrong samples out of m?
stop if P<0.05 (sample most probably contains outlier)
(P=cum. binomial distr. funct. (m-n,m,p ))
To avoid problems this requires to also verify at random!
(but we already have a random sampler anyway)
(inspired from Chum and Matas BMVC2002)

39. restrict search range to neighborhood of epipolar line
Finding more matches
restrict search range to neighborhood of epipolar line
(e.g. 1.5 pixels)
relax disparity restriction (along epipolar line)

40. Solution 1: Model selection
Degenerate cases:
Degenerate cases
Planar scene
Pure rotation
No unique solution
Remaining DOF filled by noise
Use simpler model (e.g. homography)
Solution 1: Model selection
(Torr et al., ICCV´98, Kanatani, Akaike)
Compare H and F according to expected residual error (compensate for model complexity)
Solution 2: RANSAC
Compare H and F according to inlier count
(see next slide)

41. RANSAC for quasi-degenerate cases
Full model (8pts, 1D solution)
Planar model (6pts, 3D solution)
Accept if large number of remaining inliers
Plane+parallax model (plane+2pts)
(accept inliers to solution F)
(accept inliers to solution F1,F2&F3)
closest rank-6 of Anx9 for all plane inliers
Sample for out of plane points among outliers

42. Absence of sufficient features (no texture)
More problems:
Absence of sufficient features (no texture)
Repeated structure ambiguity
Robust matcher also finds
support for wrong hypothesis
solution: detect repetition
(Schaffalitzky and Zisserman,
BMVC‘98)

43. RANSAC for ambiguous matching
Include multiple candidate matches in set of potential matches
Select according to matching probability (~ matching score)
Helps for repeated structures or scenes with similar features as it avoids an early commitment
(Tordoff and Murray ECCV02)

44. geometric relations between two views is fully
two-view geometry
geometric relations between two views is fully
described by recovered 3x3 matrix F

45. Triangulation L2 x1 C1 X L1 Triangulation calibration x2
correspondences

46. Triangulation Backprojection Triangulationx1 L1 x2 L2 X C2
Triangulation
Iterative least-squares
Maximum Likelihood Triangulation

47. Optimal 3D point in epipolar plane
Given an epipolar plane, find best 3D point for (m1,m2) m1 m2 l1 l2 l1 m1 m2 l2
m1´
m2´
Select closest points (m1´,m2´) on epipolar lines
Obtain 3D point through exact triangulation
Guarantees minimal reprojection error (given this epipolar plane)

48. Non-iterative optimal solution
Reconstruct matches in projective frame by minimizing the reprojection error
Non-iterative method
Determine the epipolar plane for reconstruction
Reconstruct optimal point from selected epipolar plane
Note: only works for two views
3DOF
(Hartley and Sturm, CVIU´97)
(polynomial of degree 6) m1 m2
l1(a)
l2(a)
1DOF