1. Multiple-view Reconstruction from Points and Lines

2. Single and Two view summary
Uncalibrated case
Recover Fundamental Matrix F
Projective reconstruction
Recover planar homography
(no decomposition possible)
Rotational homography (mosaics)
Calibration with planar rig
Calibration with 3D rig
Single view
Homography between 3D plane and image plane (rectification)
Partial calibration using vanishing points
Partial calibration and pose recovery using from single view/world homography
Calibrated case
Recover Essential Matrix decompose to R,T
3D reconstruction
Planar case
Recover homography H
decompose to R,T,n,d
Here is a picture illustrating the fundamental problem that we are interested in multiple view geometry. The scenario is that we are given multiple pictures or images of some 3D object and after establishing correspondence of certain geometric features such as a point, line or curve, we then try to use such information to recover the relative locations of the camera where these images are taken, as well as recover the 3D structure of the object.
CS 223b J. Kosecka

3. Problem formulation
Input: Corresponding images (of “features”) in multiple images.
Output: Camera motion, camera calibration, object structure.
Here is a picture illustrating the fundamental problem that we are interested in multiple view geometry. The scenario is that we are given multiple pictures or images of some 3D object and after establishing correspondence of certain geometric features such as a point, line or curve, we then try to use such information to recover the relative locations of the camera where these images are taken, as well as recover the 3D structure of the object.
CS 223b J. Kosecka

4. Affine/Orthographic projection model
Full perspective projection model
Affine camera projection model (in-homogenous coordinates)
Good approximation if the distance from the scene >> scene depth variation
Rigid body motion under affine projection model
CS 223b J. Kosecka

5. Affine Multiview Factorization
Problem – given n correspondences in m views
determine and 3D points
Obtained by minimization of the following objective function
Assume that the centroid of the structure is the origin of
the coordinate frame and denote the centroids in the following way
Minimize with respect to
CS 223b J. Kosecka

6. Affine Multiview Factorization
The choice of the frame is arbitrary – further assume that
The objective function then becomes
Writing all the constrains in the matrix form
Drop ~
for clarity
CS 223b J. Kosecka

7. Affine Multiview Factorization
Measurement matrix
Motion matrix
Structure matrix
Matrix W must have rank 3 (product of two rank 3 matrices)
In noise case we seek best rank 3 approximation of W matrix
Given the actual measurement matrix compute SVD
Best rank 3 approximation is
CS 223b J. Kosecka

8. Affine Multiview Factorization
Decomposition of is not unique
Affine ambiguity Q
The ambiguity can be resolved using rotation matrix constraints
where
[Tomasi, Kanade’IJCV 1992] Factorization approach
CS 223b J. Kosecka

9. Projective Multiview Factorization
Measurement matrix
Motion matrix
Structure matrix
Similar strategy – expect now the scales are unknown
We don’t have the matrix W
Matrix W is a product of two rank 4 matrices – i.e. is rank 4
How to apply the factorization idea to projective setting ?
- need to compute scales (possible from two view reconstruction)
- or initialize the scales and iterate
CS 223b J. Kosecka

10. Projective Factorization Algorithm
Given a set of image points in n views
Compute the projective depths using two view methods or set
Form the measurement matrix W and find the nearest rank 4 approximation using SVD
Decompose into camera matrices and structure
Optionally reproject the and iterate
(i.e. given new motions, we can compute new scales)
CS 223b J. Kosecka

11. Multi-view methods Advantages of multiview methods
- more frames – wider baseline – better conditioned algorithms
- additional complexity of matching (establishing
correspondences) across multiple views
How are multi-view and two view methods related ?
Can we just run the two view algorithm for each pair of views
and get better results ?
What if we are trying to do reconstruction using lines ?
Are there other constraints between multiple views then pairwise epipolar constraints ?
Next – brief tour of multilinear constraints
CS 223b J. Kosecka

12. Traditional multifocal constraints
For images of the same 3-D point :
Let us first see how multiple view geometry is traditionally studied. In particular, what is the nature of multilinear constraints among multiple images. For given m images of a 3-D point p, we can rewrite the equations in a single matrix form as follows. Speculate this equation for a while, we may observe that both lambda and X are associated to the 3-D location of the point p, relative to different coordinate frames. To eliminate these 3-D structural parameters, we can form the following matrix N. From the above equation, it is straightforward to see that this matrix is rank deficient. Furthermore, if this matrix has an exact rank m+3, we can recover \lambda and X as its only kernel. Hence we can reduce our study to examine this matrix N. Notice that given a camera configuration specified by the matrices Pi’s, not any set of vectors x1, …, xm would make this matrix rank deficient. That means that images from the same 3-D point should satisfy certain relationship. One way to describe such relationship is by writing down the determinants of all m+4 by m+4 submatrices of N and these determinants must be zero. Turns out these equations can be reduced to equations involving images from 2, 3, or 4 views at a time, which leads to the traditional multilinear constraints for 2, 3, 4 views…
(leading to the conventional approach)
Multilinear constraints
among 2, 3, 4-wise views
CS 223b J. Kosecka

13. Rank conditions for point feature
WLOG, choose camera frame 1 as the reference
Multiple-View Matrix
Lemma [Rank Condition for Point Features]
My opinion on this approach now is that it is way too early to write down the rank deficiency condition on N in terms of the determinants of its submatrices. After a little linear algebraic manipulation on N, we may obtain a matrix H of this form whose rank is closely related to that of N. More specifically, rank(N) is equal to m+2 plus rank(M). This gives us two possible cases: one is more generic and the other degenerate. In any case, we know that if x1, …, xm are images of some 3-D point p, the matrix H must be rank deficient, or in other words, the two columns of H are linearly dependent.
Let
then and are linearly dependent.
CS 223b J. Kosecka

14. Rank conditions vs. multifocal constraints
CS 223b J. Kosecka

15. Rank conditions vs. multi-focal constraints
CS 223b J. Kosecka

16. Rank conditions vs. multifocal constraints
These constraints are only necessary but NOT sufficient!
However, there is NO further relationship among quadruple wise
views. Quadrilinear constraints hence are redundant!
Now let us examine the rank deficient M matrix and see what we get from it. First observe that the two columns of M are linearly dependent implies that the pair of vectors from each three rows must be linearly dependent. This in fact gives rise to the well-known epipolar constraints. One thing we notice from this derivation is that, epipolar constraints are only necessary but not sufficient for matrix H to be rank deficient.
CS 223b J. Kosecka

17. Point Features – Uniqueness of the pre-image
bilinear constraints – coplanarity constraints
Extend to three views
collinear optical centers
CS 223b J. Kosecka
coplanar optical centers

18. Point Features – Uniqueness of the pre-image trilinear constraints
Given m vectors with respect to m camera frames,
They correspond to a unique point in the 3D space if the rank of the
Matrix Mp is 1. If rank is 0, the point is determined up to a line
on which all optical centers must lie .
CS 223b J. Kosecka

19. Image of a line feature Homogeneous representation of a 3-D line
Homogeneous representation of its 2-D co-image
Projection of a 3-D line to an image plane
First let us talk about line features. To describe a line in 3-D, we need to specify a base point on the line and a vector indicating the direction of the line. On the image plane we can use a three dimensional vector l to describe the image of a line L. More specifically, if x is the image of a point on this line, its inner product with l is 0.
CS 223b J. Kosecka

20. Multiple-view matrix: line vs. point
Point Features
Line Features
We may then derive a M matrix for a line feature, we call it M_l. Comparing with the M matrix for a point feature, now denoted as M_p, they both have rank 1. M_l is however a matrix of four columns. The linear dependency between any two rows of M_l gives rise to the trilinear constraints in terms of image lines, which is well-known in the literature. Geometric interpretation for the M_l matrix is no longer a sphere, but a circle. This circle gives a family of parallel lines which may give the same M_l matrix. The radius of the circle is the distance of these lines from the center of the reference camera frame and the normal of this circle is the direction of these lines.
CS 223b J. Kosecka

21. Rank conditions: line vs. point
Continue with constraints that the rank of the matrix M_l imposes on multiple images of a line feature. If rank(M) is 1, it means that all the planes through every camera center and image line intersect at a unique line in 3-D. If the rank is 0, it corresponds to the only degenerate case that all the planes are the same hence the 3-D line is determined only up to a plane on which all the camera centers must lie.
CS 223b J. Kosecka

22. Multiple-view structure and motion recovery
Given images of points:
The third problem, also the most important one, is how to recover camera configuration from given m images of a set of n points. Using the rank deficiency condition on M, the problem becomes looking for T2, R2, …, Tm, Rm such that the two columns of M are linearly dependent. That is, there exists coefficients alpha^j’s such that the equations hold. I won’t get into the detail how to determine those coefficients alpha^j’s even without knowing all the camera motions. For now, I only tell you their values depend on a choice of coordinate frames. These frames could be either Euclidean, affine or projective. Assuming these coefficients are known, then finding the camera configuration simply becomes a problem of solving a linear equation. This equation have a unique solution if we have in general more than 6 points.
CS 223b J. Kosecka

23. SVD based 4-step algorithm for SFM
Here is the outline of this algorithm. The only thing I’d like to point out is that the algorithm is initialized by a two view algorithm due to the fact that non-trivial constraints for point features start with two views.
CS 223b J. Kosecka

24. Utilizing all incidence relations
Three edges intersect at each vertex. . . .
CS 223b J. Kosecka

25. Example: simulations
CS 223b J. Kosecka

26. Example: simulations
CS 223b J. Kosecka

27. Example: experiments Errors in all right angles < 1o
CS 223b J. Kosecka

28. Summary Incidence relations <=> rank conditions
Rank conditions => multiple-view factorization
Rank conditions implies all multi-focal constraints
Rank conditions for points, lines, planes, and
(symmetric) structures.
CS 223b J. Kosecka

29. Global multiple-view analysis: examples
The rank condition actually allows to you to do multiple view analysis globally. For example if you choose a multiple view matrix as following. Its rank can only be 1 or 2. Corresponding to each value, there is a generic picture of the configuration of the 3D features involved and the relative camera configuration.
CS 223b J. Kosecka

30. A family of intersecting lines
each can randomly take the image of any of the lines:
Nonlinear constraints
among up to four views
What is the essential meaning of this rank 2 case? Here is an example explaining it. Consider a family of lines in 3D intersecting at one point p. You then randomly choose the image of any of the lines in the family in each view and form a multiple view matrix. Then this matrix in general has rank 2. We know before that if all the images chosen happen to correspond to the same 3D line, the rank of M is 1. Here, you don’t need to have exact correspondence among those lines, yet you still get some non-trivial constraints among their images… . . .
CS 223b J. Kosecka

31. Universal rank condition
Theorem [The Universal Rank Condition] for images of a point on a line:
Multi-nonlinear constraints
among 3, 4-wise images.
Multi-linear constraints
among 2, 3-wise images.
Here comes my favorite slide: Consider multiple images of a point on a line. In order to express all the incidence constraints associated to these features and all their images, you can formally define a matrix M as following, where the components D_i and D_i^perp are up to you to choose and plug in. D_I’s are really the images of either the point or the line, D_I^perp’s are the coimages. Choices for D_I and D_j are independent if I is not equal to j. The interesting thing is no matter what you choose, the rank of the resulting matrix must be one of the two cases.
The second case gives you all the multi-linear constraints, of course only up to 2 or 3 views; if you really want to know what is among four views, there are some nonlinear constraints. We will see a few examples and give you some intuitive ideas…
CS 223b J. Kosecka

32. Instances with mixed features
Examples:
Case 1: a line reference
Case 2: a point reference
Here we take two instantionations of the multiple view matrix.
All previously known constraints are the theorem’s instances.
Degenerate configurations if and only if a drop of rank.
CS 223b J. Kosecka

33. Generalization – restriction to a plane
Homogeneous representation of a 3-D plane
We have talked about points and lines, what about plane? Suppose now the point or the line feature belongs to some plane in 3D, say pi. We know in general such a plane can be expressed by the following equation. We may lump all the coefficients into a vector pi. Pi^1 is just the first three components and pi^2 is the d. Then the rank condition we had before must be modified somehow since now we have this extra restriction. It turns out all you need to do is to append an extra row to the multiple view matrix and then all the rank conditions remain exactly the same.
Corollary [Coplanar Features]
Rank conditions on the new extended
remain exactly the same!
CS 223b J. Kosecka

34. Generalization – restriction to a plane
Given that a point and line features lie on a plane in 3-D space:
GENERALIZATION – Multiple View Matrix: Coplanar Features
In addition to previous constraints, it simultaneously gives homography:
For example, the multiple view matrices for m images of a point or a line on such a plane are given by the following two matrices. According to the theorem, they both should have rank 1. What this extra row gives you is exactly the so called homography in computer vision literature. In addition to the homography, the multiple view matrix keeps all the constraints.
CS 223b J. Kosecka