1. Multiple-view Reconstruction from Points and Lines
Yi Ma
ECE Department, CSL, Beckman
University of Illinois at Urbana-Champaign

2. Problem formulation
Input: Corresponding images (of “features”) in multiple images.
Output: Camera motion, camera calibration, object structure.
Here is a picture illustrating the fundamental problem that we are interested in multiple view geometry. The scenario is that we are given multiple pictures or images of some 3D object and after establishing correspondence of certain geometric features such as a point, line or curve, we then try to use such information to recover the relative locations of the camera where these images are taken, as well as recover the 3D structure of the object.
Yi Ma
Tutorial ICRA 2003

3. Projection model – point features
Homogeneous coordinates of a 3-D point
Homogeneous coordinates of its 2-D image
Projection of a 3-D point to an image plane
Now let me quickly go through the basic mathematical model for a camera system. Here is the notation. We will use a four dimensional vector X for the homogeneous coordinates of a 3-D point p, its image on a pre-specified plane will be described also in homogeneous coordinate as a three dimensional vector x. If everything is normalized, then W and z can be chosen to be 1. We use a 3x4 matrix Pi to denote the transformation from the world frame to the camera frame. R may stand for rotation, T for translation. Then the image x and the world coordinate X of a point is related through the equation, where lambda is a scale associated to the depth of the 3D point relative to the camera center o. But in general the matrix Pi can be any 3x4 matrix, because the camera may add some unknown linear transformation on the image plane. Usually it is denoted by a 3x 3 matrix A(t).
Yi Ma
Tutorial ICRA 2003

4. Image of a line feature Homogeneous representation of a 3-D line
Homogeneous representation of its 2-D co-image
Projection of a 3-D line to an image plane
First let us talk about line features. To describe a line in 3-D, we need to specify a base point on the line and a vector indicating the direction of the line. On the image plane we can use a three dimensional vector l to describe the image of a line L. More specifically, if x is the image of a point on this line, its inner product with l is 0.
Yi Ma
Tutorial ICRA 2003

5. Incidence relations among features
Multiview constraints are nothing but incidence relations at play!
Now consider multiple images of a simplest object, say a cube. All the constraints are incidence relations, are all of the same nature. Is there any way that we can express all the constraints in a unified way? Yes, there is. . . .
Yi Ma
Tutorial ICRA 2003

6. Traditional multifocal constraints
For images of the same 3-D point :
Let us first see how multiple view geometry is traditionally studied. In particular, what is the nature of multilinear constraints among multiple images. For given m images of a 3-D point p, we can rewrite the equations in a single matrix form as follows. Speculate this equation for a while, we may observe that both lambda and X are associated to the 3-D location of the point p, relative to different coordinate frames. To eliminate these 3-D structural parameters, we can form the following matrix N. From the above equation, it is straightforward to see that this matrix is rank deficient. Furthermore, if this matrix has an exact rank m+3, we can recover \lambda and X as its only kernel. Hence we can reduce our study to examine this matrix N. Notice that given a camera configuration specified by the matrices Pi’s, not any set of vectors x1, …, xm would make this matrix rank deficient. That means that images from the same 3-D point should satisfy certain relationship. One way to describe such relationship is by writing down the determinants of all m+4 by m+4 submatrices of N and these determinants must be zero. Turns out these equations can be reduced to equations involving images from 2, 3, or 4 views at a time, which leads to the traditional multilinear constraints for 2, 3, 4 views…
(leading to the conventional approach)
Multilinear constraints
among 2, 3, 4-wise views
Yi Ma
Tutorial ICRA 2003

7. Rank conditions for point feature
WLOG, choose camera frame 1 as the reference
Multiple-View Matrix
Lemma [Rank Condition for Point Features]
My opinion on this approach now is that it is way too early to write down the rank deficiency condition on N in terms of the determinants of its submatrices. After a little linear algebraic manipulation on N, we may obtain a matrix H of this form whose rank is closely related to that of N. More specifically, rank(N) is equal to m+2 plus rank(M). This gives us two possible cases: one is more generic and the other degenerate. In any case, we know that if x1, …, xm are images of some 3-D point p, the matrix H must be rank deficient, or in other words, the two columns of H are linearly dependent.
Let
then and are linearly dependent.
Yi Ma
Tutorial ICRA 2003

8. Rank conditions vs. multifocal constraints
These constraints are only necessary but NOT sufficient!
However, there is NO further relationship among quadruple wise
views. Quadrilinear constraints hence are redundant!
Now let us examine the rank deficient M matrix and see what we get from it. First observe that the two columns of M are linearly dependent implies that the pair of vectors from each three rows must be linearly dependent. This in fact gives rise to the well-known epipolar constraints. One thing we notice from this derivation is that, epipolar constraints are only necessary but not sufficient for matrix H to be rank deficient.
Yi Ma
Tutorial ICRA 2003

9. Multiple-view matrix: line vs. point
Point Features
Line Features
We may then derive a M matrix for a line feature, we call it M_l. Comparing with the M matrix for a point feature, now denoted as M_p, they both have rank 1. M_l is however a matrix of four columns. The linear dependency between any two rows of M_l gives rise to the trilinear constraints in terms of image lines, which is well-known in the literature. Geometric interpretation for the M_l matrix is no longer a sphere, but a circle. This circle gives a family of parallel lines which may give the same M_l matrix. The radius of the circle is the distance of these lines from the center of the reference camera frame and the normal of this circle is the direction of these lines.
Yi Ma
Tutorial ICRA 2003

10. Rank conditions: line vs. point
Continue with constraints that the rank of the matrix M_l imposes on multiple images of a line feature. If rank(M) is 1, it means that all the planes through every camera center and image line intersect at a unique line in 3-D. If the rank is 0, it corresponds to the only degenerate case that all the planes are the same hence the 3-D line is determined only up to a plane on which all the camera centers must lie.
Yi Ma
Tutorial ICRA 2003

11. Global multiple-view analysis: examples
The rank condition actually allows to you to do multiple view analysis globally. For example if you choose a multiple view matrix as following. Its rank can only be 1 or 2. Corresponding to each value, there is a generic picture of the configuration of the 3D features involved and the relative camera configuration.
Yi Ma
Tutorial ICRA 2003

12. A family of intersecting lines
each can randomly take the image of any of the lines:
What is the essential meaning of this rank 2 case? Here is an example explaining it. Consider a family of lines in 3D intersecting at one point p. You then randomly choose the image of any of the lines in the family in each view and form a multiple view matrix. Then this matrix in general has rank 2. We know before that if all the images chosen happen to correspond to the same 3D line, the rank of M is 1. Here, you don’t need to have exact correspondence among those lines, yet you still get some non-trivial constraints among their images…
Nonlinear constraints
among up to four views . . .
Yi Ma
Tutorial ICRA 2003

13. Universal rank condition
Theorem [The Universal Rank Condition] for images of a point on a line:
Multi-nonlinear constraints
among 3, 4-wise images.
Multi-linear constraints
among 2, 3-wise images.
Here comes my favorite slide: Consider multiple images of a point on a line. In order to express all the incidence constraints associated to these features and all their images, you can formally define a matrix M as following, where the components D_i and D_i^perp are up to you to choose and plug in. D_I’s are really the images of either the point or the line, D_I^perp’s are the coimages. Choices for D_I and D_j are independent if I is not equal to j. The interesting thing is no matter what you choose, the rank of the resulting matrix must be one of the two cases.
The second case gives you all the multi-linear constraints, of course only up to 2 or 3 views; if you really want to know what is among four views, there are some nonlinear constraints. We will see a few examples and give you some intuitive ideas…
Yi Ma
Tutorial ICRA 2003

14. Instances with mixed features
Examples:
Case 1: a line reference
Case 2: a point reference
Here we take two instantionations of the multiple view matrix.
All previously known constraints are the theorem’s instances.
Degenerate configurations if and only if a drop of rank.
Yi Ma
Tutorial ICRA 2003

15. Generalization – restriction to a plane
Homogeneous representation of a 3-D plane
We have talked about points and lines, what about plane? Suppose now the point or the line feature belongs to some plane in 3D, say pi. We know in general such a plane can be expressed by the following equation. We may lump all the coefficients into a vector pi. Pi^1 is just the first three components and pi^2 is the d. Then the rank condition we had before must be modified somehow since now we have this extra restriction. It turns out all you need to do is to append an extra row to the multiple view matrix and then all the rank conditions remain exactly the same.
Corollary [Coplanar Features]
Rank conditions on the new extended
remain exactly the same!
Yi Ma
Tutorial ICRA 2003

16. Generalization – restriction to a plane
Given that a point and line features lie on a plane in 3-D space:
GENERALIZATION – Multiple View Matrix: Coplanar Features
In addition to previous constraints, it simultaneously gives homography:
For example, the multiple view matrices for m images of a point or a line on such a plane are given by the following two matrices. According to the theorem, they both should have rank 1. What this extra row gives you is exactly the so called homography in computer vision literature. In addition to the homography, the multiple view matrix keeps all the constraints.
Yi Ma
Tutorial ICRA 2003

17. Multiple-view structure and motion recovery
Given images of points:
The third problem, also the most important one, is how to recover camera configuration from given m images of a set of n points. Using the rank deficiency condition on M, the problem becomes looking for T2, R2, …, Tm, Rm such that the two columns of M are linearly dependent. That is, there exists coefficients alpha^j’s such that the equations hold. I won’t get into the detail how to determine those coefficients alpha^j’s even without knowing all the camera motions. For now, I only tell you their values depend on a choice of coordinate frames. These frames could be either Euclidean, affine or projective. Assuming these coefficients are known, then finding the camera configuration simply becomes a problem of solving a linear equation. This equation have a unique solution if we have in general more than 6 points.
Yi Ma
Tutorial ICRA 2003

18. SVD based 4-step algorithm for SFM
Here is the outline of this algorithm. The only thing I’d like to point out is that the algorithm is initialized by a two view algorithm due to the fact that non-trivial constraints for point features start with two views.
Yi Ma
Tutorial ICRA 2003

19. Utilizing all incidence relations
Three edges intersect at each vertex. . . .
Yi Ma
Tutorial ICRA 2003

20. Example: simulations
Yi Ma
Tutorial ICRA 2003

21. Example: simulations
Yi Ma
Tutorial ICRA 2003

22. Example: experiments Errors in all right angles < 1o Yi Ma
Tutorial ICRA 2003

23. Summary Incidence relations <=> rank conditions
Rank conditions => multiple-view factorization
Rank conditions implies all multi-focal constraints
Rank conditions for points, lines, planes, and
(symmetric) structures.
Yi Ma
Tutorial ICRA 2003