2. Multiple-View Geometry for Image-Based Modeling (Course 42)
Lecturers:
Yi Ma (UIUC)
Stefano Soatto (UCLA)
Jana Kosecka (GMU)
Rene Vidal (UC Berkeley)
Yizhou Yu (UIUC)

3. COURSE LECTURE OUTLINE
A. Introduction (Ma)
B. Preliminaries: geometry & image formation (Ma)
C. Image primitives & correspondence (Soatto)
D. Two calibrated views (Kosecka)
E. Uncalibrated geometry and stratification (Soatto)
F. Multiple-view geometry & algebra (Vidal, Ma)
G. Reconstruction from scene knowledge (Ma)
H. Step-by-step building of 3D model (Kosecka, Soatto)
I. Image-based texture mapping and rendering (Yu)

4. Introduction (Lecture A)
Multiple-View Geometry for Image-Based Modeling
Introduction (Lecture A)
Yi Ma
Perception & Decision Laboratory
Decision & Control Group, CSL
Image Formation & Processing Group, Beckman
Electrical & Computer Engineering Dept., UIUC

5. IMAGES AND GEOMETRY – A Little History of Perspective Imaging
Pinhole (perspective) imaging, in most ancient civilizations.
Euclid, perspective projection, 4th century B.C., Alexandria
Pompeii frescos, 1st century A.D.
Image courtesy of C. Taylor

6. IMAGES AND GEOMETRY – A Little History of Perspective Imaging
Fillippo Brunelleschi, first Renaissance artist painted with
correct perspective,1413
“Della Pictura”, Leone Battista Alberti, 1435
Leonardo Da Vinci, stereopsis, shading, color, 1500s
“The scholar of Athens”, Raphael, 1518
Image courtesy of C. Taylor

7. IMAGES AND GEOMETRY – The Fundamental Problem
Input: Corresponding “features” in multiple images.
Output: Camera calibration, pose, scene structure, surface photometry.
Jana’s apartment

8. IMAGES AND GEOMETRY – History of “Modern” Geometric Vision
Chasles, formulated the two-view seven-point problem,1855
Hesse, solved the above problem, 1863
Kruppa, solved the two-view five-point problem, 1913
Longuet-Higgins, the two-view eight-point algorithm, 1981
Liu and Huang, the three-view trilinear constraints, 1986
Huang and Faugeras, SVD based eight-point algorithm, 1989
Tomasi and Kanade, (orthographic) factorization method, 1992
Ma, Huang, Kosecka, Vidal, multiple-view rank condition, 2000

9. APPLICATIONS – 3-D Modeling and Rendering

10. APPLICATIONS – 3-D Modeling and Rendering
Image courtesy of Paul Debevec

11. APPLICATIONS – Image Morphing, Mosaicing, Alignment
Images of CSL, UIUC

12. First-down line and virtual advertising
APPLICATIONS – Real-Time Sports Coverage
First-down line and virtual advertising
Image courtesy of Princeton Video Image, Inc.

13. APPLICATIONS – Real-Time Virtual Object Insertion
UCLA Vision Lab

14. APPLICATIONS – Autonomous Highway Vehicles
Image courtesy of E.D. Dickmanns

15. APPLICATIONS – Unmanned Aerial Vehicles (UAVs)
Rate: 10Hz
Accuracy: 5cm, 4o
Berkeley Aerial Robot (BEAR) Project

16. Preliminaries: Imaging Geometry & Image Formation (Lecture B)
Multiple-View Geometry for Image-Based Modeling
Preliminaries: Imaging Geometry & Image Formation (Lecture B)
Yi Ma
Perception & Decision Laboratory
Decision & Control Group, CSL
Image Formation & Processing Group, Beckman
Electrical & Computer Engineering Dept., UIUC

17. Preliminaries: Imaging Geometry and Image Formation
INTRODUCTION
3D EUCLIDEAN SPACE & RIGID-BODY MOTION
Coordinates and coordinate frames
Rigid-body motion and homogeneous coordinates
GEOMETRIC MODELS OF IMAGE FORMATION
Lens & Lambertian surfaces
Pinhole camera model
CAMERA INTRINSIC PARAMETERS & RADIAL DISTORTION
From space to pixel coordinates
Notation: image, preimage, and coimage
Radial distortion and correction
SUMMARY OF NOTATION

18. Coordinates of a point in space:
3D EUCLIDEAN SPACE - Cartesian Coordinate Frame
Coordinates of a point in space:
Standard base vectors:
We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.

19. A “free” vector is defined by a pair of points :
3D EUCLIDEAN SPACE - Vectors
A “free” vector is defined by a pair
of points :
Coordinates of the vector :
We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.

20. 3D EUCLIDEAN SPACE – Inner Product and Cross Product
Inner product between two vectors:
Cross product between two vectors:
We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.

21. Coordinates are related by:
RIGID-BODY MOTION – Rotation
Rotation matrix:
We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.
Coordinates are related by:

22. Coordinates are related by:
RIGID-BODY MOTION – Rotation and Translation
We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.
Coordinates are related by:

23. 3D coordinates are related by:
RIGID-BODY MOTION – Homogeneous Coordinates
3D coordinates are related by:
Homogeneous coordinates are related by:
Homogeneous coordinates of a vector:
We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.

24. BRDF Lambertian thin lens small FOV
IMAGE FORMATION – Lens, Light, and Surfaces
image irradiance
surface radiance
We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.
BRDF
Lambertian
thin lens
small FOV

25. Pinhole Frontal pinhole IMAGE FORMATION – Pinhole Camera Model
We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.

26. Homogeneous coordinates
IMAGE FORMATION – Pinhole Camera Model
2D coordinates
Homogeneous coordinates
We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.

27. CAMERA PARAMETERS – Pixel Coordinates
calibrated
coordinates
Linear transformation
pixel
coordinates
We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.

28. (intrinsic parameters)
CAMERA PARAMETERS – Calibration Matrix and Camera Model
Ideal pinhole
Pixel coordinates
Calibration matrix
(intrinsic parameters)
Projection matrix
We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.
Camera model

29. Nonlinear transformation along the radial direction
CAMERA PARAMETERS – Radial Distortion
Nonlinear transformation along the radial direction
We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.
Distortion correction: make lines straight

30. IMAGE FORMATION – Image of a Point
Homogeneous coordinates of a 3-D point
Homogeneous coordinates of its 2-D image
Projection of a 3-D point to an image plane
Now let me quickly go through the basic mathematical model for a camera system. Here is the notation. We will use a four dimensional vector X for the homogeneous coordinates of a 3-D point p, its image on a pre-specified plane will be described also in homogeneous coordinate as a three dimensional vector x. If everything is normalized, then W and z can be chosen to be 1. We use a 3x4 matrix Pi to denote the transformation from the world frame to the camera frame. R may stand for rotation, T for translation. Then the image x and the world coordinate X of a point is related through the equation, where lambda is a scale associated to the depth of the 3D point relative to the camera center o. But in general the matrix Pi can be any 3x4 matrix, because the camera may add some unknown linear transformation on the image plane. Usually it is denoted by a 3x 3 matrix A(t).

31. Image of a 3-D point Coimage of the point Preimage of the point
NOTATION – Image, Coimage, Preimage of a Point
Image of a 3-D point
Coimage of the point
Preimage of the point
Now let me quickly go through the basic mathematical model for a camera system. Here is the notation. We will use a four dimensional vector X for the homogeneous coordinates of a 3-D point p, its image on a pre-specified plane will be described also in homogeneous coordinate as a three dimensional vector x. If everything is normalized, then W and z can be chosen to be 1. We use a 3x4 matrix Pi to denote the transformation from the world frame to the camera frame. R may stand for rotation, T for translation. Then the image x and the world coordinate X of a point is related through the equation, where lambda is a scale associated to the depth of the 3D point relative to the camera center o. But in general the matrix Pi can be any 3x4 matrix, because the camera may add some unknown linear transformation on the image plane. Usually it is denoted by a 3x 3 matrix A(t).

32. Coimage of a 3-D line Preimage of the line Image of the line
NOTATION – Image, Coimage, Preimage of a Line
Coimage of a 3-D line
Preimage of the line
Image of the line
Now let me quickly go through the basic mathematical model for a camera system. Here is the notation. We will use a four dimensional vector X for the homogeneous coordinates of a 3-D point p, its image on a pre-specified plane will be described also in homogeneous coordinate as a three dimensional vector x. If everything is normalized, then W and z can be chosen to be 1. We use a 3x4 matrix Pi to denote the transformation from the world frame to the camera frame. R may stand for rotation, T for translation. Then the image x and the world coordinate X of a point is related through the equation, where lambda is a scale associated to the depth of the 3D point relative to the camera center o. But in general the matrix Pi can be any 3x4 matrix, because the camera may add some unknown linear transformation on the image plane. Usually it is denoted by a 3x 3 matrix A(t).

33. IMAGE FORMATION – Coimage of a Line
Homogeneous representation of a 3-D line
Homogeneous representation of its 2-D coimage
Projection of a 3-D line to an image plane
First let us talk about line features. To describe a line in 3-D, we need to specify a base point on the line and a vector indicating the direction of the line. On the image plane we can use a three dimensional vector l to describe the image of a line L. More specifically, if x is the image of a point on this line, its inner product with l is 0.

34. IMAGE FORMATION – Multiple Images
“Preimages” are all incident at the corresponding features.
Now consider multiple images of a simplest object, say a cube. All the constraints are incidence relations, are all of the same nature. Is there any way that we can express all the constraints in a unified way? Yes, there is. . . .

35. An Invitation to 3-D Vision: Ma, Soatto, Kosecka, Sastry,
LIST OF REFERENCES
Chapters 2 & 3
An Invitation to 3-D Vision:
From Images to Geometric Models,
Ma, Soatto, Kosecka, Sastry,
Springer-Verlag, 2003.
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.