Multiple-View Geometry for Image-Based Modeling (Course 42) Lecturers: Yi Ma (UIUC) Stefano Soatto (UCLA) Jana Kosecka (GMU) Rene Vidal (UC Berkeley) Yizhou Yu (UIUC)

COURSE LECTURE OUTLINE A. Introduction (Ma) B. Preliminaries: geometry & image formation (Ma) C. Image primitives & correspondence (Soatto) D. Two calibrated views (Kosecka) E. Uncalibrated geometry and stratification (Soatto) F. Multiple-view geometry & algebra (Vidal, Ma) G. Reconstruction from scene knowledge (Ma) H. Step-by-step building of 3D model (Kosecka, Soatto) I. Image-based texture mapping and rendering (Yu)

Introduction (Lecture A) Multiple-View Geometry for Image-Based Modeling Introduction (Lecture A) Yi Ma Perception & Decision Laboratory Decision & Control Group, CSL Image Formation & Processing Group, Beckman Electrical & Computer Engineering Dept., UIUC http://decision.csl.uiuc.edu/~yima

IMAGES AND GEOMETRY – A Little History of Perspective Imaging Pinhole (perspective) imaging, in most ancient civilizations. Euclid, perspective projection, 4th century B.C., Alexandria Pompeii frescos, 1st century A.D. Image courtesy of C. Taylor

IMAGES AND GEOMETRY – A Little History of Perspective Imaging Fillippo Brunelleschi, first Renaissance artist painted with correct perspective,1413 “Della Pictura”, Leone Battista Alberti, 1435 Leonardo Da Vinci, stereopsis, shading, color, 1500s “The scholar of Athens”, Raphael, 1518 Image courtesy of C. Taylor

IMAGES AND GEOMETRY – The Fundamental Problem Input: Corresponding “features” in multiple images. Output: Camera calibration, pose, scene structure, surface photometry. Jana’s apartment

IMAGES AND GEOMETRY – History of “Modern” Geometric Vision Chasles, formulated the two-view seven-point problem,1855 Hesse, solved the above problem, 1863 Kruppa, solved the two-view five-point problem, 1913 Longuet-Higgins, the two-view eight-point algorithm, 1981 Liu and Huang, the three-view trilinear constraints, 1986 Huang and Faugeras, SVD based eight-point algorithm, 1989 Tomasi and Kanade, (orthographic) factorization method, 1992 Ma, Huang, Kosecka, Vidal, multiple-view rank condition, 2000

APPLICATIONS – 3-D Modeling and Rendering

APPLICATIONS – 3-D Modeling and Rendering Image courtesy of Paul Debevec

APPLICATIONS – Image Morphing, Mosaicing, Alignment Images of CSL, UIUC

First-down line and virtual advertising APPLICATIONS – Real-Time Sports Coverage First-down line and virtual advertising Image courtesy of Princeton Video Image, Inc.

APPLICATIONS – Real-Time Virtual Object Insertion UCLA Vision Lab

APPLICATIONS – Autonomous Highway Vehicles Image courtesy of E.D. Dickmanns

APPLICATIONS – Unmanned Aerial Vehicles (UAVs) Rate: 10Hz Accuracy: 5cm, 4o Berkeley Aerial Robot (BEAR) Project

Preliminaries: Imaging Geometry & Image Formation (Lecture B) Multiple-View Geometry for Image-Based Modeling Preliminaries: Imaging Geometry & Image Formation (Lecture B) Yi Ma Perception & Decision Laboratory Decision & Control Group, CSL Image Formation & Processing Group, Beckman Electrical & Computer Engineering Dept., UIUC http://decision.csl.uiuc.edu/~yima

Preliminaries: Imaging Geometry and Image Formation INTRODUCTION 3D EUCLIDEAN SPACE & RIGID-BODY MOTION Coordinates and coordinate frames Rigid-body motion and homogeneous coordinates GEOMETRIC MODELS OF IMAGE FORMATION Lens & Lambertian surfaces Pinhole camera model CAMERA INTRINSIC PARAMETERS & RADIAL DISTORTION From space to pixel coordinates Notation: image, preimage, and coimage Radial distortion and correction SUMMARY OF NOTATION

Coordinates of a point in space: 3D EUCLIDEAN SPACE - Cartesian Coordinate Frame Coordinates of a point in space: Standard base vectors: We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.

A “free” vector is defined by a pair of points : 3D EUCLIDEAN SPACE - Vectors A “free” vector is defined by a pair of points : Coordinates of the vector : We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.

3D EUCLIDEAN SPACE – Inner Product and Cross Product Inner product between two vectors: Cross product between two vectors: We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.

Coordinates are related by: RIGID-BODY MOTION – Rotation Rotation matrix: We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero. Coordinates are related by:

Coordinates are related by: RIGID-BODY MOTION – Rotation and Translation We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero. Coordinates are related by:

3D coordinates are related by: RIGID-BODY MOTION – Homogeneous Coordinates 3D coordinates are related by: Homogeneous coordinates are related by: Homogeneous coordinates of a vector: We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.

BRDF Lambertian thin lens small FOV IMAGE FORMATION – Lens, Light, and Surfaces image irradiance surface radiance We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero. BRDF Lambertian thin lens small FOV

Pinhole Frontal pinhole IMAGE FORMATION – Pinhole Camera Model We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.

Homogeneous coordinates IMAGE FORMATION – Pinhole Camera Model 2D coordinates Homogeneous coordinates We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.

CAMERA PARAMETERS – Pixel Coordinates calibrated coordinates Linear transformation pixel coordinates We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero.

(intrinsic parameters) CAMERA PARAMETERS – Calibration Matrix and Camera Model Ideal pinhole Pixel coordinates Calibration matrix (intrinsic parameters) Projection matrix We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero. Camera model

Nonlinear transformation along the radial direction CAMERA PARAMETERS – Radial Distortion Nonlinear transformation along the radial direction We will always use column vectors, except for one case which I will mention later on. Given a three dimensional vector u, we use u-hat to represent a 3x3 skew symmetric matrix associated to it. In the literature, people also use u-product for the same thing. Using this notation, u-hat multiplying a vector v is then equal to their cross product. In particular, u crosses with u itself gets zero. Distortion correction: make lines straight

IMAGE FORMATION – Image of a Point Homogeneous coordinates of a 3-D point Homogeneous coordinates of its 2-D image Projection of a 3-D point to an image plane Now let me quickly go through the basic mathematical model for a camera system. Here is the notation. We will use a four dimensional vector X for the homogeneous coordinates of a 3-D point p, its image on a pre-specified plane will be described also in homogeneous coordinate as a three dimensional vector x. If everything is normalized, then W and z can be chosen to be 1. We use a 3x4 matrix Pi to denote the transformation from the world frame to the camera frame. R may stand for rotation, T for translation. Then the image x and the world coordinate X of a point is related through the equation, where lambda is a scale associated to the depth of the 3D point relative to the camera center o. But in general the matrix Pi can be any 3x4 matrix, because the camera may add some unknown linear transformation on the image plane. Usually it is denoted by a 3x 3 matrix A(t).

Image of a 3-D point Coimage of the point Preimage of the point NOTATION – Image, Coimage, Preimage of a Point Image of a 3-D point Coimage of the point Preimage of the point Now let me quickly go through the basic mathematical model for a camera system. Here is the notation. We will use a four dimensional vector X for the homogeneous coordinates of a 3-D point p, its image on a pre-specified plane will be described also in homogeneous coordinate as a three dimensional vector x. If everything is normalized, then W and z can be chosen to be 1. We use a 3x4 matrix Pi to denote the transformation from the world frame to the camera frame. R may stand for rotation, T for translation. Then the image x and the world coordinate X of a point is related through the equation, where lambda is a scale associated to the depth of the 3D point relative to the camera center o. But in general the matrix Pi can be any 3x4 matrix, because the camera may add some unknown linear transformation on the image plane. Usually it is denoted by a 3x 3 matrix A(t).

Coimage of a 3-D line Preimage of the line Image of the line NOTATION – Image, Coimage, Preimage of a Line Coimage of a 3-D line Preimage of the line Image of the line Now let me quickly go through the basic mathematical model for a camera system. Here is the notation. We will use a four dimensional vector X for the homogeneous coordinates of a 3-D point p, its image on a pre-specified plane will be described also in homogeneous coordinate as a three dimensional vector x. If everything is normalized, then W and z can be chosen to be 1. We use a 3x4 matrix Pi to denote the transformation from the world frame to the camera frame. R may stand for rotation, T for translation. Then the image x and the world coordinate X of a point is related through the equation, where lambda is a scale associated to the depth of the 3D point relative to the camera center o. But in general the matrix Pi can be any 3x4 matrix, because the camera may add some unknown linear transformation on the image plane. Usually it is denoted by a 3x 3 matrix A(t).

IMAGE FORMATION – Coimage of a Line Homogeneous representation of a 3-D line Homogeneous representation of its 2-D coimage Projection of a 3-D line to an image plane First let us talk about line features. To describe a line in 3-D, we need to specify a base point on the line and a vector indicating the direction of the line. On the image plane we can use a three dimensional vector l to describe the image of a line L. More specifically, if x is the image of a point on this line, its inner product with l is 0.

IMAGE FORMATION – Multiple Images “Preimages” are all incident at the corresponding features. Now consider multiple images of a simplest object, say a cube. All the constraints are incidence relations, are all of the same nature. Is there any way that we can express all the constraints in a unified way? Yes, there is. . . .

An Invitation to 3-D Vision: Ma, Soatto, Kosecka, Sastry, LIST OF REFERENCES Chapters 2 & 3 An Invitation to 3-D Vision: From Images to Geometric Models, Ma, Soatto, Kosecka, Sastry, Springer-Verlag, 2003. This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint. As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are. We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered. Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.