Geometry of Images Pinhole camera, projection A taste of projective geometry Two view geometry:  Homography  Epipolar geometry, the essential matrix  Camera calibration, the fundamental matrix Stereo vision: 3D shape reconstruction from two views Factorization: reconstruction from many views

Geometry of Images

Cameras Camera obscura dates from 15 th century First photograph on record shown in the book Basic abstraction is the pinhole camera Current cameras contain a lens and a recording device (film, CCD) The human eye functions very much like a camera

Camera Obscura "Reinerus Gemma-Frisius, observed an eclipse of the sun at Louvain on January 24, 1544, and later he used this illustration of the event in his book De Radio Astronomica et Geometrica, It is thought to be the first published illustration of a camera obscura..." Hammond, John H., The Camera Obscura, A Chronicle

Why Not Use Pinhole Camera If pinhole is too big - many directions are averaged, blurring the image Pinhole too small diffraction effects blur the image Generally, pinhole cameras are dark, because a very small set of rays from a particular point hits the screen.

Lenses

Lenses collect light from a large hole and direct it to a single point Overcome the darkness of pinhole cameras But there is a price  Focus  Radial distortions  Chromatic abberations  … Pinhole is useful as a model

Pinhole Camera

Single View Geometry f ∏

Notation O – Focal center π – Image plane Z – Optical axis f – Focal length

Projection f x y Z X Y

Perspective Projection Homogeneous Coordinates

Orthographic Projection Projection rays are parallel Image plane is fronto-parallel ( orthogonal to rays) Focal center at infinity

Scaled Orthographic Projection Also called “weak perspective”

Pros and Cons of Projection Models Weak perspective has simpler math.  Accurate when object is small and distant.  Most useful for recognition of objects. Pinhole perspective much more accurate for scenes.  Used in structure from motion. When accuracy really matters, we must model the real camera  Use perspective projection with other calibration parameters (e.g., radial lens distortion)

World Cup 66: England-Germany

World Cup 66: Second View

World Cup 66: England-Germany Conclusion: no goal (missing 3 inches) (Reid and Zisserman, “Goal-directed video metrology”)

Euclidean Geometry Answers the question what objects have the same shape (= congruent) Same shapes are related by rotation and translation

Projective Geometry Answers the question what appearances (projections) represent the same shape Same shapes are related by a projective transformation

Perspective Distortion Where do parallel lines meet? Parallel lines meet at the horizon (“vanishing line”)

Line Perspective Pencil of rays Perspective mapping

Plane Perspective

Ideal points Projective transformation can map ∞ to a real point

Coordinates in Euclidean Space ∞ Not in space

Coordinates in Projective Line ∞ k(0,1) k(1,0) k(2,1) k(1,1) k(-1,1) Points on a line P 1 are represented as rays from origin in 2D, Origin is excluded from space “Ideal point”

Coordinates in Projective Plane k(0,0,1) k(x,y,0) k(1,1,1) k(1,0,1) k(0,1,1) “Ideal point”

2D Projective Geometry: Basics A point: A line: we denote a line with a 3-vector Points and lines are dual: p is on l if Intersection of two lines: A line through two points:

Cross Product Every entry is a determinant of the two other entries Area of parallelogram bounded by u and v

Ideal points Q: How many ideal points are there in P 2 ? A: 1 degree of freedom family – the line at infinity

Projective Transformation (Homography) Any finite sequence of perspectivities is a projective transformation Projective transformations map lines to lines Represented by an invertible 3x3 linear transformation (up to scale), denote by H, or Given homography H, how does it operate on lines?

Rotation: Translation: Euclidean Transformations (Isometries)

Hierarchy of Transformations Isometry (Euclidean), Similarity, Affine, general linear Projective,

Invariants LengthAreaAnglesParallelism Isometry √√√√ Similarity × × (Scale) √√ Affine ×××√ Projective ××××

Perspective Projection Note: P and p are related by a scale factor, but it is a different factor for each point (depends on Z)

Two View Geometry When a camera changes position and orientation, the scene moves rigidly relative to the camera In two cases this results in homography:  Camera rotates around its focal point  The scene is planar In this case the mapping from one image to the second is one to one and depth cannot be recovered In the general case the induced motion is more complex and is captured by what is termed “epipolar geometry”

Camera Rotation (R is 3x3 non-singular)

Intuitively A sequence of two perspectivities Algebraically Planar Scenes Scene Camera 1 Camera 2

Two Views Related by Homography Two images are related by homography: One to one mapping from p to p’ H contains 8 degrees of freedom Given correspondences, each point determines 2 equations 4 points are required to recover H Depth cannot be recovered

General Case: Epipolar Lines epipolar line

Epipolar Plane epipolar plane epipolar line Baseline P O O’

Epipole Every plane through the baseline is an epipolar plane, and determines a pair of epipolar lines in the two images Two systems of epipolar lines are obtained, each system intersects in a point, the epipole The epipole is the projection of the center of the other camera epipolar plane epipolar lines Baseline O O’

Epipolar Lines epipolar plane epipolar line Baseline P O O’ To define an epipolar plane, we define the plane through the two camera centers O and O’ and some point P. This can be written algebraically (in some world coordinates as follows:

Epipolar Lines Set world coordinates around the first camera What to do with O’P? Every rotation changes the observed coordinate in the second image We need to de-rotate to make the second image plane parallel to the first Replacing by image points

Essential Matrix Denote this by: Then Define, then E is called the “essential matrix”

Essential Matrix E is rank 2. Its (right and left) null spaces are the two epipoles is linear and homogeneous in E, E can be recovered up to scale using 8 points The additional constraint detE=0 reduces the needed points to 7 In fact, there are only 5 degrees of freedom in E,  3 for rotation  2 for translation (up to scale), determined by epipole

Internal Calibration Camera parameters may be unknown: (c x,c y ) camera center, (a x,a y ) pixel dimensions, b skew Radial distortions are not accounted for

Fundamental Matrix F, the fundamental matrix, too is rank 2 F has 7 d.o.f. (9 entries, homogeneous, and detF=0)

Summary HomographyPerspectiveOrthographic Form ShapeOne-to-one (Group) Concentric epipolar lines Parallel epipolar lines D.o.f.8(5) 5 Eqs/pnt211 Minimal configuration 45+ (8, linear)4 DepthNoYes, up to scale No, third view required