776 Computer Vision Jan-Michael Frahm, Enrique Dunn Spring 2013

Camera

Capture an Object each object point reflects rays camera sensor captures all rays reflected in the direction of itself => not a usable image since all rays contribute to many pixels (blurred image) film, camera sensor

Capture an Object each object point reflects rays camera sensor captures all rays reflected in the direction of itself  not a usable image since all rays contribute to many pixels (blurred image) add barrier to only allow a few rays per object point film, camera sensor

Capture an Object Barrier is known as aperture o blocks of most of the rays o reduces blur image source: S. Seitz

Capture an Object Barrier is known as aperture o blocks of most of the rays o reduces blur Pinhole camera (camera obscura) Interior of camera obscura (Sunday Magazine, 1838) Camera obscura (France, 1830)

Capture an Object Barrier is known as aperture o blocks of most of the rays o reduces blur Pinhole camera (camera obscura) o infinitely small aperture (pinhole) o captures all rays going through the pinhole (pencil of rays) o image is captured on image plane image source: S. Seitz

Camera Projection Project 3D point [X,Y,Z] into image point [x,y] Z camera in origin [0,0,0] no rotation, translation of the camera

Figures © Stephen E. Palmer, 2002 Dimensionality reduction: from 3D to 2D 3D world2D image What is preserved? Straight lines, incidence (except for parallel) What have we lost? Angles, lengths Slide by A. Efros

Camera Projection Arbitrary Camera Camera rotated with R and translated with T

All points on a ray project into the same point Line in 3D is described by (Plücker Line): D is the direction of the line N is orthogonal to O and D Point [x,y,1] is on the above line and for camera image plane at Z=1 this is the 3D coordinate of the image point in R 3 All pointsρ[x,y,1] are on the ray too Projective space P 2 is the space of rays, described by ρ[x,y,1] for all x,y in R D N O

Projective Space P 2 The projective space IP is the set of hyper-planes of the vector space V where the vector space V is defined over the commutative body K. The dimension of P is then given as dim(P) = dim(V ) − 1. The elements x of P are called projective points. The point x = O dim(V)×1 is not contained in the projective space P. Each point x is represented through its spanning vector. In case of R 2 (image) the projective space P 2 is the space of the projection rays

3D Projective Space e i : basis vectors O: coordinate origin/ camera position e2e2 e3e3 e1e1 o X Vector to point relative to O: a1a1 a3a3 a2a2 Point in world coordinates: Describe 3D point relative to some origin (camera/pinhole position)

Homogeneous coordinates Unified notation: include origin in affine basis Affine basis matrix Homogeneous Coordinates of X e2e2 e3e3 o X a1a1 a3a3 a2a2

Properties of affine transformation Parallelism is preserved ratios of length, area, and volume are preserved Transformations can be concatenated: Transformation T affine combines linear mapping and coordinate shift in homogeneous coordinates  Linear mapping with A 3x3 matrix  coordinate shift with t 3 translation vector

Special transformation: Rotation Rigid transformation: Angles and lengths preserved R is orthonormal matrix defined by three angles around three coordinate axes ezez eyey exex  Rotation with angle  around e z

Special transformation: Rotation Rotation around the coordinate axes can be concatenated: ezez eyey exex    Inverse of rotation matrix is transpose:

Camera Projection in general pose Rotation [R] Camera center C X World coordinates x Still missing the projection into P 2 to obtain the image coordinates

Perspective projection Perspective projection models pinhole camera:  scene geometry is affine R 3 space with coordinates X=(X,Y,Z,1) T  camera focal point (pinhole) in C=(0,0,0,1) T, camera viewing direction along Z  image plane (x,y) in  (R 2 ) aligned with plane (X,Y) at Z= Z 0  scene point X projects onto point X p on plane surface X Y Z0Z0 C  Image plane Camera center Z x y (R 2 )

Projective Transformation Projective Transformation maps X onto X p X Y C Projective Transformation linearizes projection

Perspective Projection Dimension reduction from P 3 into P 2 by projection onto  (R 2 ) X Y C

X Y Image center c= (c x, c y ) T Projection center Z (Optical axis) Pixel scale f= (f x,f y ) T x y Pixel coordinates x = (y,x) T Image plane and image sensor A sensor with picture elements (Pixel) is added onto the image plane Image sensor Image-sensor mapping: Pixel coordinates are related to image coordinates by affine transformation K with five parameters:  Image center c=(c x,c y ) T defines optical axis  Pixel size and pixel aspect ratio defines scale f=(f x,f y ) T  image skew s to model angle between pixel rows and columns

Projection matrix P Camera projection matrix P combines:  inverse affine transformation T cam -1 from general pose to origin  Perspective projection P 0 to image plane at Z 0 =1  affine mapping K from image to sensor coordinates World to camera coord. trans. matrix (4x4) Perspective projection matrix (3x4) Camera to pixel coord. trans. matrix (3x3) = 2D point (3x1) 3D point (4x1)

Projection P matrix transforms 3D world coordinates into image coordinates

Orthographic Projection Special case of perspective projection o Distance from center of projection to image plane is infinite o Also called “parallel projection” o What’s the projection matrix? Image World Slide by Steve Seitz

Projection properties Many-to-one: any points along same visual ray map to same point in image Points → points o But projection of points on focal plane is undefined Lines → lines (collinearity is preserved) o But lines through focal point (visual rays) project to a point Planes → planes (or half-planes) o But planes through focal point project to lines slide: S. Lazebnik

Vanishing points Each direction in space has its own vanishing point o All lines going in that direction converge at that point o Exception: directions parallel to the image plane slide: S. Lazebnik

Vanishing points Each direction in space has its own vanishing point o All lines going in that direction converge at that point o Exception: directions parallel to the image plane How do we construct the vanishing point of a line? image plane camera center line on ground plane vanishing point slide: S. Lazebnik

One-point perspective Masaccio, Trinity, Santa Maria Novella, Florence, One of the first consistent uses of perspective in Western art slide: S. Lazebnik

Facing Real Cameras There are undesired effects in real situations o perspective distortion

Perspective distortion Problem for architectural photography: converging verticals Where do they converge to? image source: F. Durand

Perspective distortion Problem for architectural photography: converging verticals Solution: view camera (lens shifted w.r.t. film) Source: F. Durand Tilting the camera upwards results in converging verticals Keeping the camera level, with an ordinary lens, captures only the bottom portion of the building Shifting the lens upwards results in a picture of the entire subject

Perspective distortion Problem for architectural photography: converging verticals Result: Source: F. Durand shifted lens tilted camera with regular lens

Perspective distortion image: Wikipedia Which image is captured with a shifted lens?

Perspective distortion What does a sphere project to? Image source: F. Durand

Perspective distortion What does a sphere project to? slide: S. Lazebnik

Perspective distortion The exterior columns appear bigger The distortion is not due to lens flaws Problem pointed out by Da Vinci Slide by F. Durand

Perspective distortion: People slide: S. Lazebnik

Facing Real Cameras There are undesired effects in real situations o perspective distortion Camera artifacts

Home-made pinhole camera Why so blurry? Slide by A. Efros

Shrinking the aperture Why not make the aperture as small as possible? o Less light gets through o Diffraction effects… Slide by Steve Seitz

Shrinking the aperture

Facing Real Cameras There are undesired effects in real situations o perspective distortion Camera artifacts o aperture is not infinitely small

Adding a lens A lens focuses light onto the film o Thin lens model: Rays passing through the center are not deviated (pinhole projection model still holds) Slide by Steve Seitz

Adding a lens A lens focuses light onto the film o Thin lens model: Rays passing through the center are not deviated (pinhole projection model still holds) All parallel rays converge to one point on a plane located at the focal length f Slide by Steve Seitz focal point f

Adding a lens A lens focuses light onto the film o There is a specific distance at which objects are “in focus” other points project to a “circle of confusion” in the image “circle of confusion” Slide by Steve Seitz

Thin lens formula What is the relation between the focal length (f), the distance of the object from the optical center (D), and the distance at which the object will be in focus (D’)? f D D’ Slide by Frédo Durand object image plane lens

Thin lens formula f D D’ Similar triangles everywhere! Slide by Frédo Durand object image plane lens

Thin lens formula f D D’ Similar triangles everywhere! y’ y y’/y = D’/D Slide by Frédo Durand object image plane lens

Thin lens formula f D D’ Similar triangles everywhere! y’ y y’/y = D’/D y’/y = (D’-f)/f Slide by Frédo Durand object image plane lens

Thin lens formula f D D’ 1 D 11 f += Any point satisfying the thin lens equation is in focus. Slide by Frédo Durand object image plane lens

Real Lenses Zoom lens image: Simal

slide: S. Lazebnik Lens Flaws: Chromatic Aberration Lens has different refractive indices for different wavelengths: causes color fringing Near Lens Center Near Lens Outer Edge

Lens flaws: Spherical aberration Spherical lenses don’t focus light perfectly Rays farther from the optical axis focus closer slide: S. Lazebnik

Lens flaws: Vignetting slide: S. Lazebnik