Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Augmented Reality X Y Z Real scene, scene coordinates C xCxC yCyC zCzC R, t Camera(s) xVxV yVyV zVzV Visualization (screen, HMD) R, t Real table Augmented plant

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Example 1: ARToolkit

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Example 2: Structure + Motion [Schweighofer]

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Pinhole Camera real camera image plane π i (u,v): z = -f u v z x y principal point (u 0,v 0 ) optical axis p(u,v) P(x,y,z) f focal length f 2D projection 3D scene p(u,v) line of sight = viewing direction P(x,y,z) Pinhole C … center of projection C

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Projective Geometry π 0... z = 0π i... z = f x z y x y u v p(u,v) P(x,y,z) P P !!! projective camera, normalized camera: f = 1 1 stationary camera 1 coordinate system (x,y,z) camera-centered coordinate system scene coordinate system Only points in π 0 are not projected to π i (u 0,v 0 ) C

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Projective Images: Examples, Properties Impression of depth in images Parallel lines meet at infinity infinity is projected to finite location in the image horizon points at infinity, … [Triggs and Mohr]

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Projective Images: Scaling

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Projective Images: Foreshortening

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Projective Images: Parallel Lines Meet [Sonka, Hlavac, Boyle]

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Projective Geometry vs. Computer Vision points in π 1 straight lines of sight projective reconstruction geometry, precise known correspondences discrete pixels in π i sampling theorem lens distortion, aperture, depth of field oriented projective rec. in front of camera inherently imprecise estimation, minimization outliers robustness

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Example: Stereo Reconstruction P C1C1 C2C2 projective geometry computer vision P ~

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz A unified geometric + algebraic framework Point Line Algebraic Projective Geometry (1)

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Duality point line Unified approach: projective n-space P n point (n+1) - vector Algebraic Projective Geometry (2)

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Homogeneous Coordinates in P n Equivalence class of vectors forms P 2 … projective plane Homogeneous coordinates, but only 2 DoF inhomogeneous

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Equivalence Class of Vectors Without further knowledge, such situations cannot be distingushed ! A further example: Equivalence ofa toy car, closeup shot, and real car, distant shot

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Point Line ideal points treated like any point x 30 Fluchtpunkte line at infinity the planes horizon The Projective Plane (1) intersection of parallel lines !

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz The Projective Plane (2) Adding the ideal points to R 2 leads to the projective plane P 2 Covers all homo- geneous coordinates [Hartley+Zisserman]

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz The Projective Plane (3) π Image of the horizon of π, line at infinity of π vanishing point, Fluchtpunkt = Bild eines Fernpunktes Projective geometry can map infinitely far points / lines to finite ones Projective geometry can map infinitely far points / lines to finite ones No difference between finite and infinite No difference between finite and infinite e.g. hyperbola is one continuous conic e.g. hyperbola is one continuous conic

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz There is also Projective Space P 3 … Points Planes Lines: 4 DoF Dual line L*: duality point plane duality L L*

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Projective Transformations in P n projective transformation = collineation = projectivity = homography H Invertible mapping P n P n geradentreue Abbildung (n+1) x (n+1) matrix In P 2 : H has (n+1) 2 -1 DoF, H is non-singular

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Projective Transformations in P 2 Translation Rotation Scaling Any combination, e.g.

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz A Remark on Conics 2 nd degree equation in the plane Homog. coord: Conic C: Five DoF, 5 points define a conic

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Back to Homographies – Examples (1) Mapping between planes central projection may be expressed by x=Hx [Hartley+Zisserman]

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Back to Homographies – Examples (2) Removing projective distortion [Hartley+Zisserman]

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Back to Homographies – Examples (3) [Hartley+Zisserman]

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Transformation for Points, Lines, Conics Point Line Conic

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz A Hierarchy of Transformations / Geometries (1) Isometric / Euclidean –Invariants: length, angle, area Similarity –Invariants: ratios of length / areas, angle, parallel lines

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz A Hierarchy of Transformations / Geometries (2) Affine: – –6 DoF: 2 x scale λ 1,λ 2 ; 2 x rot. θ,Ф; 2 x translation –Invariants: parallel lines, ratios of parallel lengths, ratios of areas

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz A Hierarchy of Transformations / Geometries (3) Projective: –8 DoF: 2 x scale λ 1,λ 2 ; 2 x rot. θ,Ф; 2 x translation; 2 x line at infinity –Invariant: Cross-ratio CR of 4 collinear points ABCD

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz A Hierarchy of Transformations / Geometries (4) Projective 8dof Affine 6dof Similarity 4dof Euclidean 3dof In 2D, a square transforms to:

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz A Hierarchy of Transformations / Geometries (5) Projective 15dof Affine 12dof Similarity 7dof Euclidean 6dof In 3D, a cube transforms to:

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Stratification In AR, we take perspective images, but we require metric (Euclidean) reconstruction! How? The stratification of 3D geometry [Pollefeys 2.2]

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Stratification of 2D / 3D Geometry Many possibilities, many approaches Examples: –Known · directions –Known points, lines, planes at –Known lengths in the scene –IAC (self-calibration) –Known camera intrinsics Camera calibration + relative orientation Multiview geometry, structure+motion unknownscenes

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Stratification Examples (1) Known points, line at infinity v1v1 v2v2 l1l1 l2l2 l4l4 l3l3 l perspectiveaffine

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Stratification Examples (1) Known · directions affinemetric (similarity, unknown scale)

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Stratification Examples (2) Known plane at infinity perspectiveaffine

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Stratification Examples (2) Known · directions affinemetric (similarity, unknown scale)

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Stratification Examples (3) Known lengths metric(similarity, unknown scale) metric(Euclidean, known scale) [Pollefeys IJCV99]

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Stratification Examples (4) ARToolkit perspective metric(Euclidean, known scale)

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Video AR is (rather) simple Known artificial targets / markers Uncalibrated perspective camera –But: collineation required –Problems when e.g. strong lens distortions Augmentation of the video frames Examples –Artoolkit –Kutulakos Can be related to scene coordinates, but requires ground truth for markers

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz ARToolkit Demo ISAR 2000 observers view immersive view

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Kutulakos Calibration-Free AR [IEEE Trans. Visualization and Graphics 1998]

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Field Maintenance Support [ARVIKA]

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 1 Projective Geometry Axel Pinz Scene Structure + Camera Motion (the harder, but more general approach to AR) Many possible approaches Monocular, calibrated, known natural landmarks [Ribo] Stereo, calibrated [Schweighofer] Monocular, calibrated [Murray] Monocular, uncalibrated [Pollefeys] –not (yet?) in real time ! calibration ! calibration ! unknownscene,unknown natural naturallandmarks