1. CS 395/495-26: Spring 2004 IBMR: 2-D Projective Geometry --Introduction-- Jack Tumblin jet@cs.northwestern.edu

2. Recall: Scene & Image Light + 3D Scene: Illumination, shape, movement, surface BRDF,… Image Plane I(x,y) Angle( ,  ) Position(x,y) 2D Image: Collection of rays through a point GOAL: a Reversible Mapping Scene angles  image positions

3. View Interpolation: How? LATER: 3D Projection (Hartley,Zisserman Chapter 2)LATER: 3D Projection (Hartley,Zisserman Chapter 2) From a 3D scene Find new views of that scene view1 view2

4. View Interpolation: How? BUT FIRST, the simpler case Hartley, Zisserman Chapter 1: 2D ProjectionBUT FIRST, the simpler case Hartley, Zisserman Chapter 1: 2D Projection From a flat 2D image, Find new views of that image view1 view2

5. Answer: 2D Homogeneous Coords Chapter 1: 2D Projection From a flat 2D image, Find new views of that image view1 view2 Cartesian (x,y) coordinates in R 2 2D Homogeneous () coordinates in P 2 2D Homogeneous (x 1 x 2 x 3 ) coordinates in P 2

6. Overview Chapter 1: any 2D plane, as seen by any 3D camera view (texture mapping generalized)Chapter 1: any 2D plane, as seen by any 3D camera view (texture mapping generalized) Homogeneous Coordinates are WonderfulHomogeneous Coordinates are Wonderful Everything is a Matrix:Everything is a Matrix: –Conics (you can skip for now) –Aside: interpolation of pixels and points… Transform your Point of View: H matrixTransform your Point of View: H matrix Parts of H: useful kinds of TransformsParts of H: useful kinds of Transforms

7. 2-D Homogeneous Coordinates WHY? makes MUCH cleaner math!WHY? makes MUCH cleaner math! –Unifies lines and points –Puts perspective projection into matrix form –No divide-by-zero; defines horizon, infinities … xy But in P 2, write same point x as where: (x,y) in R 2, write point x as xyxy x1x2x3x1x2x3 x = x 1 / x 3, y = x 2 / x 3, x 3 = anything non-zero! (but usually defaults to 1)

8. Every Homog. point (x 1,x 2,x 3 ) describes a 3D rayEvery Homog. point (x 1,x 2,x 3 ) describes a 3D ray ‘Phantom’ dimension x 3 is the ‘z-buffer’ value‘Phantom’ dimension x 3 is the ‘z-buffer’ value 3D  2D ‘hard-wired’:3D  2D ‘hard-wired’: (Fig 1.1, pg 8) “Homogeneous coords allows translation matrix:” (true. But there is more!)“Homogeneous coords allows translation matrix:” (true. But there is more!) Useful 3D Graphics Ideas (0,0,-1) (x max, y max, -1) An ‘Ideal Point’ at (0,0,0) Center of Projection x y = 1 1 0 3 0 1 5 0 0 1 x+3y+51 ‘Image Plane’ (x,y,-1)

9. 2-D Homogeneous Coordinates WHAT?! Why Why ‘default’ value of 1? WHAT?! Why x 3 ? Why ‘default’ value of 1? Look at lines in R 2 :Look at lines in R 2 : –‘line’ == all (x,y) points where –scale by ‘k’  no change: Using ‘x 3 ’ for points UNIFIES notation:Using ‘x 3 ’ for points UNIFIES notation: –line is a 3-vector named l –now point (x,y) is a 3-vector too, named x ax + by + c = 0 kax + kby + kc = 0 ax + by + c = 0 = 0 x 1 x 2 x 3 abcabc x T. l = 0

10. l 1  l 2 = x x 1  x 2 = l 2D Homogeneous Coordinates Important Properties 1 (see book for details) 3 coordinates, but only 2 degrees of freedom (only 2 ratios (x 1 / x 3 ), (x 2 / x 3 ) can change)3 coordinates, but only 2 degrees of freedom (only 2 ratios (x 1 / x 3 ), (x 2 / x 3 ) can change) DUALITY: points, lines are interchangeableDUALITY: points, lines are interchangeable –Line Intersections = point: (a 3D cross-product) (a 3D cross-product) –Point ‘Intersections’ = line: –Projective theorem for lines  theorem for points!

11. Homogeneous Coordinates Important Properties 2 (see book for details) Neatly Sidesteps ‘divide-by-zero’:Neatly Sidesteps ‘divide-by-zero’: –in (x,y) space (R 2 ) –Remember, we store (x 1,x 2, x 3 ) T, then compute (x 1 / x 3 ), (x 2 / x 3 ) only if OK. “OK, yeah, sure, but so what?”...“OK, yeah, sure, but so what?”...

12. Homogeneous Coordinates Why is deferring divide-by-zero so important? Why is deferring divide-by-zero so important? Cleanly Defines all ‘Points at Infinity’ or ‘Ideal Points’ (x,y,0) TCleanly Defines all ‘Points at Infinity’ or ‘Ideal Points’ (x,y,0) T –==outermost pixels (+/- , +/-  ) of an infinite image plane ((x,y), or R 2 ) –(x,y,0) is an entire plane of points in the (semi-bogus) 3D space (x 1,x 2, x 3 ) T space –but it is a ‘circle around infinity’ in the (legitimate) P 2 projective space (x 1 /x 3, x 2 /x 3 ) T –Note! ‘Center of Projection’ is an ideal point. (Do outer limits of image plane ‘wrap around’ to that point?!?!)

13. Homogeneous Coordinates Why is deferring divide-by-zero so important? Why is deferring divide-by-zero so important? Cleanly defines one ‘Line at infinity’ l   (0,0,1) T, or “0x +0y +1=0” All ideal points are on l  : proof? l (x 1,x 2,0) T = 0All ideal points are on l  : proof? l (x 1,x 2,0) T = 0 Any line l intersects with l  line at an ideal pointAny line l intersects with l  line at an ideal point Two parallel lines l and l’ always meet at an ideal point Let l = (a,b,c) T and l’ = (a,b,c’) T … (page 7)Two parallel lines l and l’ always meet at an ideal point Let l = (a,b,c) T and l’ = (a,b,c’) T … (page 7)

14. Homogeneous Coordinates Important Properties 3 (see book for details) View Interpolation (‘Projective Transform’)View Interpolation (‘Projective Transform’) –‘Central’ image plane (x,y,-1) T Choose a known point x’ Choose a known point x’ –Apply 3x3 Matrix H to (x,y,-1) T to make point on OTHER plane Ray through known point x’ pierces unknown point x Ray through known point x’ pierces unknown point x (0,0,-1) ‘Image Plane’ (x,y,-1) (x,y,-1) (0,0,0) x x’ x1x2x3x1x2x3 h 11 h 12 h 13 h 21 h 22 h 21 h 31 h 32 h 33 = x’ 1 x’ 2 x’ 3 H x = x’ 3D world plane

15. END

16. Every xy point describes a 3D ray in (x 1,x 2,x 3 )Every xy point describes a 3D ray in (x 1,x 2,x 3 ) ‘Phantom’ dimension x 3 is the (lost) distance-to-camera‘Phantom’ dimension x 3 is the (lost) distance-to-camera Homogeneous coords allows matrix for 2-D translation:Homogeneous coords allows matrix for 2-D translation: (BUT a 3x3 transform in R 3 Euclidean Space (x,y,z) can’t translate—only rotate,scale, skew!)(BUT a 3x3 transform in R 3 Euclidean Space (x,y,z) can’t translate—only rotate,scale, skew!) Useful 3D Graphics Ideas An ‘Ideal Point’ (0,0,0) at Center of Projection x y = 1 1 0 3 0 1 5 0 0 1 x+3y+51 ‘Central Image Plane’ (x,y,1) -x 3 x2x2x2x2 x1x1x1x1 x y = z 1 0 3 0 1 5 0 0 1 x+zy+zz Ray

17. l 1  l 2 = x x 1  x 2 = l 2D Homogeneous Coordinates 3 coords, but only 2 degrees of freedom (2DOF) (x 1 / x 3 ), (x 2 / x 3 )3 coords, but only 2 degrees of freedom (2DOF) (x 1 / x 3 ), (x 2 / x 3 ) –Ideal points == (x,y,0) (every line hits the horizon at an ideal point) –Line Intersections = point: (a 3D cross-product) (a 3D cross-product) –Point ‘Intersections’ = line: –Duality: P 2 theorem for lines  P 2 theorem for points! An ‘Ideal Point’ (0,0,0) at Center of Projection ‘Central Image Plane’ (x,y,1) -x 3 x2x2x2x2 x1x1x1x1 Ray

18. l 1  l 2 = x x 1  x 2 = l 2D Homogeneous Coordinates 3 coords, but only 2 degrees of freedom (2DOF) (x 1 / x 3 ), (x 2 / x 3 )3 coords, but only 2 degrees of freedom (2DOF) (x 1 / x 3 ), (x 2 / x 3 ) –Ideal points == (x,y,0) –Line Intersections = point: (a 3D cross-product) (a 3D cross-product) –Point ‘Intersections’ = line: –Duality: P 2 theorem for lines  P 2 theorem for points! An ‘Ideal Point’ (0,0,0) at Center of Projection ‘Central Image Plane’ (x,y,1) -x 3 x2x2x2x2 x1x1x1x1 Ray Left-Handed: use –x 3 or (x,y,-1) image ctr, or...

19. x’ Apply the 3x3 matrix H x’ = Hx Apply the 3x3 matrix H x’ = Hx Projective Transform H x3x3x3x3 x2x2x2x2 x1x1x1x1 x y x’ y’ (x,y) x (x’,y’) Homog. coords x’ = [x’,y’,1] T  2D image (x’,y’) 2D image (x,y)  Homog. coords [x,y,1] T = x

20. x’ Apply the 3x3 matrix H x’ = Hx Apply the 3x3 matrix H x’ = Hx Recall: Projective Transform H x3x3x3x3 x2x2x2x2 x1x1x1x1 x y x’ y’ (x,y) x (x’,y’) Homog. coords x’ = [x’,y’,1] T  2D image (x’,y’) 2D image (x,y)  Homog. coords [x,y,1] T = x Goal: Find H for ‘Image Rectification’ ‘World Plane’ ‘View Plane’

21. H matrix: Plane-to-Plane mapping What if H is unknown, but we have four or more pairs of x,x’ points? (see pg 15, Zisserman) x 1, x 2, x 3, x 4, x’ 1, x’ 2, x’ 3, x’ 4 … Projective Transform H (0,0,-1) ‘Image Plane’ (x,y,-1) (x,y,-1) (0,0,0) x x’ x1x1x2x2x3x3x1x1x2x2x3x3 h 11 h 12 h 13 h 21 h 22 h 21 h 31 h 32 h 33 = x’ 1 x’ 2 x’ 3 H x’ = x

22. Projective Transform: H Matrix H transforms Points:Matrix H transforms Points: –Given a point x: (x 1,x 2,x 3 ) (or in R 2 : (x 1 /x 3, x 2 /x 3 ) ) x’ = H x And matrix H -T transform Lines:And matrix H -T transform Lines: –Given a line l: (a,b,c) in R 2 : (a·x+ b·y + c = 0 ) in P 2 : (a·x 1 + b·x 2 + c·x 3 = 0 ) l’ = H -T l Recall point x is on line l iff x T l =0 Jargon: points transform ‘covariantly’; lines transform ‘contravariantly’

23. The bits and pieces of H The bits and pieces of H H has 8 independent variables (DOF) Cartesian (‘3D’) decomposition:Cartesian (‘3D’) decomposition: 2D Translation(t x, t y ) 3D Scale (s x,s y, s z ) 3D Rotation (  x,  y,  z ) 2D Translation(t x, t y ) 3D Scale (s x,s y, s z ) 3D Rotation (  x,  y,  z ) Computer Vision (Projective 2D) decomp.Computer Vision (Projective 2D) decomp. Isometry --3DOF (2D translate t x,t y ; 2D rotate  z ; ) Similarity --4DOF (add uniform scale s;) Affine --6DOF (add orientable scale s ,/s, s  /s) Projective--8DOF (changes x 3 ; 3D-rotation-like)

24. The bits and pieces of H The bits and pieces of H H has 8 independent variables (DOF) Computer Graphics method (3x3 matrix):Computer Graphics method (3x3 matrix): 2D Translation(t x, t y ) 3D Scale (s x,s y, s z ) 3D Rotation (  x,  y,  z ) 2D Translation(t x, t y ) 3D Scale (s x,s y, s z ) 3D Rotation (  x,  y,  z ) Computer Vision method(2D projective):Computer Vision method(2D projective): Isometry --3DOF (2D translate t x,t y ; 2Drotate  z ; ) Similarity --4DOF (add uniform scale s;) Affine --6DOF (add orientable scale s ,/s, s  /s) Projective--8DOF (changes x 3 ; 3D-rotation-like) Affects only x1,x2

25. The bits and pieces of H The bits and pieces of H H has 8 independent variables (DOF), not 9! Decomposable into 3 useful parts: (pg 22)Decomposable into 3 useful parts: (pg 22) H = H S H A H P = H = H S H A H P = Similarity H S :Similarity H S : 2D translate, rotate, uniform scale only (4DOF)2D translate, rotate, uniform scale only (4DOF) Affine H A :Affine H A : non-uniform scale (2DOF)non-uniform scale (2DOF) Projective H P :Projective H P : Projective coupling for x 3 (2DOF)Projective coupling for x 3 (2DOF) sRtsRt0T10T1sRtsRt0T10T1 I0I0vTvvTvI0I0vTvvTv K0K00T10T1K0K00T10T1            

26. H has 8 independent variables (DOF), not 9! Decomposable into 3 useful parts: (pg 22)Decomposable into 3 useful parts: (pg 22) H = H S H A H P = H = H S H A H P = Similarity H S :Similarity H S : 2D translate, rotate, uniform scale only (4DOF) s=scale,  =translate details…2D translate, rotate, uniform scale only (4DOF) s=scale,  =translate details… Affine H A :Affine H A : non-uniform scale (2DOF)non-uniform scale (2DOF) Projective H P :Projective H P : Projective coupling for x 3 (2DOF)Projective coupling for x 3 (2DOF) The bits and pieces of H The bits and pieces of H s  cos  -s sin  t x s sin  s  cos  t y 0 0 1 0 0 1 1 00 0 1 0 v 1 v 2 v S 1 k20 k3 10 0 01

27. Compare H A and H P (recall) H = H S H A H P = 2D world space 2D image space HAHAHAHA L M m HPHPHPHP m L L (parallelogram) (quadrilateral) (square) (K matrix, 2DOF) ( v vector) K  0   0 T  1 s R  t     0 T  1 I  0    v T  v

28. 1-D Projective Geometry (?Why? We’ll use it soon…) A ‘side’ view of 2D projective geometryA ‘side’ view of 2D projective geometry Convert R 1 scalar b to a 2-vector in P 1Convert R 1 scalar b to a 2-vector in P 1 As with P 2, we can transform points:As with P 2, we can transform points: –( use various H 2x2 s to change cyan lines below) b1 H 2a H 2b

29. 1-D Projective Geometry What is invariant here? all the positions, lengths and ratios change with each transform… Answer: ‘Cross Ratio’: cross(a,b,c,d) let a=, b= …let a=, b= … cross = |a-b||c-d| |a-c||b-d| cross(a,’b,’c,d’) = cross(a’,b’,c’,d’) cross(a,’b,’c,d’) = cross(a’,b’,c’,d’) abcd a1a1a2a2 a1a1a2a2 b1b1b2b2 b1b1b2b2 a’ c’ d’ b’

30. Image Rectifying: Undo parts of H x’ = H xwhere H = H S H A H P GOAL: Put world plane x’ into view plane –Affine Rect.; (find only H P (2DOF)); –Similarity Rect.;(find only H A H P (6DOF)); –Full Rect.;(find all of H (8DOF)); METHODS: (mix & match?) 1.Full: 4-point correspondence 2.‘Vanishing Point’, Infinity line methods 3.Other feature points, conics, …

31. Image Rectifying: Undo parts of H x’ = H xwhere H = H S H A H P GOAL: Put world plane x’ into view plane –Affine Rect.; (find only H P (2DOF)); –Similarity Rect.;(find only H A H P (6DOF)); –Full Rect.;(find all of H (8DOF)); METHODS: (mix & match?) 1.Full: 4-point correspondence 2.‘Vanishing Point’, Infinity line methods 3.Conics and circular points Other DOF? make assumptions, or ignore

32. Vanishing-Point Methods: 1 –In 2D image, find two pairs of lines that are parallel in ‘world’ space. world space 2D image H

33. Vanishing-Point Methods: 1 –Find intersection (vanishing points) p1, p2 (compute line intersections with 3D cross-products (see last lecture) –Horizon line I h connects p1, p2. world space 2D image p1 p2 H IhIhIhIh

34. Vanishing-Point Methods: 1 –H -T transforms world infinity line l   (0,0,1) T to I h – Answer (see old book) find, copy the l h coordinates! H p = world space 2D image p1 p2 H IhIhIhIh 1 0 0 0 1 0 l h1 l h2 l h3

35. Vanishing-Point Methods: 1 Limitations: –Accurate point, line-finding can be tricky –Can have high error sensitivity: ‘twitchy’ –Only rectifies H p -- but what if H A, H S needed? world space 2D image p1 p2 H IhIhIhIh

36. Vanishing-Point Methods: 2 No parallel lines? –find 2D image line with a known distance ratio –Use Cross-Ratio (in P 1 along that line) to –Find vanishing point distance at point d’ world space 2D image p1 p2 H IhIhIhIh abc a’ b’ c’ d’ d (at infinity)

37. Projective Transform: Hx = x’ x1x1x2x2x3x3x1x1x2x2x3x3 h 11 h 12 h 13 h 21 h 22 h 21 h 31 h 32 h 33 = x’ 1 x’ 2 x’ 3 Finding H from point pairs (correspondences) x’ = = y’ = = = xy1 Input (or output) image is on central plane x1x1x2x2x3x3x1x1x2x2x3x3 h 21 x + h 22 y + h 21 h 31 x + h 32 y + h 33 x’ 1 x’ 3 x’ 2 x’ 3 Write R 2 expressions; (for each x,x’ pair) h 11 x + h 12 y + h 13 h 31 x + h 32 y + h 33 x’ (h 31 x + h 32 y + h 33 ) = (h 11 x + h 12 y + h 13 ) y’ (h 31 x + h 32 y + h 33 ) = (h 11 x + h 12 y + h 13 ) Rearrange, solve as a matrix problem…

38. Projective Transform: Hx = x’ x1x1x2x2x3x3x1x1x2x2x3x3 h 11 h 12 h 13 h 21 h 22 h 21 h 31 h 32 h 33 = x’ 1 x’ 2 x’ 3 Finding H from point pairs (correspondences) x’ = = y’ = = = xy1 Input (or output) image is on central plane x1x1x2x2x3x3x1x1x2x2x3x3 h 21 x + h 22 y + h 21 h 31 x + h 32 y + h 33 x’ 1 x’ 3 x’ 2 x’ 3 Write R 2 expressions; (for each x,x’ pair) h 11 x + h 12 y + h 13 h 31 x + h 32 y + h 33 x’ (h 31 x + h 32 y + h 33 ) = (h 11 x + h 12 y + h 13 ) y’ (h 31 x + h 32 y + h 33 ) = (h 11 x + h 12 y + h 13 ) Rearrange, solve as a matrix problem… —!STOP! — Trucco Book Does This… Zisserman Book calls it “naïve”, gives better method: ‘DLT’: direct linear xfm.)

39. Projective Transform H x1x1x2x2x3x3x1x1x2x2x3x3 h 11 h 12 h 13 h 21 h 22 h 21 h 31 h 32 h 33 = x’ 1 x’ 2 x’ 3 H x’ = x Finding H from point pairs (correspondences) We know that Hx’ = x, andWe know that Hx’ = x, and we know at least 4 point pairs x’ and x that satisfy it:we know at least 4 point pairs x’ and x that satisfy it: ATTEMPT 1: ‘plug & chug’: make a matrix of x and x’ values…ATTEMPT 1: ‘plug & chug’: make a matrix of x and x’ values… H = x’ 1 x’ 2 x’ 3 x’ 4 x1x1x1x1 x2x2x2x2 x3x3x3x3 x4x4x4x4

40. END --Please download the starter code, compile and run. --Then you can get started on Project 1 right away.