1. Camera models and single view geometry

2. Camera model

3. Camera: optical system d  21 thin lens small angles: Y Z  2 1 curvature radius

4. Y Z incident light beam deviated beam deviation angle ?   ’’  lens refraction index: n

5. Thin lens rules a) Y=0   = 0 f Y parallel rays converge onto a focal plane  b) f  = Y  beams through lens center: undeviated independent of y

6. r f Y h Where do all rays starting from a scene point P converge ? Z Fresnel law P Obs. For Z  ∞, r  f O p ?

7. d f a Z if d ≠ r … focussed image:  blurring circle) <image resolution depth of field: range [Z1, Z2] where image is focussed image plane P p O r  (blurring circle)=a (d-r)/r image of a point = blurring circle

8. the image of a point P belongs to the line (P,O) p P O p = image of P = image plane ∩ line(O,P) interpretation line of p: line(O,p) = locus of the scene points projecting onto image point p image plane r  f Hp: Z >> a

10. the image of a point P belongs to the line (P,O) p P O p = image of P = image plane ∩ line(O,P) interpretation line of p: line(O,p) = locus of the scene points projecting onto image point p image plane r  f Hp: Z >> a

11. p P O Z Y X c y x perspective projection f -nonlinear -not shape-preserving -not length-ratio preserving

12. Point [x,y] T expanded to [u,v,w] T Any two sets of points [u 1,v 1,w 1 ] T and [u 2,v 2,w 2 ] T represent the same point if one is multiple of the other [u,v,w] T  [x,y] with x=u/w, and y=v/w [u,v,0] T is the point at the infinite along direction (u,v) In 2D: add a third coordinate, w Homogeneous coordinates

13. Transformations translation by vector [ d x,d y ] T scaling (by different factors in x and y) rotation by angle 

14. Homogeneous coordinates In 3D: add a fourth coordinate, t Point [X,Y,Z] T expanded to [x,y,z,t] T Any two sets of points [x 1,y 1,z 1,t 1 ] T and [x 2,y 2,z 2,t 2 ] T represent the same point if one is multiple of the other [x,y,z,t] T  [X,Y,Z] with X=x/t, Y=y/t, and Z=z/t [x,y,z,0] T is the point at the infinite along direction (x,y,z)

15. Transformations scaling translation rotation Obs: rotation matrix is an orthogonal matrix i.e.: R -1 = RTRT

16. Pinhole camera model

17. with Scene->Image mapping: perspective transformation With “ad hoc” reference frames, for both image and scene

18. Let us recall them O Z Y X c y x f scene reference - centered on lens center - Z-axis orthogonal to image plane - X- and Y-axes opposite to image x- and y-axes image reference - centered on principal point - x- and y-axes parallel to the sensor rows and columns - Euclidean reference

19. O Z Y X c y x f scene reference - not attached to the camera image reference - centered on upper left corner - nonsquare pixels (aspect ratio)  noneuclidean reference Actual references are generic principal axis principal point

20. Principal point offset principal point

21. CCD camera

22. Scene-image relationship wrt actual reference frames image scene normally, s=0

23. K upper triangular : intrinsic camera parameters scene-camera tranformation extrinsic camera parameters orthogonal (3D rotation) matrix P: 10-11 degrees of freedom (10 if s=0)

24. i.e., defining x = [x, y, z] T with and

25. The locus of the points x whose image is u is a straight line through o having direction is independent of u o is the camera viewpoint (perspective projection center) line(o, d) = Interpretation line of image point u Interpretation of o: u is image of x if i.e., if

26. Intrinsic and extrinsic parameters from P M  K and R RQ-decomposition of a matrix: as the product between an orthogonal matrix and an upper triangular matrix M and m  t

27. Camera anatomy Camera center Column points Principal plane Axis plane Principal point Principal ray

28. Camera center null-space camera projection matrix For all A all points on AO project on image of A, therefore O is camera center Image of camera center is (0,0,0) T, i.e. undefined Finite cameras: Infinite cameras:

29. Column vectors Image points corresponding to X,Y,Z directions and origin

30. Row vectors Image of a point on the principal plane (the plane of the thin lens) is at the infinity  w = 0 is the principal plane

31. note: p 1,p 2 dependent on image reparametrization is the plane through the u-axis is the plane through the v-axis similarly,

32. The principal point principal point

33. The principal axis vector vector defining front side of camera the direction of the normal to the principal plane

34. Action of projective camera on point Forward projection Back-projection (pseudo-inverse)

35. Depth of points (dot product)(PO=0) If, then m 3 unit vector in positive direction

36. Camera matrix decomposition Finding the camera center (use SVD to find null-space) Finding the camera orientation and internal parameters (use RQ decomposition ~QR) Q R =( ) -1 = -1 -1 Q R (if only QR, invert)

37. When is skew non-zero? 1  arctan(1/s) for CCD/CMOS, always s=0 Image from image, s≠0 possible (non coinciding principal axes) resulting camera:

38. Euclidean vs. projective general projective interpretation Meaningfull decomposition in K,R,t requires Euclidean image and space Camera center is still valid in projective space Principal plane requires affine image and space Principal ray requires affine image and Euclidean space

39. Camera calibration

40. from scene-point to image point correspondence… …to projection matrix

41. Basic equations

42. Basic equations ctd. with(12x1) singular matrix

43. minimal solution over-determined solution  5½ correspondences needed (say 6) P has 11 dof, 2 independent eq./points n  6 points minimize subject to constraint p : eigenvector of A T A associated to its smallest eigenvalue

44. Degenerate configurations More complicate than 2D case (see Ch.21) (i)Camera and points on a twisted cubic (ii)Points lie on plane or single line passing through projection center

45. Data normalization (i)translate origin to gravity center (ii)(an)isotropic scaling

46. from line correspondences Extend DLT to lines (back-project image line) (2 independent eq.)

47. Geometric error

48. Gold Standard algorithm Objective Given n≥6 2D to 2D point correspondences {X i ↔x i ’}, determine the Maximum Likelyhood Estimation of P Algorithm (i)Linear solution: (a)Normalization: (b)DLT: (ii)Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error: (iii)Denormalization: ~~ ~

49. Calibration example (i)Canny edge detection (ii)Straight line fitting to the detected edges (iii)Intersecting the lines to obtain the images corners typically precision <1/10 (HZ rule of thumb: 5 n constraints for n unknowns

50. Exterior orientation Calibrated camera, position and orientation unkown  Pose estimation 6 dof  3 points minimal (4 solutions in general)

52. short and long focal length Radial distortion

56. Correction of distortion Choice of the distortion function and center Computing the parameters of the distortion function (i)Minimize with additional unknowns (ii)Straighten lines (iii)…

57. Properties of perspective transformations 1) vanishing points V image of the unproper point along direction d the interpretation line of V is parallel to d

58. O V d The images of parallel lines are concurrent lines

59. 2) cross ratio invariance Given four colinear points letbe their abscissae Properties of perspective transformations ctd.

60. Cross ratio invariance under perspective transformation a point on the line y=0=z its image belongs to a line its coordinate u

61. Object localization 1: three colinear points geometric model of an object a perspective image of the object position and orientation of the object ? A B C C’ A’ B’ O  calibrated camera: A B C  known  known interpretation lines

62. A B C C’ A’ B’ O  ∞ V a) orientation Cross ratio invariance: solve for V (image of ∞) V: vanishing point of the direction of (A,B,C) interpretation line of V parallel to (A,B,C)direction

63. b) position (e.g., distance(O,A)) A B C C’ A’ B’ O  V interpretation lines angles  and   

64. Object localization 2: four coplanar points O (i)orientation of (A,E,C) (ii)orientation of (B,E,D) (iii)distance (O,A) E A B C D

65. a’ b’ b” a” Find vanishing point of the field-bottom direction Off-side images of symmetric segments

66. a and b : abscissae of the endpoints of a segment c=(a+b)/2 : abscissa of segment midpoint, d=∞ : point at the infinite along the segment direction ac bd ( a’,b’ ) and ( a”, b” ) are image of symmetric segments  same image of the midpoint c’, same vanishing point d’ Harmonic 4-tuple ( a,b,c,d )

67.  solve { for c’, d’  system of two linear equations in ( c’d’ ) and ( c’+d’ )  two degree equation, whose solutions are c’ and d’ among the two solutions, the one for d’ is the value external to the range [a’,b’]

68. Action of projective camera on planes The most general transformation that can occur between a scene plane and an image plane under perspective imaging is a plane projective transformation

69. Action of projective camera on lines forward projection back-projection with Interpretation plane of line l

70. Image of a conic therefore

71. Action of projective camera on conics back-projection of a conic C to cone C

72. example: with Interpretation cone of a conic C back-projection of a conic C to cone

73. Images of smooth surfaces The contour generator  is the set of points X on S at which rays are tangent to the surface. The corresponding apparent contour  is the set of points x which are the image of X, i.e.  is the image of  The contour generator  depends only on position of projection center,  depends also on rest of P

74. Action of projective camera on quadrics apparent contour of a quadric Q dual quadric is a plane quadric: the set of planes tangent to Q Let us consider only those planes that are backprojection of image lines with its dual is

75. The plane containing the apparent contour  of a quadric Q from a camera center O follows from pole-polar relationship The cone with vertex V and tangent to the quadric Q is back-projection to cone  =QO

76. What does calibration give? An image line l defines a plane through the camera center with normal n=K T l measured in the camera’s Euclidean frame. In fact the backprojection of l is P T l=K T l

77. The image of the absolute conic mapping between  ∞ to an image is given by the planar homogaphy x=Hd, with H=KR absolute conic (IAC), represented by I 3 within  ∞ ( w=0 ) (i)IAC depends only on intrinsics (ii)angle between two rays (iii)DIAC=  * =KK T (iv)   K (Cholesky factorization) (v)image of circular points belong to  (image of absolute conic) its image (IAC)

78. A simple calibration device (i)compute H i for each square (corners  (0,0),(1,0),(0,1),(1,1)) (ii)compute the imaged circular points H i [1,±i,0] T (iii)fit a conic  to 6 imaged circular points (iv)compute K from  K -T K -1 through Cholesky factorization (= Zhang’s calibration method)

79. Orthogonality = pole-polar w.r.t. IAC

80. The calibrating conic

81. Vanishing points

82. Vanishing lines

84. Orthogonality relation

85. Calibration from vanishing points and lines