1. CSc 83020 3-D Computer Vision – Ioannis Stamos 3-D Computer Vision CSc 83020 Camera Calibration

2. CSc 83020 3-D Computer Vision – Ioannis Stamos Camera Calibration Problem: Estimate camera’s extrinsic & intrinsic parameters. Problem: Estimate camera’s extrinsic & intrinsic parameters. Method: Use image(s) of known scene. Method: Use image(s) of known scene. Tools: Tools: Geometric camera models. Geometric camera models. SVD and constrained least-squares. SVD and constrained least-squares. Line extraction methods. Line extraction methods.

3. CSc 83020 3-D Computer Vision – Ioannis Stamos Camera Calibration Feature Extraction Perspective Equations From Sebastian Thrun and Jana Kosecka

4. CSc 83020 3-D Computer Vision – Ioannis Stamos Perspective Projection, Remember? fZ X O x From Sebastian Thrun and Jana Kosecka

5. CSc 83020 3-D Computer Vision – Ioannis Stamos Intrinsic Camera Parameters Determine the intrinsic parameters of a camera (with lens) Determine the intrinsic parameters of a camera (with lens) What are Intrinsic Parameters? What are Intrinsic Parameters? (can you name 7?) From Sebastian Thrun and Jana Kosecka

6. CSc 83020 3-D Computer Vision – Ioannis Stamos Intrinsic Parameters fZ X O From Sebastian Thrun and Jana Kosecka Image center(ox, oy)

7. CSc 83020 3-D Computer Vision – Ioannis Stamos Intrinsic Camera Parameters Intrinsic Parameters: Intrinsic Parameters: Focal Length f Focal Length f Pixel size s x, s y Pixel size s x, s y Image center o x, o y Image center o x, o y (Nonlinear radial distortion coefficients k 1, k 2 …) (Nonlinear radial distortion coefficients k 1, k 2 …) Calibration = Determine the intrinsic parameters of a camera Calibration = Determine the intrinsic parameters of a camera From Sebastian Thrun and Jana Kosecka

8. CSc 83020 3-D Computer Vision – Ioannis Stamos Coordinate Frames Xw Yw Zw World Coordinate Frame Xc Yc Zc Camera Coordinate Frame Image Coordinate Frame Pixel Coordinates Intrinsic Parameters Extrinsic Parameters

9. CSc 83020 3-D Computer Vision – Ioannis Stamos Why Calibrate? Image Image point Calibration: relates points in the image to rays in the scene Scene ray

10. CSc 83020 3-D Computer Vision – Ioannis Stamos Why Calibrate? Calibration: relates points in the image to rays in the scene Image Image pointx y x y Scene ray z

11. CSc 83020 3-D Computer Vision – Ioannis Stamos Example Calibration Pattern Calibration Pattern: Object with features of known size/geometry From Sebastian Thrun and Jana Kosecka

12. CSc 83020 3-D Computer Vision – Ioannis Stamos Harris Corner Detector From Sebastian Thrun and Jana Kosecka

13. Perspective Camera (X,Y,Z) (x,y,z) Center of Projection r r’ r =(x,y,z) r’=(X,Y,Z) r/f=r’/Z f: effective focal length: distance of image plane from O. x=f * X/Z y=f * Y/Z z=f

14. Extrinsic Parameters Pc=R(Pw-T) Translation followed by rotation T

15. Extrinsic Parameters (2 nd formulation) Pc=R Pw +T Rotation followed by translation R same as before T different

16. CSc 83020 3-D Computer Vision – Ioannis Stamos The Rotation Matrix R * R = R * R = I => R = R Orthonormal Matrix Degrees of freedom? 1 0 0 I=0 1 0 0 0 1 TT T

17. CSc 83020 3-D Computer Vision – Ioannis Stamos Perspective Camera Model Step 1: Transform into camera coordinates Step 1: Transform into camera coordinates Step 2: Transform into image coordinates Step 2: Transform into image coordinates From Sebastian Thrun and Jana Kosecka

18. CSc 83020 3-D Computer Vision – Ioannis Stamos Intrinsic Parameters

19. CSc 83020 3-D Computer Vision – Ioannis Stamos Image and Camera Frames Ximage Yimage Xcamera Ycamera (xim, yim) (ox,oy) Zcamera

20. CSc 83020 3-D Computer Vision – Ioannis Stamos Geometric Model Transformation from World to Camera Frame. Perspective projection (f, R, T) Point in Camera Frame Transformation from Image to Camera Frame. (ox,oy,sx,sy) No distortion! 3D Point in Camera Coordinate Frame x= y= XcT ZcT YcT ZcT

21. CSc 83020 3-D Computer Vision – Ioannis Stamos Camera Calibration: Issues Which parameters need to be estimated. Which parameters need to be estimated. Focal length, image center, aspect ratio Focal length, image center, aspect ratio Radial distortions Radial distortions What kind of accuracy is needed. What kind of accuracy is needed. Application dependent Application dependent What kind of calibration object is used. What kind of calibration object is used. One plane, many planes One plane, many planes Complicated three dimensional object Complicated three dimensional object

22. CSc 83020 3-D Computer Vision – Ioannis Stamos Camera Calibration Calibration objectExtracted features

23. CSc 83020 3-D Computer Vision – Ioannis Stamos Camera Calibration Extract centers of circles

24. CSc 83020 3-D Computer Vision – Ioannis Stamos Basic Equations

25. CSc 83020 3-D Computer Vision – Ioannis Stamos Basic Equations In the remaining slides x means x im and y means y im

26. CSc 83020 3-D Computer Vision – Ioannis Stamos Basic Equations

27. CSc 83020 3-D Computer Vision – Ioannis Stamos Basic Equations Extrinsic Parameters 1) Rotation matrix R (3x3) 2) Translation vector T (3x1) Intrinsic Parameters 1)fx=f/sx, length in effective horizontal pixel size units. 2)α=sy/sx, aspect ratio. 3)(ox,oy), image center coordinates. 4)Radial distortion coefficients. Total number of parameters (excluding distortion): ?

28. CSc 83020 3-D Computer Vision – Ioannis Stamos Basic Equations 1)Assume that image center is known. 2)Solve for the remaining parameters. 3)Use N image points and their corresponding N world points

29. CSc 83020 3-D Computer Vision – Ioannis Stamos Basic Equations 1)Assume that image center is known. 2)Solve for the remaining parameters. 3)Use N image points and their corresponding N world points (1) Here x means x im - o x and y means y im - o y

30. CSc 83020 3-D Computer Vision – Ioannis Stamos Basic Equations 1)Assume that image center is known. 2)Solve for the remaining parameters. 3)Use N image points and their corresponding N world points (2)

31. CSc 83020 3-D Computer Vision – Ioannis Stamos Basic Equations (3)

32. CSc 83020 3-D Computer Vision – Ioannis Stamos Basic Equations (3) How would we solve this system?

33. CSc 83020 3-D Computer Vision – Ioannis Stamos Basic Equations (3) How would we solve this system? Rank of matrix A? Solution up to a scale factor.

34. CSc 83020 3-D Computer Vision – Ioannis Stamos Singular Value Decomposition Appendix A.6 (Trucco) A: m x n U: m x m, columns orthogonal unit vectors. V: n x n, -//- D: m x n, diagonal. The diagonal elements σi are the singular values σ1>= σ2>= … >= σn >= 0

35. CSc 83020 3-D Computer Vision – Ioannis Stamos Singular Value Decomposition Appendix A.6 (Trucco) 1.Square A non-singular iff σi != 0 2.For square A C=σ1/σN is the condition number 3.For rectangular A # of non-zero σi is the rank 4.For square non-singular A: 5.For square A, pseudoinverse: 6.Singular values of A = square roots of eigenvalues of and 7. Columns of U, V Eigenvectors of 8. Frobenius norm of a matrix

36. CSc 83020 3-D Computer Vision – Ioannis Stamos Singular Value Decomposition Appendix A.6 (Trucco) If rank(A)=n-1 (7 in our case) then the solution is the eigenvector which corresponds to the ONLY zero eigenvalue. Solution up to a scale factor.

37. Solving for v (3) How would we solve this system: SVD. Solution: Uknown scale factor γ=? Aspect ratio α=?

38. CSc 83020 3-D Computer Vision – Ioannis Stamos Solving for Tz and fx?

39. CSc 83020 3-D Computer Vision – Ioannis Stamos Solving for Tz and fx? How would we solve this system?

40. CSc 83020 3-D Computer Vision – Ioannis Stamos Solving for Tz and fx? How would we solve this system? Solution in the least squares sense.

41. Camera Center

42. CSc 83020 3-D Computer Vision – Ioannis Stamos Camera Models (linear versions) Transformation from World to Camera Frame. Perspective projection (f, R, T) Point in Camera Frame Transformation from Image to Camera Frame. (ox,oy,sx,sy) No distortion! 3D Point in World Coordinate Frame x= y=

43. Camera Models (linear versions) World Point (Xw, Yw,Zw) Measured Pixel (xim, yim) Elegant decomposition. No distortion! ? Homogeneous Coordinates

44. CSc 83020 3-D Computer Vision – Ioannis Stamos Camera Calibration – Other method Extracted features Step 1: Estimate P Step 2: Decompose P into internal and external parameters R,T,C

45. Camera Calibration: Step 1 Extracted features Each point (x,y) gives us two equations

46. Camera Calibration: Step 1 Extracted features Each corner (x,y) gives us two equations

47. CSc 83020 3-D Computer Vision – Ioannis Stamos Camera Calibration: Step 1 Extracted features 2n n points gives us 2n equations

48. CSc 83020 3-D Computer Vision – Ioannis Stamos Camera Calibration: Step 1 Extracted features 2n We need to solve In the presence of noise we need to solve The solution is given by the eigenvector with the smallest eigenvalue of

49. CSc 83020 3-D Computer Vision – Ioannis Stamos Camera Calibration: Step 1 Extracted features The result can be improved through non-linear minimization.

50. CSc 83020 3-D Computer Vision – Ioannis Stamos Camera Calibration: Step 1 Extracted features The result can be improved through non-linear minimization. Minimize the distance between the predicted and detected features.