1. A Flexible New Technique for Camera Calibration Zhengyou Zhang Sung Huh CSPS 643 Individual Presentation 1 February 25, 2009 1

2. Outline Introduction Equations and Constraints Calibration and Procedure Experimental Results Conclusion 2

3. Outline Introduction Equations and Constraints Calibration and Procedure Experimental Results Conclusion 3

4. Introduction Extract metric information from 2D images Much work has been done by photogrammetry and computer vision community ◦ Photogrammetric calibration ◦ Self-calibration 4

5. Photogrammetric Calibration (Three-dimensional reference object-based calibration) Observing a calibration object with known geometry in 3D space Can be done very efficiently Calibration object usually consists of two or three planes orthogonal to each other ◦ A plane undergoing a precisely known translation is also used Expensive calibration apparatus and elaborate setup required 5

6. Self-Calibration Do not use any calibration object Moving camera in static scene The rigidity of the scene provides constraints on camera’s internal parameters Correspondences b/w images are sufficient to recover both internal and external parameters ◦ Allow to reconstruct 3D structure up to a similarity Very flexible, but not mature ◦ Cannot always obtain reliable results due to many parameters to estimate 6

7. Other Techniques Vanishing points for orthogonal directions Calibration from pure rotation 7

8. New Technique from Author Focused on a desktop vision system (DVS) Considered flexibility, robustness, and low cost Only require the camera to observe a planar pattern shown at a few (minimum 2) different orientations ◦ Pattern can be printed and attached on planer surface ◦ Either camera or planar pattern can be moved by hand More flexible and robust than traditional techniques ◦ Easy setup ◦ Anyone can make calibration pattern 8

9. Outline Introduction Equations and Constraints Calibration and Procedure Experimental Results Conclusion 9

10. Notation 2D point, 3D point, Augmented Vector, Relationship b/w 3D point M and image projection m 10 (1)(1)

11. Notation s : extrinsic parameters that relates the world coord. system to the camera coord. System A : Camera intrinsic matrix ( u 0, v 0 ): coordinates of the principal point α, β : scale factors in image u and v axes γ : parameter describing the skew of the two image 11

12. Homography b/w the Model Plane and Its Image Assume the model plane is on Z = 0 Denote i th column of the rotation matrix R by r i Relation b/w model point M and image m H is homography and defined up to a scale factor 12 (2)(2)

13. Constraints on Intrinsic Parameters Let H be H = [ h 1 h 2 h 3 ] Homography has 8 degrees of freedom & 6 extrinsic parameters Two basic constraints on intrinsic parameter 13 (3)(3) (4)(4)

14. Geometric Interpretation Model plane described in camera coordinate system Model plane intersects the plane at infinity at a line 14

15. Geometric Interpretation x ∞ is circular point and satisfy, or a 2 + b 2 = 0 Two intersection points This point is invariant to Euclidean transformation 15

16. Geometric Interpretation Projection of x ∞ in the image plane Point is on the image of the absolute conic, described by A -T A -1 Setting zero on both real and imaginary parts yield two intrinsic parameter constraints 16

17. Outline Introduction Equations and Constraints Calibration and Procedure Experimental Results Conclusion 17

18. Calibration Analytical solution Nonlinear optimization technique based on the maximum-likelihood criterion 18

19. Closed-Form Solution Define B = A -T A -1 ≡ B is defined by 6D vector b 19 (5)(5) (6)(6)

20. Closed-Form Solution i th column of H = h i Following relation hold 20 (7)(7)

21. Closed-Form Solution Two fundamental constraints, from homography, become If observed n images of model plane V is 2n x 6 matrix Solution of Vb = 0 is the eigenvector of V T V associated w/ smallest eigenvalue Therefore, we can estimate b 21 (8)(8) (9)(9)

22. Closed-Form Solution If n ≥ 3, unique solution b defined up to a scale factor If n = 2, impose skewless constraint γ = 0 If n = 1, can only solve two camera intrinsic parameters, α and β, assuming u 0 and v 0 are known and γ = 0 22

23. Closed-Form Solution Estimate B up to scale factor, B = λA T A -1 B is symmetric matrix defined by b B in terms of intrinsic parameter is known Intrinsic parameters are then 23

24. Closed-Form Solution Calculating extrinsic parameter from Homography H = [h 1 h 2 h 3 ] = λA[r 1 r 2 t] R = [r 1 r 2 r 3 ] does not, in general, satisfy properties of a rotation matrix because of noise in data R can be obtained through singular value decomposition 24

25. Maximum-Likelihood Estimation Given n images of model plane with m points on model plane Assumption ◦ Corrupted Image points by independent and identically distributed noise Minimizing following function yield maximum likelihood estimate 25 ( 10 )

26. Maximum-Likelihood Estimation is the projection of point M j in image i R is parameterized by a vector of three parameters ◦ Parallel to the rotation axis and magnitude is equal to the rotation angle R and r are related by the Rodrigues formula Nonlinear minimization problem solved with Levenberg-Marquardt Algorithm Require initial guess 26

27. Calibration Procedure 1. Print a pattern and attach to a planar surface 2. Take few images of the model plane under different orientations 3. Detect feature points in the images 4. Estimate five intrinsic parameters and all the extrinsic parameters using the closed-form solution 5. Refine all parameters by obtaining maximum- likelihood estimate 27

28. Outline Introduction Equations and Constraints Calibration and Procedure Experimental Results Conclusion 28

29. Experimental Results Off-the-shelf PULNiX CCD camera w/ 6mm lense 640 x 480 image resolution 5 images at close range (set A) 5 images at larger distance (set B) Applied calibration algorithm on set A, set B and Set A+B 29

30. Experimental Result Angle b/w image axes 30

31. Experimental Result http://research.microsoft.com/en-us/um/people/zhang/calib/ 31

32. Outline Introduction Equations and Constraints Calibration and Procedure Experimental Results Conclusion 32

33. Conclusion Technique only requires the camera to observe a planar pattern from different orientation Pattern could be anything, as long as the metric on the plane is known Good test result obtained from both computer simulation and real data Proposed technique gains considerable flexibility 33

34. Appendix Estimating Homography b/w the Model Plane and its Image Method based on a maximum-likelihood criterion (Other option available) Let M i and m i be the model and image point, respectively Assume m i is corrupted by Gaussian noise with mean 0 and covariance matrix Λ m i 34

35. Appendix Minimizing following function yield maximum-likelihood estimation of H where with = i th row of H 35

36. Appendix Assume for all i Problem become nonlinear least-squares one, i.e. Nonlinear minimization is conducted with Levenberg-Marquardt Algorithm that requires an initial guess with following procedure to obtain 36

37. Appendix Let Then (2) become n above equation with given n point and can be written in matrix equation as Lx = 0 L is 2n x 9 matrix x is define dup to a scale factor Solution of x L T L associated with the smallest eigenvalue 37

38. Appendix Elements of L ◦ Constant 1 ◦ Pixels ◦ World coordinates ◦ Multiplication of both 38

39. Possible Future Work Improving distortion parameter caused by lens distortion 39

40. Question? 40