1. © 2005 Yusuf Akgul Gebze Institute of Technology Department of Computer Engineering Computer Vision Geometric Camera Calibration

2. © 2005 Yusuf Akgul Geometric Camera Calibration Geometric camera calibration is the estimation of camera parameters (intrinsic, extrinsic, or both) Why do we need camera calibration? Most vision techniques need camera parameters but there are a few that don’t need the parameters.

3. © 2005 Yusuf Akgul Camera Calibration We need to link the known world coordinates with the corresponding image coordinates. To be able to know the world coordinates, we need an object whose measurements are known, which are called calibration patterns.

4. © 2005 Yusuf Akgul Camera Calibration Is it enough to solve just the above matrix to do the camera calibration?

5. © 2005 Yusuf Akgul Camera Calibration We will see two ways of calibration First method: Direct camera calibration estimates the intrinsic and extrinsic camera parameters directly. Second method: The projection matrix is estimated first and then the camera parameters. So which one should we prefer?

6. © 2005 Yusuf Akgul Direct Camera Calibration Extrinsic parameters to recover are R, 3x3 rotation matrix T, 3D translation vector.

7. © 2005 Yusuf Akgul Direct Camera Calibration

8. © 2005 Yusuf Akgul Direct Camera Calibration If we ignore radial distortions, then we have Focal length Pixel size in x and y Image center x and y 5 dependent parameters

9. © 2005 Yusuf Akgul Direct Camera calibration Intrinsic camera parameters

10. © 2005 Yusuf Akgul Direct Camera Calibration

11. © 2005 Yusuf Akgul Direct Camera Calibration Two main steps Assume the image center is known, estimate all the remaining parameters. We will assume (ox, oy) is (0,0) Estimate the image center later.

12. © 2005 Yusuf Akgul Direct Camera Calibration Denominators are the same! So, for each pair, we can write

13. © 2005 Yusuf Akgul Direct Camera Calibration If we write N equations like this one, we would have a homogenous linear system of equations of the form

14. © 2005 Yusuf Akgul Direct Camera Calibration How do we solve v?

15. © 2005 Yusuf Akgul Direct Camera Calibration Homogeneous System of N Linear Equations –Given N corresponding pairs {(Xi, Yi,, Zi) (xi, yi) }, i=1,2,…N –8 unknowns v = (v1,…,v8) T, 7 independent parameters The system has a nontrivial solution (up to a scale) –IF N >= 7 and N points are not coplanar => Rank (A) = 7 –Can be determined from the Singular Value Decomposition (SVD) of A

16. © 2005 Yusuf Akgul Singular Value Decomposition: –Any mxn matrix can be written as the product of three matrices n Singular values  i are fully determined by A n D is diagonal: d ij =0 if i  j; dii =  i (i=1,2,…,n) n  1   2  …   N  0 n Both U and V are not unique n Columns of each are mutual orthogonal vectors V1V1 U1U1

17. © 2005 Yusuf Akgul Singular Value Decomposition 1. Singularity and Condition Number –nxn A is nonsingular IFF all singular values are nonzero –Condition number : degree of singularity of A A is ill-conditioned if 1/C is comparable to the arithmetic precision of your machine; almost singular 2. Rank of a square matrix A –Rank (A) = number of nonzero singular values 3. Inverse of a square Matrix –If A is nonsingular –In general, the pseudo-inverse of A 4. Eigenvalues and Eigenvectors –Eigenvalues of both A T A and AA T are si 2 (si > 0) –The columns of U are the eigenvectors of AA T (mxm) –The columns of V are the eigenvectors of A T A (nxn)

18. © 2005 Yusuf Akgul Least Squares –Solve a system of m equations for n unknowns x(m >= n) –A is a mxn matrix of the coefficients –b (  0) is the m-D vector of the data –Solution: –How to solve: compute the pseudo-inverse of A T A by SVD (A T A) + is more likely to coincide with (A T A) -1 given m > n Always a good idea to look at the condition number of A T A nxn matrixPseudo-inverse Singular Value Decomposition

19. © 2005 Yusuf Akgul Homogeneous System –m equations for n unknowns x(m >= n-1) –Rank (A) = n-1 (by looking at the SVD of A) –A non-trivial solution (up to a arbitrary scale) by SVD: –Simply proportional to the eigenvector corresponding to the only zero eigenvalue of A T A (nxn matrix) Note: –All the other eigenvalues are positive because Rank (A)=n-1 –In practice, the eigenvector (i.e. v n ) corresponding to the minimum eigenvalue of A T A, i.e.  n 2 Singular Value Decomposition

20. © 2005 Yusuf Akgul Singular Value Decomposition Problem Statements –Numerical estimate of a matrix A whose entries are not independent –Errors introduced by noise alter the estimate to  Enforcing Constraints by SVD –Take orthogonal matrix A as an example –Find the closest matrix to Â, which satisfies the constraints exactly SVD of  Observation: D = I (all the singular values are 1) if A is orthogonal Solution: changing the singular values to those expected

21. © 2005 Yusuf Akgul Homogeneous System of N Linear Equations –Given N corresponding pairs {(Xi, Yi,, Zi) (xi, yi) }, i=1,2,…N –8 unknowns v = (v1,…,v8) T, 7 independent parameters The system has a nontrivial solution (up to a scale) –IF N >= 7 and N points are not coplanar => Rank (A) = 7 –Can be determined from the SVD of A –Rows of V T : eigenvectors {e i } of A T A –Solution: the 8 th row e 8 corresponding to the only zero singular value 8 =0 Practical Consideration –The errors in localizing image and world points may make the rank of A to be maximum (8) –In this case select the eigenvector corresponding to the smallest eigenvalue. Direct Camera Calibration

22. © 2005 Yusuf Akgul Equations for scale factor  and aspect ratio  How do we find the scale factor and alpha? How do we find other parameters? v 1 v 2 v 3 v 4 v 5 v 6 v 7 v 8 Direct Camera Calibration How do we find the scale factor and alpha?

23. © 2005 Yusuf Akgul Equations for scale factor  and aspect ratio  Knowledge: R is an orthogonal matrix Second row (i=2): First row (i=1) v 1 v 2 v 3 v 4 v 5 v 6 v 7 v 8 Direct Camera Calibration How do we find the scale factor and alpha?

24. © 2005 Yusuf Akgul Rotation R and Translation T Equations for first 2 rows of R and T gave us  and |  | First 2 rows of R and T can be found up to a common sign s (+ or -) The third row of the rotation matrix by vector product Remaining Questions : –How to find the sign s? –Is R orthogonal? –How to find Tz and fx, fy? Direct Camera Calibration

25. © 2005 Yusuf Akgul Find the sign s Facts: –fx > 0 –Zc >0 –x known –Xw,Yw,Zw known Solution  Check the sign of Xc  Should be opposite to x ZwZw XwXw YwYw Y X Z x y O PwPw P p x im y im (x im,y im ) Direct Camera Calibration

26. © 2005 Yusuf Akgul Rotation R : Orthogonality Question: –First 2 rows of R are calculated without using the mutual orthogonal constraint Solution: –Use SVD of estimate R Replace the diagonal matrix D with the 3x3 identity matrix Direct Camera Calibration

27. © 2005 Yusuf Akgul Direct Camera Calibration Find Tz, Fx and Fy Solution –Solve the system of N linear equations with two unknown Tx, fx –Least Square method –SVD method to find inverse a i2 a i1 bi

28. © 2005 Yusuf Akgul 1.Measure N 3D coordinates (Xi, Yi,Zi) 2.Locate their corresponding image points (xi,yi) - Edge, Corner, Hough 3.Build matrix A of a homogeneous system Av = 0 4.Compute SVD of A, solution v 5.Determine aspect ratio  and scale |  | 6.Recover the first two rows of R and the first two components of T up to a sign 7.Determine sign s of  by checking the projection equation 8.Compute the 3 rd row of R by vector product, and enforce orthogonality constraint by SVD 9.Solve Tz and fx using Least Square and SVD, then fy = fx /  YwYw XwXw ZwZw Direct Camera Calibration Algorithm

29. © 2005 Yusuf Akgul Estimating the Image Center Vanishing points: –Due to perspective projection, all parallel lines in 3D space appear to meet in a point on the image - the vanishing point, which is the common intersection of all the image lines VP1 VP2 Direct Camera Calibration

30. © 2005 Yusuf Akgul Direct Camera Calibration Estimating the Image Center Orthocenter Theorem: –Input: three mutually orthogonal sets of parallel lines in an image –T: a triangle on the image plane defined by the three vanishing points –Image center = orthocenter of triangle T –Orthocenter of a triangle is the common intersection of the three altitudes VP1 VP2 VP3 h3h3 h1h1 h1h1 (o x,o y )

31. © 2005 Yusuf Akgul VP1 VP2 VP3 h3h3 h1h1 h1h1 (o x,o y ) ? Direct Camera Calibration Estimating the Image Center Wrong aspect ratio would produce wrong image center.

32. © 2005 Yusuf Akgul Homework Read the Projection Matrix Based Camera Calibration In Your Book.