1. Camera diagram Computing K from 1 image HZ8.8 IAC and K HZ8.5 Camera matrix from F HZ9.5 IAC HZ3.5-3.7, 8.5 Extracting camera parameters HZ6.2 Camera matrix HZ6.1 Calibration using Q* HZ19.3 Hartley 92

2. Camera terminology a camera is defined by an optical center c and an image plane image plane (or focal plane) is at distance f from the camera center –f is called the focal length camera center = optical center principal axis = the line through camera center orthogonal to image plane principal point = intersection of principal axis with image plane –an indication of the camera center in the image principal plane = the plane through camera center parallel to image plane

3. Camera matrix Computing K from 1 image HZ8.8 IAC and K HZ8.5 Camera matrix from F HZ9.5 IAC HZ3.5-3.7, 8.5 Extracting camera parameters HZ6.2 Camera matrix HZ6.1 Calibration using Q* HZ19.3 Hartley 92

4. Encoding in a camera matrix the act of imaging is encoded by the camera matrix P, which is 3x4: –x = point in image = 3-vector –X = point in 3-space being imaged = 4-vector –PX = x in an early lecture, we saw that perspective projection is a linear operation in projective space, which allows this encoding we will develop the camera matrix in stages, generalizing as we go

5. Camera matrix 1 assumption set: 1.square pixels 2.origin of 3D world frame = camera center 3.z-axis of 3D world frame = principal axis 4.origin of 2D image space = principal point P = diag(f,f,1)[I 0] –(X,Y,Z,1)  (fX, fY, Z)

6. Camera matrix 2 (general image space) remove assumption #4: origin of 2D image space is arbitrary, so principal point is (px,py) Euclidean space: (X,Y,Z)  (fX/Z + px, fY/Z + py) projective space: (fX + Zpx, fY + Zpy, Z) matrix: P = K[I 0] where K = (f 0 px; 0 f py; 0 0 1)

7. Camera matrix 3 (general world frame) remove assumptions #2 and #3: origin and z-axis of the world frame are arbitrary, or equivalently, the world frame has no explicit connection to the camera freeing the origin from the camera center involves a translation C freeing the z-axis from the principal axis involves a rotation R let X be a point in world frame coordinates and X cam be the point in camera frame coordinates (with origin and z-axis aligned with camera): –Euclidean: X cam = R(X-C) –projective: X cam = [R –RC; 0 1] X so imaging process is x = PX cam = K[I 0] X cam = [K 0][R –RC; 0 1] X = [KR -KRC] X = KR[I –C] X x = KR[I –C]X or P = KR[I –C] HZ154-156 for last 3 slides

8. Camera matrix 4 (rectangular pixels) remove assumption #1: pixels may no longer be square cameras typically have non-square pixels –let mx = # of pixels / x-unit in image coordinates –let my = # of pixels / y-unit in image coordinates –e.g., mx = 1.333 and my = 1 if pixels are wider X camRect = diag (mx, my, 1) X camSquare = diag(mx,my,1) [R –RC; 0 1] X still write P = KR [I –C], but now the calibration matrix is –K = diag(mx,my,1) [f 0 px; 0 f py; 0 0 1] = [fm x 0 m x p x ; 0 fm y m y p y ; 0 0 1] –K is called the calibration matrix because it contains all of the internal camera parameters –we will be solving for K in the last stages as we solve for metric structure –K now has 10dof (encodes a CCD camera) –can also add a skew parameter s in k 12 entry, yielding a finite projective camera HZ156-7

9. Camera parameters Computing K from 1 image HZ8.8 IAC and K HZ8.5 Camera matrix from F HZ9.5 IAC HZ3.5-3.7, 8.5 Extracting camera parameters HZ6.2 Camera matrix HZ6.1 Calibration using Q* HZ19.3 Hartley 92

10. Extracting camera parameters from P what do we want to know about a camera? a camera is defined by its center (position) and image plane (part of orientation) can compute image plane from center, principal plane, and focal length –can also compute image plane from center, principal axis, and focal length all of these can be extracted from the camera matrix P also important to know orientation of the camera –that is, the rotation from the world frame to the camera frame –gives orientation of image rectangle therefore, want rotation matrix R (as in P = KR [I –C]) the principal point is extractable from the camera center and image plane, but we can also find it directly from the camera matrix, if so desired

11. Camera center from P let’s characterize finite cameras (with finite camera centers) Fact: camera matrices of finite cameras = 3x4 matrices with nonsingular 3x3 left submatrix (nonsingular KR) –easy to see that KR is nonsingular for finite focal lengths the camera center C (of a finite camera) is the null vector of camera matrix P –P is 3x4 with nonsingular 3x3 left submatrix, so has a 1D null space –proof: let C be null vector; all points on a line through C will project to the same point since P((1-t)C + tA) = PA; the only lines that map to a point are those through the camera center –another proof: the projection of C, PC, is undefined, which characterizes the camera center C = (-M -1 p 4 1) where p 4 = last column of P and M = left 3x3 submatrix –recall that C in KR[I –C] was the camera center (in Euclidean space) –we want to extract –C from P –let P = KR [I –C’]; if C = (C’ 1), then PC = 0 –p 4 = -KRC’ = -MC’ exercise: where do you think camera center of infinite camera is? infinite camera: center is defined by the null vector of M: C = (d, 0) where Md = 0 –M is singular in this case, so it does have a null vector –note that this is a point at infinity HZ157-9

12. Exercise before we talk about principal planes, we need a projective representation for the plane What do you think it is? Hint: it is the natural generalization of the projective representation of a line Take 2 minutes.

13. Principal plane from P a plane may be represented by ax+by+cz+d = 0; which is represented in projective space by the 4-vector (a,b,c,d) points on the principal plane are distinguished as the only ones that are mapped to infinity by the camera –that is, PX is an ideal point with x3=0 –that is, P 3T X = 0, where P 3T is the 3 rd row of P –that is, the principal plane is P 3T note that this plane does contain the camera center, as it should –PC = 0  P 3T C = 0 HZ160

14. Focal length and R from P finding focal length reduces to finding the calibration matrix K –focal length is encoded on the diagonal of K: (fmx fmy 1) –where (mx, my) is the pixel’s aspect ratio recall that K and R are embedded in the front of the camera matrix P –P = KR [I –C] we simply need to factor let M = KR = left 3x3 submatrix of P –K is upper triangular, R is orthogonal recall QR decomposition from numerical computing –A = QR where Q is orthogonal (e.g., product of Householders) and R is upper triangular –perfect, except wrong order RQ decomposition is analogous HZ157

15. RQ decomposition A = RQ, where R = upper triangular, Q = orthogonal we want to reduce A to upper triangular: A  R need to zero a21, a31 and a32 use 3 Givens rotations –A G1 G2 G3 = R –so A = R (G1 G2 G3)^t –Q = (G1 G2 G3)^t zero carefully to avoid contamination –zero a32 with x-rotation G1, then a31 with y-rotation G2, then a21 with z-rotation G3 –i.e., reverse order of A entries –Qy leaves 2 nd column alone (so a32 remains zero) –Qz creates new 1 st and 2 nd columns from linear combinations of old 1 st and 2 nd columns, so 3 rd row remains zeroed (cute!) Q represents a rotation, which can be encoded by roll/pitch/yaw angles associated with Gi are these 3 angles of roll/pitch/yaw (also called Euler angles) HZ579

16. Solving for Euler angle the first Givens rotation zeroes a32 let G1 = (1 0 0; 0 c –s; 0 s c) rotate about x a32 element of AG1 = c*a32 + s*a33 a32’ = 0  c:s = -a33 : a32 c = k(-a33); s = k(a32); c 2 + s 2 = 1 k2(a332 + a322) = 1  k = sqrt(a322 + a332) we now know G1 solve for G2 and G3 analogously

17. Exercise before we talk about principal axis, we need to recall a basic fact about the normal of a plane what is the normal of the plane ax+by+cz+d=0? why? take 2 minutes as a group

18. Principal axis from P the normal of a plane ax+by+cz+d = 0 is (a,b,c) –why? (X-P).N = 0 becomes X.N – P.N = 0 the principal axis is a normal of the principal plane recall: the principal plane is the third row of P (we called it P 3T ) Exercise: what is the normal of the principal plane? let M 3T = third row of M normal of the principal plane = M 3 so principal axis direction = M 3 which direction, M 3 or –M 3, points towards front of camera? answer: det(M) M 3 –det(M) protects it against sign changes principal axis = det(M) M 3

19. Principal point from P the principal point is less important, but let’s look at it anyway we could find the principal point by intersecting the line normal to the principal plane and through the camera center with the image plane –but this does not relate back directly to the camera matrix –let’s relate directly to P principal point = image of the point at infinity in the direction of the principal plane’s normal the point at infinity in this direction is (M 3 0) P (M 3 0) = MM 3 principal point = MM 3, where M 3T is the third row of M HZ160-161