2 In today’s showHow positions in the image relate to 3D positions in the world?We will use analytical geometry to quantify more precisely the relationship between a camera, the objects it observes, and the pictures of these objectsWe start by briefly recalling elementary notions of analytical Euclidean geometry.We then introduce the various physical parameters that relate the world and camera coordinate frames, and present as an application various methods for estimating these parameters, a process known as geometric camera calibration.We also present along the way a linear least-squares technique for parameter estimation
3 MotivationHow positions in the image relate to 3D positions in the world?The reconstruction of 3D image is not trivial. We have to reconstruct the third coordinate!
4 Example Rabbit or Man? The information lays within the 3rd coordinate Markus Raetz, Metamorphose II,
5 2D projections are not the “same” as the real object as we usually see everyday!
7 IntroductionCamera calibration: estimation of the unknown values in a camera model.Intrinsic parameters - Link the frame coordinates of an image point with its corresponding camera coordinatesExtrinsic parameters - define the location and orientation of the camera coordinate system with respect to the world coordinate system
8 Euclidean Geometry - reminder Orthonormal coordinate frame (F) is defined bya point O in E3 and three unit vectors i, j and k orthogonal to each other
9 TransformationsFP - the coordinate vector of the point P in the frame FLet’s consider two frames : A and B(A) = (OA, iA, jA, kA)(B) =(OB, iB, jB, kB)How can we express BP as a function of AP?Let us suppose that the basis vectors of both coordinate systems are parallel to each other, i.e., iA = iB, jA = jB and kA = kB, but the origins OA and OB are distinctWe say that two coordinate systems are separated by a pure translation,
14 The inverse of a rotation matrix is equal to its transpose Its determinant is equal to 1 – the transform preserves the volume.Not every transformation that preserves the volume keeps the sign. For example - reflectionThis ortho normal transform preserves length and angles.
15 Example (Pure rotation) kA=kB=kThe vector iB is obtained by applying to the vector iA a counterclockwise rotation of angle θ about k.
17 As a single matrix equation: BPAPRotationTransformation
18 Homogenous coordinates Add an extra coordinate and use an equivalence relationFor 3D, equivalence relation k*(X,Y,Z,T) is the same as (X,Y,Z,T)Motivation – it will be possible to write the action of a perspective camera as a matrix
19 Homogenous/Non-Homogenous transformation for 3D point Non-homogenous to homogenous – add “1” as the 4th coordinate:Homogenous to non- homogenous – devide 1st 3 coordinates by the 4th
20 Homogenous/Non-Homogenous transformation for 2D point Non-homogenous to homogenous – add “1” as the 3rd coordinate:Homogenous to non- homogenous – devide 1st 2 coordinates by the 3rd
21 Camera calibrationUse the camera to tell you things about the worldRelationship between coordinates in the world and coordinates in the image : geometric calibration(We will not discuss here the relationship between intensities in the world and intensities in the image : photometric camera calibration.)
22 Three coordinate systems involved Camera: perspective projection.Image: intrinsic/internal camera parametersWorld: extrinsic/external camera parameters
23 The camera perspective equation The coordinates (x, y, z) of a scene point P observed by a pinhole camera are related to its image coordinates (x’, y’) by the perspective equationWe have by similar triangles (x,y,z)-> (f x/z, f y/z, -f ).Ignoring the third coordinate, we get (x,y,z)-> (f x/z, f y/z)PP’
24 Intrinsic parametersRelate the camera’s coordinate system to the idealized coordinate systemWe can associate with a camera two different image planes: the first one is a normalized plane located at a unit distance from the pinhole. We attach to this plane its own coordinate system with an origin located at the point where the optical axis pierces it. According to the perspective eq :Perspective projection:
26 Intrinsic parameters(cont) The second is the physical retina. It is located at a distance f ≠1 from the pinhole, and the image coordinates (u,v) are usually expressed in pixel units.Pixels are usually rectangular, so the camera has two additional scale parameters k and l, and:f is a distance in metersDefine :
27 Intrinsic parameters(cont) The actual origin of the camera coordinate system is at a corner C of the retina, and not at its center. It adds two parameters u0 and v0 that define the position (in pixel units) of C0 in the retinal coordinate system.
28 Intrinsic parameters(cont) The camera coordinate system may also be skewed, due to some manufacturing error, so the angle θ between the two image axes is not equal to 90°.
29 Intrinsic parameters(cont) Using homogenous coordinates:3x4 matrix
30 Intrinsic parameters(cont) The physical size of the pixels and the skew are always fixed for a given camera, and they can in principle be measured during manufacturing
31 Extrinsic parametersRelate the camera’s coordinate system to a fixed world coordinate system and specify its position and orientation in space.We consider the case where the camera frame (C) is distinct from the world frame (W).Non-homogenous coordinatesHomogenous coordinates
33 Combining extrinsic and intrinsic calibration parameters M can be defined with 11 free coefficients5 are intrinsic parameters – α,β,u0,v0,θ6 are extrinsic – the 3 angles defining R, 3 coordinates of t3 coords of t3 raws of RM is only defined up to scale in this setting!!
34 Rewriting the equation World coordinatesPixel coordinates
35 Z is in the camera coordinate system, but we can solve it, cause And we getRelation between image positions, u,v to points at 3D positions in P (homogenous coordinates)
36 Which features should we choose? Calibration methodsTechniques for estimating the intrinsic and extrinsic parameters of a cameraSuppose that a camera observes n geometric features such as points or lines with known positions in some fixed world coordinate system.We will:Compute the perspective projection matrix M associated with the camera in this coordinate systemCompute the intrinsic and extrinsic parameters of the camera from this matrixWhich features should we choose?
37 A linear approach to camera calibration For each feature point, i, we have :For n features, we will get 2n equations
38 A linear approach to camera calibration(cont) =0Pm
39 A linear approach to camera calibration(cont) When n>=6, the system is over-constrained, i.e. there is no non-zero vector m in R12 that satisfies exactly these equations.On the other hand, a zero vector is always a solution.According to the linear least-squares methods, we want to compute the value of the unit vector m that minimizes |Pm|2.In particular, estimating the vector m reduces to computing the eigenvectors and eigenvalues of the 12x12 matrix PTP
40 Linear least squares methods Let us consider a system of n equations, p unknowns:A is n x p matrix with coefficients aij, x = (x1,..,xp)TThere is no single solution if n≥p. The non trivial solution exists only if A is non singular.We will try to find vector x that minimize E (the error measure):
41 Linear least squares methods(cont) Need to impose a contraint on x, since x=0 yields the minimum.Since E(λx) = λ2E(x), we will use the contraint |x2|=1.E=xT(ATA)x , where ATA (pxp) matrix is positive smmetric matrixIt can be diagonalized in an orthonormal basis of eigenvectors ei (i=1,..,p) associated with eigenvalues 0≤ λ1 ≤ λ1 ≤ … ≤ λpwe can write x as x=μ1e1+… μnen, that (μ12 +…+ μp2 )=1 e1 minimizes the error E. It is the eigenvector associated with the minimum eigenvalue of ATA (λ1 ):
42 Recovering the intrinsic and extrinsic parameters Once we have the M matrix, we can recover the intrinsic and extrinsic parameters in a simple mathematical process, described in Forsyth&Ponce, section 6.3.1
43 Camera Calibration with a Single Image Sometimes more than one view of the same picture is used to estimate calibration parameters (For example, Stereo)Most Camera parameters can be estimated from the measurements of a single image when sufficient geometric object knowledge is available.The object knowledge used in the approach described here consists of parallelism and perpendicularity assumptions of straight object edges.In buildings parallel and perpendicular edges are usually abundant. Therefore, this method is often applicable for historic imagery of possibly demolished buildings taken with an unknown camera.
45 Projection of each point gives us two equations and there are 11 unknowns. 6 points in general position are sufficient for calibration.
46 The 6 anchor points clicked by the user are represented in green The 6 anchor points clicked by the user are represented in green. If the user had clicked more accurately, they should lie exactly at the corners of the small white squares. Using these 6 points and the corresponding 3D anchor points, the program computes an initial estimate of the projection matrix.
47 We take as input a set of at least 6 non-coplanar 3D anchor points, and their 2D images. The 2D coordinates do not need to be very accurate, they are typically obtained manually by a user who clicks their approximate position.
48 Summary We saw the goal of calibration We mentioned Euclidean Geometry We learned about internal/external camera parametersWe learned how to compute them from a given set of pointsWe saw an example of calibration by one picture onlyWe will see (stereo lecture) computing 3D coordinates from more than one picture (More than one view).