1. Geometric Models & Camera Calibration
Reading: Chap. 2 & 3

2. Introduction
We have seen that a camera captures both geometric and photometric information of a 3D scene.
Geometric: shape, e.g. lines, angles, curves
Photometric: color, intensity
What is the geometric and photometric relationship between a 3D scene and its 2D image?
We will understand these in terms of models.

3. Models Models are approximations of reality.
“Reality” is often too complex, or computationally intractable, to handle.
Examples:
Newton’s Laws of Motion vs. Einstein’s Theory of Relativity
Light: waves or particles?
No model is perfect.
Need to understand its strengths & limitations.

4. Camera Projection Models
3D scenes project to 2D images
Most common model: pinhole camera model
From this we may derive several types of projections:
orthographic
weak-perspective
para-perspective
perspective
least accurate
most accurate

5. Pinhole Camera Model real image plane virtual image plane object Z Y f
optical centre
object Z Y f f
image is inverted
image is not inverted

6. Real vs. Virtual Image Plane
The image is physically formed on the real image plane (retina).
The image is vertically and laterally inverted.
We can imagine a virtual image plane at a distance of f in front of the camera optical center, where f is the focal length.
The image on this virtual image plane is not inverted.
This is actually more convenient.
Henceforth, when we say “image plane” we will mean the virtual image plane.

7. Perspective Projectionf u

8. Perspective Projection
The imaging process is a many-to-one mapping
all points on the 3D ray map to a single image point.
Therefore, depth information is lost
From similar triangles, can write down the perspective equation:

9. Perspective Projection
This assumes coordinate axes is at the pinhole.

10. t : World origin w.r.t. camera
3D Scene point P
Camera coordinate
system
Rotation R
t : World origin w.r.t. camera
coordinate axes
World coordinate
system
P : 3D position of scene point
w.r.t world coordinate axes
R : Rotation matrix to align world
coordinate axes to camera axes

11. Perspective Projection
α, β : scaling in image u, v axes, respectively
Θ : skew angle, that is, angle between u, v axes
u0, v0 : origin offset
Note: α=kf, β=lf where k, l are the magnification factors

12. Intrinsic vs. Extrinsic parameters
Intrinsic: internal camera parameters.
6 of them (5 if you don’t care about focal length)
Focal length, horiz. & vert. magnification, horiz. & vert. offset, skew angle
Extrinsic: external parameters
6 of them
3 rotation angles, 3 translation param
Imposing assumptions will reduce # params.
Estimating params is called camera calibration.

13. Representing Rotations
Euler angles
pitch: rotation about x axis :
yaw: rotation about y axis:
roll: rotation about z axis: X Y Z

14. Rotation Matrix Two properties of rotation matrix:
R is orthogonal: RTR = I
det(R) = 1

15. Orthographic Projection
Projection rays are parallel
Camera
centre
optical
axis
Image plane
Parallel lines

16. Orthographic Projection Equations
: first two rows of R
: first two elements of t
This projection has 5 degrees of freedom.

17. Weak-perspective Projection
Image plane
Parallel lines

18. Weak-perspective Projection
This projection has 7 degrees of freedom.

19. Para-perspective Projection
Parallel lines
Image plane

20. Para-perspective Projection
Camera matrix given in textbook Table 2.1, page 36.
Note: Written a bit differently.
This has 9 d.o.f.

21. Real cameras Real cameras use lenses
Lens distortion causes distortion in image
Lines may project into curves
See Chap 1.2, 3.3 for details
Change of focal length (zooming) scales the image
not true if assuming orthographic projection)
Color distortions too

22. Color Aberration Bad White Balance
Unlike human vision, cameras can not adapt their spectral responses to cope with different lighting conditions; as a result, the acquired image may have an undesirable shift in the entire color range. Digital cameras have an function called white balancing. If improperly white balanced, the color and tone of the whole picture will be wrong.
Over-exposure at the time.

23. Color Aberration Purple Fringing
When camera lens is wide open and there's high contrast in the scene, what normally should be a white line could appear purple. That is called "purple fringing" or a "halo" effect. The purple fringe around the edges is mainly caused by color interpolation combined with chromatic aberration.
There are little patches of light purple discoloration all around the background and a heavy purple fringing artifact appeared along the leaning branch.

24. Color Aberration Vignetting
we can notice that the corners of the print are getting darker. Darkening of an image around the edges and corners is called “vignetting”. It’s mainly caused by two reasons: One is the design of the camera or lens itself. All lenses falloff toward the corners, and the physical lens barrier would prevent some light from entering the lens. Therefore, the central part of an image would be brighter than that of the parts at the corner or along the edges. The other cause may be light hitting the film or CCD sensor surface at an oblique angle. It’s more often occurring when using wide-angle lenses. In flash photography a similar effect can be caused by the coverage of the flash being insufficient to illuminate the field of view.

25. Geometric Camera Calibration

26. Recap A general projective matrix:
This has 11 d.o.f. , i.e. 11 parameters
5 intrinsic, 6 extrinsic
How to estimate all of these?
Geometric camera calibration

27. Key Idea Capture image of a known 3D object (calibration object).
Establish correspondences between 3D points and their 2D image projections.
Estimate M
Estimate K, R, and t from M
Estimation can be done using linear or non-linear methods.
We will study linear methods first.

28. Setting things up Calibration object: 3D object, or
2D planar object captured at different locations

29. Setting it up Suppose we have N point correspondences:
P1, P2, … PN are 3D scene points
u1, u2, … uN are corresponding image points
Let mT1, mT2, mT3 be the 3 rows of M
Let (ui, vi) be the (non-homogeneous) coords of the i th image point.
Let Pi be the homogeneous coords of i th scene point.

30. Math For the i th point Each point gives 2 equations.
Using all N points gives 2N equations:

31. Math A m = 0 A : 2N x 12 matrix, rank is 11 m : 12 x 1 column vector
Note that m has only 11 d.o.f.

32. Math Solution? m is in the Nullspace of A ! Use SVD: A = U ∑ VT
m = last column of V
m is only up to an unknown scale
In this case, SVD solution makes || m || = 1

33. Estimating K, R, t Now that we have M (3x4 matrix)
c is the 3D coords of camera center wrt World axes
Let ĉ be homogeneous coords of c
It can be shown that M ĉ = 0
So ĉ is in the Nullspace of M
And t is computed from –Rc, once R is known.

34. Estimating K, R, t B, the left 3x3 submatrix of M is KR
Perform “RQ” decomposition on B.
The “Q” is the required rotation matrix R
And the upper triangular “R” is our K matrix !
Impose the condition that diagonals of K are positive
We are done!

35. RQ factorization Any n x n matrix A can be factored into A = RQ
Where both R, Q are n x n
R is upper triangular, Q is orthogonal
Not the same as QR factorization
Trick: post multiply A by Givens rotation matrices: Qx ,Qy, Qz

36. RQ factorization c, s chosen to make a particular entry of A zero
For example, to make a21 = 0, we solve
Choose Qx ,Qy, Qz such that

37. Degeneracy Caution: The 3D scene points should not all lie on a plane.
Otherwise no solution
In practice, choose N >= 6 points in “general position”
Note: the above assumes no lens distortion.
See Chap 3.3 for how to deal with lens distortion.

38. Summary Presented pinhole camera model
From this we get hierarchy of projection types
Perspective, Para-perspective, Weak-perspective, Orthographic
Showed how to calibrate camera to estimate intrinsic + extrinsic parameters.