1. Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/4/19
Camera calibration
Digital Visual Effects, Spring 2006
Yung-Yu Chuang
2005/4/19
with slides by Richard Szeliski, Steve Seitz, and Marc Pollefyes

2. Announcements Artifacts for assignment #1 voting

3. Outline Camera projection models Camera calibration
Nonlinear least square methods
Bundle adjustment

4. Camera projection models

5. Pinhole camera

6. Pinhole camera model
origin
principal point
(optical
center)

7. Pinhole camera model

8. Pinhole camera model

9. Principal point offset
intrinsic matrix

10. Is this form of K good enough?
Intrinsic matrix
Is this form of K good enough?
non-square pixels (digital video)
skew
radial distortion

11. Radial distortion

12. Camera rotation and translation
extrinsic matrix

13. Two kinds of parameters
internal or intrinsic parameters such as focal length, optical center, aspect ratio: what kind of camera?
external or extrinsic (pose) parameters including rotation and translation: where is the camera?

14. Other projection models

15. Orthographic projection
Special case of perspective projection
Distance from the COP to the PP is infinite
Also called “parallel projection”: (x, y, z) → (x, y)
Image
World

16. Other types of projections
Scaled orthographic
Also called “weak perspective”
Affine projection
Also called “paraperspective”

17. Fun with perspective

18. Perspective cues

19. Perspective cues

20. Fun with perspective
Ames room

21. Forced perspective in LOTR
LOTR I Disc 4 -> Visual effects -> scale 0:00-10:00 or 15:00

22. Camera calibration

23. Camera calibration Estimate both intrinsic and extrinsic parameters
Mainly, two categories:
Photometric calibration: uses reference objects with known geometry
Self calibration: only assumes static scene, e.g. structure from motion

24. Camera calibration approaches
linear regression (least squares)
nonlinear optinization
multiple planar patterns

25. Chromaglyphs (HP research)

26. Linear regression

27. Linear regression
Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)

28. Linear regression

29. Linear regression

30. Linear regression
Solve for Projection Matrix M using least-square techniques

31. Normal equation Given an overdetermined system
the normal equation is that which minimizes the sum of the square differences between left and right sides
Why?

32. nxm, n equations, m variables
Normal equation
nxm, n equations, m variables

33. Normal equation

34. Normal equation →

35. Linear regression Advantages: Disadvantages:
All specifics of the camera summarized in one matrix
Can predict where any world point will map to in the image
Disadvantages:
Doesn’t tell us about particular parameters
Mixes up internal and external parameters
pose specific: move the camera and everything breaks

36. Nonlinear optimization
Feature measurement equations
Likelihood of M given {(ui,vi)}

37. Optimal estimation Log likelihood of M given {(ui,vi)}
How do we minimize C?
Non-linear regression (least squares), because ûi and vi are non-linear functions of M
We can use Levenberg-Marquardt method to minimize it

38. Multi-plane calibration
Images courtesy Jean-Yves Bouguet, Intel Corp.
Advantage
Only requires a plane
Don’t have to know positions/orientations
Good code available online!
Intel’s OpenCV library:
Matlab version by Jean-Yves Bouget:
Zhengyou Zhang’s web site:

39. Step 1: data acquisition

40. Step 2: specify corner order

41. Step 3: corner extraction

42. Step 3: corner extraction

43. Step 4: minimize projection error

44. Step 4: camera calibration

45. Step 4: camera calibration

46. Step 5: refinement

47. Nonlinear least square methods

48. Least square fitting
number of data points
number of parameters

49. Linear least square fittingy t
prediction
residual
is linear, too.

50. Nonlinear least square fitting

51. Function minimization
Least square is related to function minimization.
It is very hard to solve in general. Here, we only consider a simpler problem of finding local minimum.

52. Function minimization

53. Quadratic functions Approximate the function with
a quadratic function within
a small neighborhood

54. Quadratic functions A is positive definite. All eigenvalues
are positive.
Fall all x,
xTAx>0.
negative definite
A is singular
A is indefinite

55. Function minimization

56. Descent methods

57. Descent direction

58. Steepest descent method
the decrease of F(x) per
unit along h direction →
hsd is a descent direction because hTsd F’(x)=- F’(x)2<0
It has good performance in the initial stage of the iterative process. Converge very slow with a linear rate.

59. Steepest descent method
isocontour
gradient

60. Line search

61. Line search

62. Steepest descent method

63. Newton’s method → → → →
It has good performance in the final stage of the iterative process, where x is close to x*.

64. Hybrid method
This needs to calculate second-order derivative which might not be available.

65. Levenberg-Marquardt method
LM can be thought of as a combination of steepest descent and the Newton method. When the current solution is far from the correct one, the algorithm behaves like a steepest descent method: slow, but guaranteed to converge. When the current solution is close to the correct solution, it becomes a Newton’s method.

66. Nonlinear least square

67. Levenberg-Marquardt method

68. Levenberg-Marquardt method
μ=0 → Newton’s method
μ→∞ → steepest descent method
Strategy for choosing μ
Start with some small μ
If F is not reduced, keep trying larger μ until it does
If F is reduced, accept it and reduce μ for the next iteration

69. Bundle adjustment

70. Bundle adjustment
Bundle adjustment (BA) is a technique for simultaneously refining the 3D structure and camera parameters
It is capable of obtaining an optimal reconstruction under certain assumptions on image error models. For zero-mean Gaussian image errors, BA is the maximum likelihood estimator.

71. Bundle adjustment n 3D points are seen in m views
xij is the projection of the i-th point on image j
aj is the parameters for the j-th camera
bi is the parameters for the i-th point
BA attempts to minimize the projection error
predicted projection
Euclidean distance

72. Bundle adjustment

73. Bundle adjustment
3 views and 4 points

74. Typical Jacobian

75. Block structure of normal equation

76. Bundle adjustment

77. Bundle adjustment
Multiplied by

78. Reference
Manolis Lourakis and Antonis Argyros, The Design and Implementation of a Generic Sparse Bundle Adjustment Software Package Based on the Levenberg-Marquardt Algorithm, FORTH-ICS/TR
K. Madsen, H.B. Nielsen, O. Timgleff, Methods for Non-Linear Least Squares Problems, 2004.
Zhengyou Zhang, A Flexible New Techniques for Camera Calibration, MSR-TR-98-71, 1998.
Bill Triggs, Philip McLauchlan, Richard Hartley and Andrew Fitzgibbon, Bundle Adjustment - A Modern Symthesis, Proceedings of the International Workshop on Vision Algorithms: Theory and Practice, pp , 1999.