Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Stanford CS223B Computer Vision, Winter 2007 Lecture 4 Camera Calibration Professors Sebastian Thrun and Jana Kosecka CAs: Vaibhav Vaish and David Stavens

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Today’s Goals Calibration: Problem definition Solution by nonlinear Least Squares Solution via Singular Value Decomposition Homogeneous Coordinates Distortion Calibration Software

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Camera Calibration Feature Extraction Perspective Equations

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Perspective Projection, Remember? fZ X O x

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Intrinsic Camera Parameters Determine the intrinsic parameters of a camera (with lens) What are Intrinsic Parameters? (can you name 7?)

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Intrinsic Parameters fZ X O

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Intrinsic Camera Parameters Intrinsic Parameters: –Focal Length f –Pixel size s x, s y –Image center o x, o y –(Nonlinear radial distortion coefficients k 1, k 2 …) Calibration = Determine the intrinsic parameters of a camera

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Why Intrinsic Parameters Matter

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Questions Can we determine the intrinsic parameters by exposing the camera to many known objects? If so, –How often do we have to see the object? –How many features on the object do we need?

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Example Calibration Pattern Calibration Pattern: Object with features of known size/geometry

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Harris Corner Detector (see Assignment 2)

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Why Tilt the Board?

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Experiment 1: Parallel Board

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter cm10cm20cm Projective Perspective of Parallel Board

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Experiment 2: Tilted Board

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter cm10cm20cm 500cm50cm100cm Projective Perspective of Tilted Board

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Intrinsics and Extrinsics Intrinsics: –Focal Length f –Pixel size s x, s y –Image center o x, o y Extrinsics: –Location and orientation of k-th calib. pattern:

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Perspective Camera Model Step 1: Transform into camera coordinates Step 2: Transform into image coordinates

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Perspective Camera Model Step 1: Transform into camera coordinates Step 2: Transform into image coordinates

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 The Full Perspective Camera Model

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 The Calibration Problem Given –Calibration pattern with N corners –K views of this calibration pattern Recover the intrinsic parameters –We’ll also recover the extrinsics, but we won’t care about them

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Calibration Questions Can we determine the intrinsic parameters by exposing the camera to many known objects? If so, –How often do we have to see the object? –How many features on the object do we need? –Do we need to see object at angle? Yes.

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Today’s Goals Calibration: Problem definition Solution by nonlinear Least Squares Solution via Singular Value Decomposition Homogeneous Coordinates Distortion Calibration Software

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Calibration constraints Step 1: Transform into camera coordinates Step 2: Transform into image coordinates

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Camera Calibration

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Calibration by nonlinear Least Squares Least Mean Square Gradient descent:

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 The Calibration Problem Quiz Given –Calibration pattern with N corners –K views of this calibration pattern How large would N and K have to be? Can we recover all intrinsic parameters?

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Intrinsic Parameters, Degeneracy fZ X O

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Summary Parameters, Revisited Extrinsic –Rotation –Translation Intrinsic –Focal length –Pixel size –Image center coordinates Focal length, in pixel units Aspect ratio

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 The Calibration Problem Quiz Given –Calibration pattern with N corners –K views of this calibration pattern How large would N and K have to be? Can we recover all intrinsic parameters? N K

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Constraints N points K images  2NK constraints 4 intrinsics (distortion: +2) 6K extrinsics  need 2NK ≥ 6K+4  (N-3)K ≥ 2 Hint: may not be co-linear

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 The Calibration Problem Quiz N K No Yes need (N-3)K ≥ 2 Hint: may not be co-linear

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Problem with Least Squares Many parameters (=slow) Many local minima! (=slower)

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Today’s Goals Calibration: Problem definition Solution by nonlinear Least Squares Solution via Singular Value Decomposition Homogeneous Coordinates Distortion Calibration Software

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Perspective Camera Model Step 1: Transform into camera coordinates Step 2: Transform into image coordinates

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Calibration Model (extrinsic) (Homogeneous Coordinates) (nonlinear perspective projection)

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Affine Problem Relaxation 

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Affine Problem Relaxation 

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Calibration via SVD [see Trucco/Verri]

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Calibration via SVD N>=7 points, not coplanar

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Calibration via SVD

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Calibration via SVD A has rank 7 (without proof)

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Calibration via SVD Remaining Problem: See book

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Summary, SVD Solution Replace rotation matrix by arbitrary matrix Transform into linear set of equations Solve via SVD Enforce rotation matrix (see book) Solve for remaining parameters (see book)

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Comparison Nonlinear least squares Gaussian image noise Many local minima Iterative Can incorporate non- linear distortion Singular Value Decomp. Gaussian parameter noise (algebraic) No local minima “Closed” form No distortion

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Today’s Goals Calibration: Problem definition Solution by nonlinear Least Squares Solution via Singular Value Decomposition Homogeneous Coordinates Distortion Calibration Software

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Homogeneous Coordinates Idea: In homogeneous coordinates most operations become linear! Extract Image Coordinates by Z- normalization

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Today’s Goals Calibration: Problem definition Solution by nonlinear Least Squares Solution via Singular Value Decomposition Homogeneous Coordinates Distortion Calibration Software

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Advanced Calibration: Nonlinear Distortions Barrel and Pincushion Tangential

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Barrel and Pincushion Distortion telewideangle

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Models of Radial Distortion distance from center

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Tangential Distortion cheap glue cheap CMOS chip cheap lens image cheap camera

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Image Rectification (to be continued)

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Distorted Camera Calibration Set k 1  k 2 , solve for undistorted case Find optimal k 1  k 2 via nonlinear least squares Iterate  Tends to generate good calibrations

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Today’s Goals Calibration: Problem definition Solution by nonlinear Least Squares Solution via Singular Value Decomposition Homogeneous Coordinates Distortion Calibration Software

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Calibration Software: Matlab

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Calibration Software: OpenCV

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007 Summary Calibration: Problem definition Solution by nonlinear Least Squares Solution via Singular Value Decomposition Homogeneous Coordinates Distortion Calibration Software