Metric Self Calibration From Screw-Transform Manifolds Russell Manning and Charles Dyer University of Wisconsin -- Madison

Goal Find internal camera calibration Only input is images (views) Camera… –has fixed internal parameters –undergoes general motion

Input assorted views

Output Internal calibration (Optional) scene reconstruction reconstruction

Quick Overview Find pairwise fundamental matrices F i F1F1 F2F2

Don ’ t need an initial projective reconstruction Just need fundamental matrices, which are derived from view pairs So related to methods based on the Kruppa constraints rather than stratified calibration methods like the modulus constraint

Each fundamental matrix induces a curved surface (manifold) in R 5 F1F1 every manifold point is a legal internal calibration

There is a unique mutual intersection point to all these manifolds F4F4 F1F1 F2F2 F3F3 intersection point gives calibration

Details What is a manifold? What is a screw transform? Screw-transform manifolds and their coordinate system Finding intersection point: voting algorithm

Manifold Surface in R n that has a coordinate system

Manifold grid paper example kappa theta

Screw Transformation

AB A

AB

AB

AB

AB

AB

AB B

the scene camera A camera B (=camera A in arbitrary new position) screw transformation

Equivalent to Rising Turntable rising turntable: just like a normal turntable except can also translate (rise) parallel to its rotation axis *turntable is viewed by a single camera, fixed in position camera A scene rising turntable

Equivalent to Rising Turntable rising turntable: just like a normal turntable except can also translate (rise) parallel to its rotation axis *turntable is viewed by a single camera, fixed in position

Equivalent to Rising Turntable rising turntable: just like a normal turntable except can also translate (rise) parallel to its rotation axis *turntable is viewed by a single camera, fixed in position

Equivalent to Rising Turntable rising turntable: just like a normal turntable except can also translate (rise) parallel to its rotation axis *turntable is viewed by a single camera, fixed in position

Equivalent to Rising Turntable rising turntable: just like a normal turntable except can also translate (rise) parallel to its rotation axis camera B translated by gamma rotated by theta *turntable is viewed by a single camera, fixed in position

The Two Camera Matrices camera A... is located at position (1,0,0) is tilted by a rotation matrix R and has internal calibration K

The Two Camera Matrices camera B... is exactly like camera A, but... the world has first undergone a screw transform S

The Two Camera Matrices

rotation by theta around the z-axis (== screw axis) translation by gamma parallel to z-axis

The Two Camera Matrices so we can refer to the column vectors of the camera matrices

Fundamental Matrix screw rotation angle screw translation columns from camera matrices H = [ h 1 h 2 h 3 ] = K R

Fundamental Matrix through this representation, we will be able to find K from F by picking three real numbers: theta, gamma, and kappa ! these real numbers parameterize a “ screw-transform manifold ”

Screw Transform Manifold You give me 3 real numbers (kappa, theta, gamma) and a fundamental matrix I run the numbers through an algorithm (given in the paper) to get a legal internal calibration matrix (K) Set of legal K ’ s is the screw-transform manifold (kappa, theta, gamma) are the coordinate system Where does manifold live?

Screw Transform Manifold Manifold lives in R 5 because K is 3x3 upper-triangular matrix: xxx 0 0 x0 xx only 5 degrees of freedom since don’t care about scale (e.g., make norm 1)

Mutual Intersection Point Manifold is a set of points Each point is a legal internal calibration matrix Manifold corresponds to fundamental matrix The mutual intersection point of all the manifolds is a legal internal calibration matrix for every fundamental matrix

Voting Algorithm

Basic Idea “ sketch ” the manifolds within initial search region find approximate region where manifolds seem to intersect “ zoom in ” : repeat the process starting in the new, smaller search region

Manifold defined by function f(t) with one underlying real-valued parameter t In the limited search space shown at left, t might vary between 0 and 1 Pick t at random, say t=0.6, and find f(t) f(t) is shown as a dotted line because we don ’ t know it ’ s shape initially

Continue picking random values for t in range [0,1] to sketch manifold f(t) To utilize the randomly-selected points ( “ votes ” ), first drop a voxel grid over the search space Mark voxels that receive at least one vote

Sketch the second manifold in the same way

Now consider tabulating the votes For each voxel, keep track of how many manifolds have voted for that voxel lighter color means higher number of votes When some region receives enough votes, zoom in on that region

Now “ zoom in ” -- start over using the smaller search region as the original search region “ binary search ”

Repeat the sketching and voting procedure. A winner emerges. Determine the new “ zoom in ” region

Voxelize the new search space No consensus!

Properties of Voting Algorithm Benefits from using lots of fundamental matrices Noisy manifolds will contribute votes in empty regions and thus be ignored Zoom-in step eliminates bad manifolds Determines a small region where K is rather than an exact point Quick convergence at end when manifolds become linear

Properties of Voting Algorithm General technique -- use whenever need to intersect manifolds

begin

end

(1) What is self calibration? (2) What is a screw transform? (3) Fundamental matrix + choice of three screw transform parameters  internal calibration matrix. (4) Internal calibration matrix K lives in a 5D space (termed “K-space”). (5) Each fundamental matrix yields a 3D manifold in K-space, termed “screw-transform manifold.” (6) Intersection of 3 or more manifolds gives K matrix common to all view pairs (i.e., finds internal calibration).

Map of the Talk (1) A sample application of self calibration (2) The “ K ” matrix (internal calibration matrix). (3) Relationship between K and “ screw transform ” that underlies pairwise views. (5) The screw-transform manifold. (6) Intersecting screw-transform manifolds. (7) Algorithm for self calibration. (8) Voting algorithm to find intersection point. (4) What is a manifold?

(pictures of me taking photos of scene…leading to self calibration and scene reconstruction)

Map of the Talk (1) A sample application of self calibration (2) The “ K ” matrix (internal calibration matrix). (3) Relationship between K and “ screw transform ” that underlies pairwise views. (5) The screw-transform manifold. (6) Intersecting screw-transform manifolds. (7) Algorithm for self calibration. (8) Voting algorithm to find intersection point. (4) What is a manifold?

Camera Equation Consider the equation for how points in 3D get projected onto image plane of camera …

Camera Equation p == K [ R | t ] P position P in space… …gets translated by t and rotated by R… …then transformed to camera’s internal coordinate system by K p is projected position on image plane (in homogeneous coords)

Self Calibration K is an upper triangular, 3x3 matrix: xxx 0 0 x0 xx only 5 degrees of freedom since don’t care about scale (e.g., make norm 1) p == K [ R | t ] P Goal of self calibration is to find K up to a scale factor.

Map of the Talk (1) A sample application of self calibration (2) The “ K ” matrix (internal calibration matrix). (3) Relationship between K and “ screw transform ” that underlies pairwise views. (5) The screw-transform manifold. (6) Intersecting screw-transform manifolds. (7) Algorithm for self calibration. (8) Voting algorithm to find intersection point. (4) What is a manifold?

Screw Transformation arbitrary movement in space

Screw Transformation screw axis

Screw Transformation screw axis

Screw Transformation screw axis

screw rotation Screw Transformation screw axis

screw rotation Screw Transformation screw axis

screw rotation Screw Transformation screw axis

screw rotation Screw Transformation screw axis

screw rotation Screw Transformation screw axis

screw rotation Screw Transformation screw axis

Screw Transformation the scene camera A camera B (=camera A in arbitrary new position) screw transformation

Equivalent to Rising Turntable rising turntable: just like a normal turntable except can also translate (rise) parallel to its rotation axis *is viewed by a single camera, fixed in position camera A scene rising turntable

Equivalent to Rising Turntable rising turntable: just like a normal turntable except can also translate (rise) parallel to its rotation axis *is viewed by a single camera, fixed in position

Equivalent to Rising Turntable rising turntable: just like a normal turntable except can also translate (rise) parallel to its rotation axis *is viewed by a single camera, fixed in position

Equivalent to Rising Turntable rising turntable: just like a normal turntable except can also translate (rise) parallel to its rotation axis *is viewed by a single camera, fixed in position

Equivalent to Rising Turntable rising turntable: just like a normal turntable except can also translate (rise) parallel to its rotation axis *is viewed by a single camera, fixed in position camera B translated by gamma rotated by theta

The Two Camera Matrices camera A... is located at position (1,0,0) is tilted by a rotation matrix R and has internal calibration K

The Two Camera Matrices camera B... is exactly like camera A, but... the world has first undergone a screw transform S

The Two Camera Matrices

rotation by theta around the z-axis (== screw axis) translation by gamma parallel to z-axis

The Two Camera Matrices so we can refer to the column vectors of the camera matrices

Fundamental Matrix screw rotation angle screw translation columns from camera matrices H = [ h 1 h 2 h 3 ] = K R

Fundamental Matrix through this representation, we will be able to find K from F by picking three real numbers: theta, gamma, and kappa ! these real numbers parameterize a “ screw-transform manifold ”

Finding K from F F and kappa  h 3 h 3 and theta  h 1 and h 2 up to a scale factor gamma  h 1 and h 2 h 1, h 2, and h 3  K R = [ h 1 h 2 h 3 ] K R  K since (K R) (K R) T = K R R T K T = K K T, which gives K by Cholesky factorization

Map of the Talk (1) A sample application of self calibration (2) The “ K ” matrix (internal calibration matrix). (3) Relationship between K and “ screw transform ” that underlies pairwise views. (5) The screw-transform manifold. (6) Intersecting screw-transform manifolds. (7) Algorithm for self calibration. (8) Voting algorithm to find intersection point. (4) What is a manifold?

Manifolds Example 1: A one-dimensional manifold in 2D space

Manifolds A 1D manifold in the 2D space Another 1D manifold Manifold can be seen as a function f(t) that maps real numbers R into R 2 t “ parameterizes ” the manifold Part of a 2D space

Manifolds Manifolds are locally flat That is, locally a manifold behaves like its underlying parameter space (which is R n ) In this case, each manifold is locally like a line

Manifolds Example 2: A two-dimensional manifold in 3D space

Map of the Talk (1) A sample application of self calibration (2) The “ K ” matrix (internal calibration matrix). (3) Relationship between K and “ screw transform ” that underlies pairwise views. (5) The screw-transform manifold. (6) Intersecting screw-transform manifolds. (7) Algorithm for self calibration. (8) Voting algorithm to find intersection point. (4) What is a manifold?

Screw-Transform Manifold Using this formulation, it can be shown that once a fundamental matrix between two views is known, then internal calibration K is known up to three parameters (i.e., K has three degrees of freedom). Earlier it was shown how the fundamental matrix between two views relates to the parameters of the underlying screw transformation.

Screw-Transform Manifold The set of possible K matrices that are consistent with a given fundamental matrix is a 3D manifold in (5D) K-space. K-space == space of possible K matrices (of norm 1) theta = angle of rotation for the screw transform gamma = translation parallel to the screw axis kappa = determines location of screw axis The three underlying parameters:

Map of the Talk (1) A sample application of self calibration (2) The “ K ” matrix (internal calibration matrix). (3) Relationship between K and “ screw transform ” that underlies pairwise views. (5) The screw-transform manifold. (6) Intersecting screw-transform manifolds. (7) Algorithm for self calibration. (8) Voting algorithm to find intersection point. (4) What is a manifold?

Mutual Intersection Point Every point on screw-transform manifold i represents a K matrix that is consistent with fundamental matrix i. Thus a point that lies in the intersection of all the screw transform manifolds is a legal K that is consistent with every fundamental matrix. At least one such point exists (since the camera has an internal calibration matrix). Others have shown that, under the right conditions, there is a unique intersection point.

double intersection all manifolds intersect here for self calibration, we will want this point of mutual intersection

Map of the Talk (1) A sample application of self calibration (2) The “ K ” matrix (internal calibration matrix). (3) Relationship between K and “ screw transform ” that underlies pairwise views. (5) The screw-transform manifold. (6) Intersecting screw-transform manifolds. (7) Algorithm for self calibration. (8) Voting algorithm to find intersection point. (4) What is a manifold?

(1) Take several pictures of a scene.

(2) Find all pairwise fundamental matrices.

(3) Each fundamental matrix leads to a screw- transform manifold. F1F1 F2F2 F3F3

(4) Find K by intersecting the manifolds.

Map of the Talk (1) A sample application of self calibration (2) The “ K ” matrix (internal calibration matrix). (3) Relationship between K and “ screw transform ” that underlies pairwise views. (5) The screw-transform manifold. (6) Intersecting screw-transform manifolds. (7) Algorithm for self calibration. (8) Voting algorithm to find intersection point. (4) What is a manifold?

Basic Idea “ sketch ” the manifolds within initial search region find approximate region where manifolds seem to intersect “ zoom in ” : repeat the process starting in the new, smaller search region

Manifold defined by function f(t) with one underlying real-valued parameter t In the limited search space shown at left, t might vary between 0 and 1 Pick t at random, say t=0.6, and find f(t) f(t) is shown as a dotted line because we don ’ t know it ’ s shape initially

Continue picking random values for t in range [0,1] to sketch manifold f(t) To utilize the randomly-selected points ( “ votes ” ), first drop a voxel grid over the search space Mark voxels that receive at least one vote

Sketch the second manifold in the same way

Now consider tabulating the votes For each voxel, keep track of how many manifolds have voted for that voxel lighter color means higher number of votes When some region receives enough votes, zoom in on that region

Now “ zoom in ” -- start over using the smaller search region as the original search region “ binary search ”

Repeat the sketching and voting procedure. A winner emerges. Determine the new “ zoom in ” region

Voxelize the new search space No consensus!

Experimental Results

Performance

Concluding Remarks

Last Slide

Self Calibration

Voting Algorithm

Tell me again why we want to find the mutual intersection point?

Every point in K-space corresponds to a possible K matrix (where K is the internal calibration that we wish to find). Each fundamental matrix F induces a “screw-transform manifold” in K-space consisting of all possible K matrices that are consistent with F. Hence the mutual intersection point is consistent with all the fundamental matrices and must represent the internal calibration of the camera. 3 or more fundamental matrices are necessary for the manifold intersection to be a single point (or finite set of points).

More on K-space K-space is 5D; each STM is a 3D manifold in this space.

Screw Transformation (two photos of me moving camera to different positions)

Screw Transformation