1. © 2003 by Davi GeigerComputer Vision October 2003 L1.1 Structure-from-EgoMotion (based on notes from David Jacobs, CS-Maryland) Determining the 3-D structure of the world, and/or the motion of a camera using a sequence of images taken by a moving camera. –Equivalently, we can think of the world as moving and the camera as fixed. Like stereo, but the position of the camera isn’t known (and it’s more natural to use many images with little motion between them, not just two with a lot of motion). –We may or may not assume we know the parameters of the camera, such as its focal length.

2. © 2003 by Davi GeigerComputer Vision October 2003 L1.2 As with stereo, we can divide the problem: –Correspondence. –Reconstruction. We will focus on the reconstruction. –So we assume that each image contains some points, and we know which points match which. Structure-from-EgoMotion …

3. © 2003 by Davi GeigerComputer Vision October 2003 L1.3 Representation We’ll talk about a fixed camera, and moving object. We use scaled orthographic projection (weak perspective). –we remove the z coordinate and scale all x and y coordinates the same amount. Key point : Points Some matrix The image Then:

4. © 2003 by Davi GeigerComputer Vision October 2003 L1.4 Examples: S = [s, 0, 0, 0; 0, s, 0, 0]; This is just projection (3D to 2D), with scaling by s. S = [s, 0, 0, s*tx; 0, s, 0, s*ty]; This is 3D translation by (tx,ty,anything), projection, and scaling. S encodes: –Projection, Scaling,Translation (by tx and ty), and Rotation. –Let us now derive S.

5. © 2003 by Davi GeigerComputer Vision October 2003 L1.5 Rotation R can represent a 3D rotation of the points in P. What are the constraints on R?. First, look at 2D rotation (easier) RR T = Identity. R T is also a rotation matrix, in the opposite direction to R.

6. © 2003 by Davi GeigerComputer Vision October 2003 L1.6 Full 3D Rotation Any rotation can be expressed as combination of three rotations about three axes. Rows (and columns) of R are orthonormal vectors. R has determinant 1 (not -1). Rotation about z axis: Rotates x,y coordinates. Leaves z coordinates fixed.

7. © 2003 by Davi GeigerComputer Vision October 2003 L1.7 Intuitively, it makes sense that 3D rotations can be expressed as 3 separate rotations about fixed axes. Rotations have 3 degrees of freedom; two describe an axis of rotation, and one the amount. Rotations preserve the length of a vector, and the angle between two vectors. Therefore, (1,0,0), (0,1,0), (0,0,1) must be orthonormal after rotation. After rotation, they are the three columns of R. So these columns must be orthonormal vectors for R to be a rotation. Similarly, if they are orthonormal vectors (with determinant 1) R will have the effect of rotating (1,0,0), (0,1,0), (0,0,1). Same reasoning as 2D tells us all other points rotate too. Note if R has determinant -1, then R is a rotation plus a reflection.

8. © 2003 by Davi GeigerComputer Vision October 2003 L1.8 S: Putting it Together Scale Projection 3D Translation 3D Rotation We can just write st x as t x and st y as t y.

9. © 2003 by Davi GeigerComputer Vision October 2003 L1.9 Affine Structure from Motion

10. © 2003 by Davi GeigerComputer Vision October 2003 L1.10 Affine Structure-from-Motion: Two Frames (1) To simplify, suppose for the first four points:

11. © 2003 by Davi GeigerComputer Vision October 2003 L1.11 Affine Structure-from-Motion: Two Frames (2) Looking at the first four points, we get: We can solve for motion by inverting matrix of points. Or, explicitly, we see that first column on left (images of first point) give the translations. After solving for these, we can solve for the each column of the s components of the motion using the images of each point, in turn.

12. © 2003 by Davi GeigerComputer Vision October 2003 L1.12 Affine Structure-from-Motion: Two Frames (3) Once we know the motion, we can use the images of another point to solve for the structure. We have four linear equations, with three unknowns.

13. © 2003 by Davi GeigerComputer Vision October 2003 L1.13 Affine Structure-from-Motion: Two Frames (4) Suppose we just know where the k’th point is in image 1. Then, we can use the first two equations to write a k and b k as linear in c k. The final two equations lead to two linear equations in the missing values and c k. If we eliminate c k we get one linear equation in the missing values. This means the unknown point lies on a known line. That is, we recover the epipolar constraint. Furthermore, these lines are all parallel.

14. © 2003 by Davi GeigerComputer Vision October 2003 L1.14 Affine Structure-from-Motion: Two Frames (5) But, what if the first four points aren’t so simple? Then we define A, affine transformation, so that: This is always possible as long as the points aren’t coplanar. Then, given: Note that corresponds to translation of the points, plus a linear transformation.

15. © 2003 by Davi GeigerComputer Vision October 2003 L1.15 Affine Structure-from-Motion: Two Frames (6) We have: And: Then: is our motion. Thus, we can never determine the exact 3D structure of the scene. We can only determine it up to some transformation, A.

16. © 2003 by Davi GeigerComputer Vision October 2003 L1.16 Affine Structure-from-Motion: Many frames (1) ISP

17. © 2003 by Davi GeigerComputer Vision October 2003 L1.17 First Step: Solve for Translation (1) We pick the center of mass as origin, i.e., the average of all 3d points. It also averages noise locations. Rotation doesn’t move the origin, which is now the center of mass. Neither does scaled orthographic projection.

18. © 2003 by Davi GeigerComputer Vision October 2003 L1.18 First Step: Solve for Translation (2) Thus, translation can be eliminated.

19. © 2003 by Davi GeigerComputer Vision October 2003 L1.19 Rank Theorem has rank 3. This means there are 3 vectors such that every row of is a linear combination of these vectors. These vectors are the rows of P S P P SVD is made to do this. D is diagonal with non-increasing values, select the first/top three values, i.e., make D a 3 x 3 matrix. U and V have orthonormal rows, 2f x 3 and 3 x n respectively.

20. © 2003 by Davi GeigerComputer Vision October 2003 L1.20 Linear Ambiguity (as before) = U(:,1:3) * D(1:3,1:3) * V(1:3,:) = (U(:,1:3) * A) * (inv(A) *D(1:3,1:3) * V(1:3,:)) has full rank. Best solution is to estimate I that’s as near to as possible, with estimate of I having rank 3. Our current method does this. Noise

21. © 2003 by Davi GeigerComputer Vision October 2003 L1.21 Weak Perspective Motion S P Row 2k and 2k+1 of S should be orthogonal. All rows should be unit vectors. (Push all scale into P). =(U(:,1:3)*A)*(inv(A) *D(1:3,1:3)*V(1:3,:)) Choose A so (U(:,1:3) * A) satisfies these conditions.

22. © 2003 by Davi GeigerComputer Vision October 2003 L1.22 Related problems we won’t cover Missing data. Points with different, known noise. Multiple moving objects.

23. © 2003 by Davi GeigerComputer Vision October 2003 L1.23 Final Messages Structure-from-egomotion for points can be reduced to linear algebra. Epipolar constraint reemerges. SVD useful. Rank Theorem says the images a scene produces aren’t complicated (also important for recognition).