1. CSE 803 Fall 2008 Stockman1 Structure from Motion A moving camera/computer computes the 3D structure of the scene and its own motion

2. CSE 803 Fall 2008 Stockman2 Sensing 3D scene structure via a moving camera We now have two views over time/space compared to stereo which has multiple views at the same time.

3. CSE 803 Fall 2008 Stockman3 Assumptions for now The scene is rigid. The scene may move or the camera may move giving a sequence of 2 or more 2D images Corresponding 2D image points (Pi, Pj) are available across the images

4. CSE 803 Fall 2008 Stockman4 What can be computed The 3D coordinates of the scene points The motion of the camera Camera sees many frames of 2D points Rigid scene with many 3D interest points From Jabara, Azarbayejani, Pentland

5. CSE 803 Fall 2008 Stockman5 From 2D point correspondences, compute 3D points and TR

6. CSE 803 Fall 2008 Stockman6 applications We can compute a 3D model of a landmark from a video We can compute the trajectory of the sensor relative to the 3D object points

7. CSE 803 Fall 2008 Stockman7 Use only 2D correspondences, SfM can compute 3D jig pts … up to one scale factor.

8. CSE 803 Fall 2008 Stockman8 http://www1.cs.columbia.edu/~je bara/htmlpapers/SFM/sfm.html Jabara, Azarbayejani, Pentland a)Two video frames with corresponding 2D interest points. 3D points can be computed from SfM method. b)Some edges detected from 2D gradients. c)Texture mapping from 2D frames onto 3D polyhedral model. d)3D model can be viewed arbitrarily!

9. CSE 803 Fall 2008 Stockman9 Virtual museums Much work, and software, from about 10 years ago. 3D models, including shape and texture can be made of famous places (Notre Dame, Taj Mahal, Titanic, etc.) and made available to those who cannot travel to see the real landmark. Theoretically, only quality video is required. Usually, some handwork is needed.

10. CSE 803 Fall 2008 Stockman10 SfM methods Typically require careful mathematics EX: from 5 matched points, get 10 equations to estimate 10 unknowns; also a more popular 8 pt linear method Methods must consider effects of noise See Faugeras et al and ******** Methods can run in real time

11. CSE 803 Fall 2008 Stockman11 Special mathematics Epipolar geometry Fundamental matrix: computed from a pair of cameras Essential matrix: specialization of fundamental matrix when calibration is available

12. CSE 803 Fall 2008 Stockman12 Finishing the course: 2 options HW7 on stereo computation (Sat 6 Dec) (60%/7) Final Exam (25%): Monday 8 Dec 12:45-2:45 SfM report (6 Dec) and demonstration. Find existing code and get it to work on CSE 803 images (60%/7 + 10%) Final Exam (15%) limited problems