1. Paper presentation topics 2. More on feature detection and descriptors 3. Shape and Matching 4. Indexing and Retrieval 5. More on 3D reconstruction 1. Segmentation

2. Depth from disparity f xx’x’ baseline z CC’C’ X f input image (1 of 2) [Szeliski & Kang ‘95] depth map 3D rendering

3. Real-time stereo Used for robot navigation (and other tasks) Several software-based real-time stereo techniques have been developed (most based on simple discrete search) Nomad robot Nomad robot searches for meteorites in Antartica http://www.frc.ri.cmu.edu/projects/meteorobot/index.html

4. Camera calibration errors Poor image resolution Occlusions Violations of brightness constancy (specular reflections) Large motions Low-contrast image regions Stereo reconstruction pipeline Steps Calibrate cameras Rectify images Compute disparity Estimate depth What will cause errors?

5. Spacetime Stereo Li Zhang, Noah Snavely, Brian Curless, Steven Seitz CVPR 2003, SIGGRAPH 2004

6. Stereo

7. ? ? ?

8. Marker-based Face Capture The Polar Express, 2004 “The largest intractable problem with ‘The Polar Express’ is that the motion-capture technology used to create the human figures has resulted in a film filled with creepily unlifelike beings.” New York Times Review, Nov 2004

9. Stereo

10. Frame-by-Frame Stereo W  H = 15  15 Window A Pair of Videos 640  480@60fps Each Inaccurate & Jittering

11. 3D Surface Spacetime Stereo

12. Time 3D Surface

13. Spacetime Stereo Time 3D Surface

14. Spacetime Stereo 3D Surface Time

15. Spacetime Stereo Surface Motion Time

16. Spacetime Stereo Surface Motion Time=0

17. Spacetime Stereo Surface Motion Time=1

18. Spacetime Stereo Surface Motion Time=2

19. Spacetime Stereo Surface Motion Time=3

20. Spacetime Stereo Surface Motion Time=4

21. Surface Motion Matching Volumetric Window Affine Window Deformation Key ideas: Spacetime Stereo Time

22. Spacetime Stereo Time

23. Spacetime Stereo Time

24. Spacetime Stereo

25. A Pair of Videos 640  480@60fps Each Spacetime Stereo W  H  T = 9  5  5 Window

26. Frame-by-Frame vs. Spacetime Stereo Spacetime Stereo W  H  T = 9  5  5 Window Frame-by-Frame W  H = 15  15 Window Spatially More Accurate Temporally More Stable

27. Video Projectors Color Cameras Black & White Cameras Spacetime Face Capture System

28. System in Action

29. Input Videos (640  480, 60fps)

30. Spacetime Stereo Reconstruction

31. Creating a Face Database

32. [Zhang et al. SIGGRAPH’04] …

33. Application 1: Expression Synthesis [Zhang et al. SIGGRAPH’04] … A New Expression:

34. Application 2: Facial Animation [Zhang et al. SIGGRAPH’04] …

35. Keyframe Animation

36. Some Applications Entertainment: Games & Movies Medical Practice: Prosthetics

37. Some books on linear algebra Linear Algebra, Serge Lang, 2004 Finite Dimensional Vector Spaces, Paul R. Halmos, 1947 Matrix Computation, Gene H. Golub, Charles F. Van Loan, 1996 Linear Algebra and its Applications, Gilbert Strang, 1988

38. Multiview Stereo

39. width of a pixel Choosing the stereo baseline What’s the optimal baseline? Too small: large depth error Too large: difficult search problem Large Baseline Small Baseline all of these points project to the same pair of pixels

40. The Effect of Baseline on Depth Estimation

41. 1/z width of a pixel width of a pixel 1/z pixel matching score

43. Multibaseline Stereo Basic Approach Choose a reference view Use your favorite stereo algorithm BUT >replace two-view SSD with SSD over all baselines Limitations Must choose a reference view (bad) Visibility!

44. MSR Image based Reality Project http://research.microsoft.com/~larryz/videoviewinterpolation.htm …|…|