Paper presentation topics 2. More on feature detection and descriptors 3. Shape and Matching 4. Indexing and Retrieval 5. More on 3D reconstruction 1. Segmentation

Depth from disparity f xx’x’ baseline z CC’C’ X f input image (1 of 2) [Szeliski & Kang ‘95] depth map 3D rendering

Real-time stereo Used for robot navigation (and other tasks) Several software-based real-time stereo techniques have been developed (most based on simple discrete search) Nomad robot Nomad robot searches for meteorites in Antartica

Camera calibration errors Poor image resolution Occlusions Violations of brightness constancy (specular reflections) Large motions Low-contrast image regions Stereo reconstruction pipeline Steps Calibrate cameras Rectify images Compute disparity Estimate depth What will cause errors?

Spacetime Stereo Li Zhang, Noah Snavely, Brian Curless, Steven Seitz CVPR 2003, SIGGRAPH 2004

Stereo

? ? ?

Marker-based Face Capture The Polar Express, 2004 “The largest intractable problem with ‘The Polar Express’ is that the motion-capture technology used to create the human figures has resulted in a film filled with creepily unlifelike beings.” New York Times Review, Nov 2004

Stereo

Frame-by-Frame Stereo W  H = 15  15 Window A Pair of Videos 640  Each Inaccurate & Jittering

3D Surface Spacetime Stereo

Time 3D Surface

Spacetime Stereo Time 3D Surface

Spacetime Stereo 3D Surface Time

Spacetime Stereo Surface Motion Time

Spacetime Stereo Surface Motion Time=0

Spacetime Stereo Surface Motion Time=1

Spacetime Stereo Surface Motion Time=2

Spacetime Stereo Surface Motion Time=3

Spacetime Stereo Surface Motion Time=4

Surface Motion Matching Volumetric Window Affine Window Deformation Key ideas: Spacetime Stereo Time

Spacetime Stereo Time

Spacetime Stereo Time

Spacetime Stereo

A Pair of Videos 640  Each Spacetime Stereo W  H  T = 9  5  5 Window

Frame-by-Frame vs. Spacetime Stereo Spacetime Stereo W  H  T = 9  5  5 Window Frame-by-Frame W  H = 15  15 Window Spatially More Accurate Temporally More Stable

Video Projectors Color Cameras Black & White Cameras Spacetime Face Capture System

System in Action

Input Videos (640  480, 60fps)

Spacetime Stereo Reconstruction

Creating a Face Database

[Zhang et al. SIGGRAPH’04] …

Application 1: Expression Synthesis [Zhang et al. SIGGRAPH’04] … A New Expression:

Application 2: Facial Animation [Zhang et al. SIGGRAPH’04] …

Keyframe Animation

Some Applications Entertainment: Games & Movies Medical Practice: Prosthetics

Some books on linear algebra Linear Algebra, Serge Lang, 2004 Finite Dimensional Vector Spaces, Paul R. Halmos, 1947 Matrix Computation, Gene H. Golub, Charles F. Van Loan, 1996 Linear Algebra and its Applications, Gilbert Strang, 1988

Multiview Stereo

width of a pixel Choosing the stereo baseline What’s the optimal baseline? Too small: large depth error Too large: difficult search problem Large Baseline Small Baseline all of these points project to the same pair of pixels

The Effect of Baseline on Depth Estimation

1/z width of a pixel width of a pixel 1/z pixel matching score

Multibaseline Stereo Basic Approach Choose a reference view Use your favorite stereo algorithm BUT >replace two-view SSD with SSD over all baselines Limitations Must choose a reference view (bad) Visibility!

MSR Image based Reality Project …|…|