3D photography Marc Pollefeys Fall

3D Photography Obtaining 3D shape (and appearance) of real-world objects

Motivation Applications in many different areas A few examples …

Surgery - simulation simulate results of surgery allows preoperative planning

Surgery - teaching Capture models of surgery for interactive learning

Clothing Scan a person, custom-fit clothing

Forensics Crime scene recording and analysis

Forensics

3D urban modeling UNC/UKY UrbanScape project

Industrial inspection Verify specifications Compare measured model with CAD

Scanning industrial sites as-build 3D model of off-shore oil platform

Robot navigation small tethered rover pan/tilt stereo head ESA project our task: Calibration + Terrain modelling + Visualization

Architecture Survey Stability analysis Plan renovations

Architecture Survey Stability analysis Plan renovations

Cultural heritage Virtual Monticello Allow virtual visits

Cultural heritage Stanford’s Digital Michelangelo Digital archive Art historic studies

Archaeology accuracy ~1/500 from DV video (i.e. 140kb jpegs 576x720)

Record different excavation layers Archaeology Generate & verify construction hypothesis Layer 1 Layer 2 Generate 4D excavation record

Computer games

Course objectives To understand the concepts that allow to recover 3D shape from images Explore the state of the art in 3D photography Implement a 3D photography system/algorithm

Material Slides and more Also check out on-line “shape-from-video” tutorial: Other interesting stuff: Book by Hartley & Zisserman, Multiple View Geometry Peter Allen’s 3D photography class webpage

Learning approach Lectures: cover main 3D photography concepts and approaches. Exercice sessions: Assignments (1 st half semester) Paper presentations and project updates (2 nd half semester) 3D photography project: Choose topic, define scope (1 st half semster) Implement algorithm/system (2 nd half semester) Grade distribution Assignments & paper presentation: 50% Course project: 50%

Content camera model and calibration single-view metrology triangulation epipolar geometry, stereo and rectification structured-light, active techniques feature tracking and matching structure-from-motion shape-from-silhouettes space-carving …

3D photography course schedule (tentative) LectureExercise Sept 17Introduction- Sept 24Geometry & Camera modelCamera calibration Oct. 1Single View MetrologyMeasuring in images Oct. 8Feature Tracking/matching (Jan-Michael Frahm?) Correspondence computation Oct. 15Epipolar Geometry (David Gallup?) F-matrix computation Oct. 22Shape-from-SilhouettesVisual-hull computation Oct. 29Stereo matchingProject proposals Nov. 5Structured light and active range sensing (TBD) Papers Nov. 12Structure from motionPapers Nov. 19Multi-view geometry and self-calibration Papers Nov. 26Shape-from-XPapers Dec. 33D modeling and registrationPapers Dec. 10Appearance modeling and image-based rendering (TBD) Papers Dec. 17Final project presentations

Fast Forward! Quick overview of what is coming…

Camera models and geometry Pinhole camera Geometric transformations in 2D and 3D or

Camera Calibration Know 2D/3D correspondences, compute projection matrix also radial distortion (non-linear)

Single View Metrology

Antonio Criminisi

Feature tracking and matching Harris corners, KLT features, SIFT features key concepts: invariance of extraction, descriptors to viewpoint, exposure and illumination changes

l2l2 3D from images C1C1 m1m1 M? L1L1 m2m2 L2L2 M C2C2 Triangulation - calibration - correspondences

Epipolar Geometry Fundamental matrix Essential matrix Also how to robustly compute from images

Stereo and rectification Warp images to simplify epipolar geometry Compute correspondences for all pixels

Structured-light Projector = camera Use specific patterns to obtain correspondences

Initialize Motion (P 1,P 2 compatibel with F) Initialize Structure (minimize reprojection error) Extend motion (compute pose through matches seen in 2 or more previous views) Extend structure (Initialize new structure, refine existing structure) Structure from motion

** ** projection constraints Auto-calibration

Shape-from-Silhouette

Space-carving Only keep photo-consistent voxels

Shape-from-X Shape-from-focus Shape-from-texture time-of-flight

3D modeling and texturing Multiple depth imagesSurface model Texture integration patchwork texture map

Papers and discussion Will cover recent state of the art Each student will present a paper, discussion Select paper related to project

Course project: Build your own 3D scanner! Example: Bouguet ICCV’98 Students can work in group or alone

3D photography course team Marc Pollefeys CAB F66 Tel David Gallup CAB F64.2 Tel