Peter Sturm INRIA Grenoble – Rhône-Alpes (Institut National de Recherche en Informatique et Automatique) An overview of multi-view stereo and other topics in 3D modeling from 2D images Currently: sabbatical at TUM (Humboldt-fellow)

Overview Before about 1995 [Mohr et al.] Currently [Furukawa et al.]

Overview Before about 1995 [Mohr et al.] Currently [Furukawa et al.]

Overview Ingredients and factors of improvement: Geometry Image matching Statistics Better data Multi-view stereo Other approaches and related subjects: Other 3D modeling approaches Camera and object tracking Omnidirectional vision Sensor design... Numerical optimization Current research directions

Before about 1995 [Mohr et al.] 3D modeling from multiple images: image matching often done by hand main focus: given the matching, how to estimate camera positions, camera calibration, 3D point positions? 1990’s: golden age of multi-view geometry Exceptions, concerning manual matching: binocular stereo, aerial imagery close-range photogrammetry, mo-cap 3D modeling from silhouettes

1996 [Beardsley et al.] The first fully automatic approaches automatic matching systematic use of robust statistics going towards surface models (here, meshes)

1998 [Fitzgibbon et al.] Handling longer image sequences better numerical optimization methods system integration better data: image quality, calibration

Around 1998 Matchmoving applications (tracking of camera motion) in AR [Microsoft]

Around 1998 Matchmoving applications (tracking of camera motion) in AR [2d3] VIDEO…

At about the same time Polyhedral models from many georeferenced images [MIT City Scanning Project]

From 1998 onwards Going from “hacks” to mathematically founded surface representations and evolutions: level sets (now, increased usage of meshes) [Keriven et al.]

About 2003 [Strecha et al.] probabilistic occlusion handling Towards truly dense 3D models: point-based surface representations

About 2007 [Furukawa et al.] patch-based surface representation Current state of the art

About 2007 [Furukawa et al.] patch-based surface representation Current state of the art

Multi-View Stereo [Gargallo et al.] Optimization:

Quick Summary surface representation: point-based, patch-based, mesh, implicit surface, voxel sets,... Ingredients of multi-view stereo: quality measure: “photo-consistency”, invariance to lighting differences,... prior assumptions (regularization): smoothness, piecewise planarity, class-specific priors (e.g. faces) optimization approach, resolution strategy [Vetter et al.]

Overview Ingredients and factors of improvement: Geometry Image matching Statistics Better data Multi-view stereo... Numerical optimization Current research directions Other approaches and related subjects: Other 3D modeling approaches Camera and object tracking Omnidirectional vision Sensor design

[Criminisi et al.] Other approaches and subjects Interactive 3D modeling based on user annotations single-image 3D modeling

Other approaches and subjects Interactive 3D modeling based on user annotations multi-image 3D modeling [Debevec et al.]

Other approaches and subjects Exploitation of other depth cues than stereo shading focus / defocus

Active approaches using controlled illumination: Other approaches and subjects Photometric stereo Flash photography Structured light...

3D Modeling from silhouettes (“visual hull”) [Hernandez et al.] Other approaches and subjects Possible in real-time

Other approaches and subjects VIDEO…

Other approaches and subjects Real-time camera tracking: Model-based object tracking [Fua et al.]

Other approaches and subjects Real-time camera tracking: SLAM (simultaneous localization and mapping) Model-based object tracking [Lhuillier et al.] VIDEO…

Other approaches and subjects Omnidirectional vision: wide angle cameras surveillance navigation panoramic imaging

Other approaches and subjects Filters for HDR mosaics Mirror design [Nayar et al.] Some works on sensor design: Lens arrays

Other approaches and subjects Some works on sensor design: Filters for HDR mosaics Lens arrays, e.g. for computational photography [Georgiev et al.]

Overview Ingredients and factors of improvement: Geometry Image matching Statistics Better data Multi-view stereo Global optimization methods for geometry computations Use of community image collections Modeling surface appearance Modeling deformable / articulated objects... Numerical optimization Current research directions: Other approaches and related subjects

[Snavely et al.] Current research directions PhotoSynth / PhotoTourism Exploration of community image collections:

[Furukawa et al.] Current research directions Interior of buildings, cities Detailed 3D modeling of large scenes VIDEO…

Current research directions Shiny objects Modeling surface appearance: [Lensch et al.]

Current research directions Shiny objects Modeling surface appearance: [Birkbeck et al.]

Current research directions “The Light Stage” Modeling surface appearance: [Debevec et al.]

Current research directions Tanslucid, refractive, reflective surfaces, smoke, fire,... Modeling surface appearance:

Current research directions Modeling complex geometries: [Quan et al.] [Paris et al.]

Current research directions Modeling deformable / articulated objects: [Fua et al.]

Peter Sturm INRIA Grenoble – Rhône-Alpes (Institut National de Recherche en Informatique et Automatique) An overview of multi-view stereo and other topics in 3D modeling from 2D images Currently: sabbatical at TUM (Humboldt-fellow)