1. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill 1 3D Photography (Image-based Model Acquisition) Funky Image Goes Here

2. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography2 “Analog” 3D photography ! “3D stereoscopic imaging” –been around as long as cameras have –Use camera with 2 or more lenses (or stereo attachment) –Use stereo viewer to create impression of 3D

3. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography3 Motivation Digitizing real world objects Getting realistic models humans objects places

4. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography4 3D Photography : Definition Sometimes called “3D Scanning” Use cameras and light to capture the shape & appearance of real objects Shape == geometry (point sampling + surface reconstruction + fairing) Appearance == surface attributes (color/texture, material properties, reflectance) Final result = richly detailed model

5. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography5 Applications in Industry Human body / head / face scans –Avatar creation for virtual worlds –3d conferencing –medical applications –product design –Platforms: Cyberware RD3030 Others (Geomagic, Metacreations, Cyrax, Geometrix…)

6. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography6 More applications Historical preservation, dissemination of museum artifacts (Digital Michelangelo, Monticello, …) CAD/CAM (eg. Legacy motorcycle parts scanned by Geomagic for Harley-Davidson). Marketing (models of products on the web) 3D games & simulation Reverse engineering

7. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography7 Technology Overview The Imaging Pipeline –Real World –Optics –Recorder –Digitizer –Vision & Graphics

8. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography8 Quick Notes on Optics Model lenses with all their properties - aberration, distortion, flare, vignetting etc. We correct for some of these effects (eg. distortion) in the calibration, ignore others. CCD (charged coupled devices) are the most popular recording media.

9. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography9 Theory : Passive Methods Stereo pair matching Structure from motion Shape from shading Photometric stereo

10. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography10 Stereo Matching Stereo Matching Basics –Needs two images, like stereoscopy –Given correspondence between points in 2 views, we can find depth by triangulation –But correspondence is hard prob! –A lot of literature on solving it… Stereo Matching output 3D point cloud Remove outliers and pass through surface reconstructor

11. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography11 Structure from Motion Camera moving, objects static Compute camera motion and object geometry from motion of image points Assumption - orthographic projn (use telephoto) If: world origin = 3D centroid camera origin = 2D centroid Then: camera translation drops out

12. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography12 Structure from Motion Camera moving, objects static Compute camera motion and object geometry from motion of image points

13. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography13 Structure from Motion Factorization [Tomasi & Kanade, 92] Find M, S using Singular Value Decomposition of W. SVD gives: S’  S modulo linear transform A. Solve for A using constraints on M.

14. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography14 More methods Shape from shading, [Horn] –Invert Lambert’s Law (L=I k cos  ) knowing the intensity at image point to solve for normal Photometric stereo [Woodham] –An extension of the above –Two or more images under different illumination conditions. –Each image provides one normal –Three images provide unique solution for a pixel.

15. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography15 Active Sensing Passive methods (eg. stereo matching) suffer from ambiguities - many similar regions in an image correspond to a point in the other. Project known / regular pattern (“structured light”) into scene to disambiguate get precise reconstruction by combining views –Laser rangefinder –Projectors and imperceptible structured light

16. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography16 Desktop 3D Photography Jean-Yves Bouguet, Pietro Perona An active sensing technique using “weak structured lighting” Need: camera, lamp, chessboard, pencil, stick Idea: –Light object with lamp & aim camera at it –Move stick around & capture shadow sequence –Use image of deformed shadow to calc 3D shape

17. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography17 Desktop 3D Photography Jean-Yves Bouguet, Pietro Perona Computation of 3d position from the plane of light source, stick and shadow

18. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography18 Volumetric Methods Chevette Project, Debevec, 1991

19. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography19 Voxel Models from Images When there are 2 colors in the image - use volume intersection [Szeliski 1993] –Back-project silhouettes from camera views & intersect

20. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography20 Voxel Models from Images With more colors but constrained viewpoints, we use voxel coloring [Seitz & Dyer, 1997] –Choose a voxel & project to it from all views –Color if enough matches –Prob - determining visibility of a point from a view –Solution - depth ordered traversal using a “view indep. d.o.” (dist from separating plane)

21. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography21 Voxel Models from Images A view-independent depth order may not exist (for some configuration of viewpoints / scene geometry). Use Space Carving [Kutulakos & Seitz, 1998] –Computes 3D (voxel) shape from multiple color photos –Computes “maximally photo-consistent shape” maximal superset of all 3D shapes that produce the given photos

22. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography22 Space Carving Algorithm: a) Initialize V to volume containing true scene b) For each voxel, check if photo-consistent if not, remove (“carve”) it. Can be shown to converge to maximal photo-consistent scene (union of all photo-consistent scenes).

23. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography23 Space Carving : Results House walkthru - 24 rendered input views Results best as seen from one of the original views

24. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography24 Modeling from a single view (Criminisi et al, 1999) Compute 3D affine measurements of the scene from single perspective image Use minimal geom info –vanishing line for a pencil of planes || to reference plane –vanishing point of parallel lines along a direction outside reference plane

25. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography25 Modeling from a single view (Criminisi et al, 1999) Compute “ratio of parallel distances” Creating a 3D model from a photograph –horizontal lines used to compute vanishing line –parallel vertical lines used to compute vanishing point Can generate geometrically correct model from a Renaissance painting (with correct perspective)

26. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography26 Extracting color, reflectance Photographs have lighting/shading effects that we estimate (reflectance function) and compensate for (specular highlight removal) or change (relighting) Work of Paul Debevec & others at Berkeley (acquiring reflectance field) Wood et al at U. Washington (surface light lield for 3D photography)

27. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography27 Surface Light Field [Wood et al, 2000] A 4D function on the surface - at surface parameter (u,v), for every direction ( ,  ), stores the color. Fixed illumination conditions. Photographs taken from a lot of different directions sample the surface light field. Continuous function (piecewise linear over ,  ) estimated by pointwise fairing.

28. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography28 Reflectance from Photographs (Yu, Debevec et al, 1999) Estimating reflectance for entire scenes –Too general a problem, parameterize thus: Assume surface can be divided into patches Diffuse reflectance function (albedo), varies across a patch Specular reflectance function taken as const across a region Assume known lighting, calib, geometry known Approach - Inverse Global Illumination –Estimate BRDF for direct illumination - f(u,v, ,  )

29. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography29 Reflectance from Photographs (Yu, Debevec et al, 1999) Inverse Global Illumination –Known Li (measure), Ii (calc fm known light sources) at every pixel –Estimate BRDF for direct illumination - f(u,v,  i,  i,  r,  r ) Write BRDF as a constant diffuse term and a specular term which is a function of incoming & outgoing  and roughness. Solve for the constants (  d,  s,  ) For indirect illumination - estimate the parameters (and indirect illumination coeffs with other patches) iteratively

30. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography30 Case study - Façade Debevec, Taylor & Malik, 1996 Modeling architectural scenes from photographs Not fully automatic (user inputs blocky 3D model) –Using blocks leads to fewer params in architectural models User marks corresponding features on photo Computer solves for block size, scale, camera rotation by minimizing error of corresponding features Reprojects textures from the photographs onto the reconstructed model

31. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography31 Modeling and Rendering Architecture from Photographs (Debevec, Taylor, and Malik 1996) Block Model User-Marked Edges Recovered Model

32. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography32 Arches and Surfaces of Revolution Taj Mahal modeled from one photograph

33. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography33 Case study - Digital Michelangelo Project 3D scanning of large statues (SIGGRAPH 00) Separate geometry and color scans –custom rig : laser scanner & camera mounted concurrently Range scan post-processing –Combine range scans from different positions Use volumetric modeling methods (Curless, Levoy 1996) –Fill holes using space carving

34. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography34 Case study - Digital Michelangelo Project Color scan processing –Compensate for ambient lighting subtract image with & without spotlight –Subtract out shadows & specularities –find surface orientation (inverse lighting computation) –convert color to RGB reflectance (acquire light field) Using estimated BRDF of marble modeling subsurface scattering

35. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography35 Digital Michelangelo Scanning a large object calibrated motions –pitch (yellow) –pan (blue) –horizontal translation (orange) uncalibrated motions –vertical translation –remounting the scan head –moving the entire gantry

36. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography36 References [Bouguet98] Bouguet, J.-Y., P. Perona. 3D Photography on your Desk. In Proc. ICCV 1998 [Bouguet00] Bouguet, J.-Y. Presentation on Desktop 3D Photography, in SIGGRAPH course notes on 3D Photography, 2000 [Criminisi99] Criminisi, A., I. Reid and A. Zisserman. Single View Metrology. In Proc. ICCV, pp 434-442, September 1999 [Curless96] Curless, B. and M. Levoy. A Volumetric Method for Building Complex Models from Range Images. In Proc. SIGGRAPH 1996 [Debevec96] Debevec, P., C. Taylor and J. Malik. Façade - Modeling and Rendering Architectural Scenes from Photographs. In Proc. SIGGRAPH 1996 [Debevec00a] Debevec, P. Presentation on the Façade, from SIGGRAPH course notes on 3D Photography, 1999, 2000. [Debevec00b] Debevec, P., T. Hawkins, C. Tchou, H.P.Duiker, W. Sarokin and M. Sagar. Acquiring the Reflectance Field of a Human Face. In Proc. SIGGRAPH 2000.

37. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography37 More References [Horn70] Horn, B.K.P. Shape from Shading : A Method for Obtaining the Shape of a Smooth Opaque Object from One View. Ph.D. Thesis, Dept of EE, MIT, 1970. [Kutulakos98] Kutulakos, K. N. and S. Seitz. A Theory of Shape by Space Carving. URCS TR#692, May 1998, appeared in Proc. ICCV 1999. [Levoy96] Levoy, M. and P. Hanrahan. Light Field Rendering. In Proc. SIGGRAPH 1996. [Levoy00a] Levoy, M., Pulli, K., Curless, B. et al. The Digital Michelangelo Project - 3D Scanning of Large Statues. In Proc. SIGGRAPH 2000. [Levoy00b] Levoy, M. Presentation on the Digital Michelangelo Project, in SIGGRAPH course notes on 3D Photography, 2000. [Seitz97] Seitz & Dyer. Photorealistic Scene Reconstruction by Voxel Coloring. In Proc. CVPR 1997, pp. 1067-1073.

38. 11/6/2000Deepak Bandyopadhyay / 258 / 3D Photography38 Still More References [Seitz00] Seitz, S. SIGGRAPH course notes on 3D photography, 1999, 2000. [Szeliski93] Szeliski, R. Rapid Octree Construction from Image Sequences. CGVIP : Image Understanding, vol. 58, no. 1, pp 23-32, 1993. [Wood00] Wood, D., D. I. Azuma, K. Aldinger, B. Curless, T. Duchamp, D.H. Salesin and W. Stuetzle. Surface Light Fields for 3D Photography. In Proc. SIGGRAPH 2000. [Woodham80] Woodham, R. Photometric Stereo for Determining Surface Orientation from Multiple Images. Journal of Optical Engineering, vol. 19, no. 1, pp 138-144, 1980. [Yu99] Yu, Y., P. Debevec, J. Malik and T. Hawkins. Inverse Global Illumination - Recovering Reflectance Models of Real Scenes from Photographs.