1. Spring 2015 CSc 83020: 3D Photography Prof. Ioannis Stamos Mondays 4:15 – 6:15 istamos@hunter.cuny.edu http://www.cs.hunter.cuny.edu/~ioannis

2. 3D Photography > complex / accurate representation of a 3D scene

3. Overview Create geometric and photometric 3D models Use Range and Image Sensing Fusing image data Comprehensive system with automation

4. Computer Vision Physical 3D World Illumination Vision System Scene Description Sensors: Digital Cameras (2D) Range Scanners (3D) GeometryReflectance

5. 3D Photography & Graphics Model of the Physical World Model of Illumination Scene Description GeometryReflectance Modeling [Representation of 3D objects] Rendering [Construction of 2D images from 3D models] Animation [Simulating changes over time] Captured from Vision System

6. 3D PHOTOGRAPHY EXAMPLE Ten scans were acquired of façade of Thomas Hunter Building (NYC) Registration details Automatic registration. Each scan has a different color.

7. 3D PHOTOGRAPHY EXAMPLE 24 scans were acquired of façade of Shepard Hall (City College of NY)

8. DataAcquisition Leica ScanStation 2 Spherical field of view. Registered color camera. Max Range: 300 meters. Scanning time: 2-3 times faster Accuracy: ~5mm per range point

9. Data Acquisition, Leica Scan Station 2, Park Avenue and 70 th Street, NY

10. Applications Virtual environment generation – Google Earth – acquire model for use in VRML, entertainment, etc – Realistic sets: movies and video games Reverse engineering – acquiring a model from a part copying/modification Part inspection – compare acquired model to “acceptable” model 3D FAX – transmit acquired model to remote RP machine Architectural site modeling Urban Planning Historical Preservation and Archaeology Reverse Engineering of Buildings

11. Data Acquisition Example Color Image of Thomas Hunter Building, New York City. Range Image. One million 3D points. Pseudocolor corresponds to laser intensity.

12. Traditional Pipeline Range Images Segmentation 3D feature extraction Partial Model Complete Model Range-Range Registration Brightness Images 2D feature extraction Range-Intensity Registration FINAL MODEL FINAL PHOTOREALISTIC MODEL

13. Other range sensors (some based on PrimeSense/Kinect) Matterport DotProduct Google Project Tango Prototype

14. Course Format Instructor will present recent topics and necessary background material Each student will present one or two papers Two projects Final grade: 50% : projects 30% : presentation 20% : class participation

15. Major Topics Acquiring images: 2D and 3D sensors (digital cameras and laser range scanners). 3D- and 2D-image registration. Geometry: – representation of 3D models, – simplification of 3D models, – detection of symmetry. Classification. Rendering 3D models. Image based rendering. Texture mapping.

16. Stereo Vision depth map

17. Segmentation

18. REGISTRATION pairwise & global

19. 3D Modeling (Mesh or volumetric)

20. Model simplification

21. Passive techniques: Stereo and Structure from Motion

22. 3D range to 2D image registration 3D scene 2D image

23. Texture mapped 3D model Corresponding 2D/proj. 3D lines 3D range to 2D image registration

24. TEXTURE MAP ANIMATION

25. The Façade Modeling System

26. Symmetry Detection

27. Image-Based Rendering Chen and Williams (1993) - view interpolation Chen and Williams (1993) - view interpolation McMillan and Bishop (1995) - plenoptic modeling McMillan and Bishop (1995) - plenoptic modeling Levoy and Hanrahan (1996) - light field rendering Levoy and Hanrahan (1996) - light field rendering Slide by Ravi Ramamoorthi, Columbia University

28. modelinganimationrendering 3D scanning motion capture image-based rendering The graphics pipeline the traditional pipeline the new pipeline? Slide courtesy Marc Levoy

29. Dynamic Scenes Image sequence (CMU, Virtualized Reality Project) http://www.ri.cmu.edu/projects/project_144.html

30. Dynamic Scenes Dynamic 3D model (CMU, Virtualized Reality Project) http://www.ri.cmu.edu/projects/project_144.html

31. Dynamic Scenes Dynamic texture-mapped model (CMU, Virtualized Reality Project) http://www.ri.cmu.edu/projects/project_144.html

32. Libraries Open Inventor Graphics Libraries Coin3D implements Open Inventor API: http://www.coin3d.org/ Online book: http://www-evasion.imag.fr/Membres/Francois.Faure/doc/inventorMentor/sgi_html/ Book: The Inventor Mentor : Programming Object- Oriented 3D Graphics with Open Inventor, Release 2

33. 33 Experimental Results Details of merged segments Merged points Surface meshes

34. Ball Pivoting Algorithm F. Bernardini, J. Mittleman, H. Rushmeier, C. Silva, G. Taubin. The ball-pivoting algorithm for surface reconstruction. IEEE Trans. on Vis. and Comp. Graph. 5 (4), pages 349-359, October-December 1999. A sequence of ball-pivoting operations. From left to right: A seed triangle is found; pivoting around an edge of the current front adds a new triangle to the mesh; after a number of pivoting operations, the active front closes on itself; a final ball-pivoting completes the mesh. Closely related to alpha-shapes, Edelsbrunner 94

35. Ball Pivoting Algorithm Results 3D mesh detail

36. RECOVERED CAMERA POSITIONS

37. GRAND CENTRAL (4)

38. GRAND CENTRAL (5)

39. Image-Based Rendering Image-Based Rendering – Chen and Williams (1993) - view interpolation – McMillan and Bishop (1995) - plenoptic modeling – Levoy and Hanrahan (1996) - light field rendering – Ng and Levoy (2005) – light field photography -> lytro camera – http://graphics.stanford.edu/projects/lightfield/ Slide by Ravi Ramamoorthi, Columbia University

40. 3D Modeling (people) Ioannis Stamos – CSCI 493.69 F08 Slide courtesy of Sebastian Thrun http://cs223b.stanford.edu Stanford

41. Head of Michelangelo’s David photograph1.0 mm computer model