1. Procam and Campro Shree K. Nayar Computer Science Columbia University Support: NSF, ONR Procams 2006 PROCAMS Shree K. Nayar,

2. Convergence of Cameras and Projectors Cameras for Projectors (Campro): Display better or new images (Geometry, Shadows, Radiometry, Focus, Multi-Projectors) P C Projectors & Cameras for Apps (ProCamApp): Enhance existing or create new application (Immersion, Augmentation, Navigation, Surgery) P C Projectors for Cameras (Procam): Capture better or new images (Quality, Matting, Relighting, Refocusing) P C PROCAMS Shree K. Nayar,

3. Brief History of the Flash Gas Flame Ibbetson 1840 Flashbulb Chauffour 1893 Electric Flashbub Vierkotter 1925 Commercial Flashbub Ostermeier 1930 Digital Camera Flash 2006 Flash/No Flash Multi-Flash Flash Matting PROCAMS Shree K. Nayar,

4. Projector as a Programmable Flash LightingCapture 2D 4D PROCAMS Shree K. Nayar,

5. Projector as a Programmable Flash Depth from Projector Defocus Image-Complete Depth Maps (with Li Zhang, Siggraph 2006) Separation of Direct and Global Illumination Scattering Properties of Real Materials (with Krishnan, Grossberg, Raskar, Siggraph 2006) PROCAMS Shree K. Nayar,

6. Courtesy of Koesbong @ flickr.com Camera Defocus PROCAMS Shree K. Nayar,

7. Focal Plane Surface Camera Depth from Camera Defocus ( Pentland 87, Subbarao 88, Darrell and Wohn 88,Nayar and Nakagawa 89, Ens and Lawrence 91, Bove 93, Nayar et al. 95, Watanabe et al. 98 ) Camera Defocus: Captured PROCAMS Shree K. Nayar,

8. Focal Plane Lamp Projector Projector Defocus PROCAMS Shree K. Nayar,

9. Focal Plane Surface Projector Lamp Projector Defocus PROCAMS Shree K. Nayar,

10. Focal Plane Surface Projector Lamp Projector Defocus Projector Defocus: Projected PROCAMS Shree K. Nayar,

11. Camera Defocus: Captured Camera Defocus vs. Projector Defocus Projector Defocus: Projected PROCAMS Shree K. Nayar,

12. Focal Plane Lamp Shift Surface Projector Camera Projector Defocus Model: Temporal Defocus Method Spectrum Depth Estimation: PROCAMS Shree K. Nayar,

13. Projector Half-mirror Camera Half-mirror Projector Camera Experimental Setup PROCAMS Shree K. Nayar,

14. PROCAMS Shree K. Nayar,

15. Scene Depth Map Fence and Leaves: Complex Occlusions PROCAMS Shree K. Nayar,

16. Scene Depth Map 3D Model Fence and Leaves: Complex Occlusions PROCAMS Shree K. Nayar,

17. Refocusing PROCAMS Shree K. Nayar,

18. SceneDepth Map Playing Cards: Object Insertion PROCAMS Shree K. Nayar,

19. SceneDepth Map Green Screen CaptureDepth-based Composition Playing Cards: Object Insertion PROCAMS Shree K. Nayar,

20. Projector’s Narrow Depth of Field PROCAMS Shree K. Nayar,

21. Multiple Projectors: (Bimber and Emmerling 2006) Focused Projection at Multiple Depths PROCAMS Shree K. Nayar,

22. Single Projector: Focused Projection at Multiple Depths PROCAMS Shree K. Nayar,

23. Projector Defocus: Three Plane Example 15 inches PROCAMS Shree K. Nayar,

24. Compensated Projection Front PlaneMiddle PlaneBack Plane Original Image Normal Projection Projector Defocus Compensation: Results PROCAMS Shree K. Nayar,

25. Projector as a Programmable Flash Depth from Projector Defocus Image-Complete Depth Maps (with Li Zhang, Siggraph 2006) Separation of Direct and Global Illumination Scattering Properties of Real Materials (with Krishnan, Grossberg, Raskar, Siggraph 2006)

26. source surface P Direct and Global Illumination A A : Direct B B : Interrelection C C : Subsurface D participating medium D : Volumetric translucent surface E E : Diffusion camera

27. Related Work (Seitz et. al., ICCV 05) Inverse Light Transport (Sen et. al., Siggraph 05) T Dual Photography Fast Separation of Direct and Global Images PROCAMS Shree K. Nayar,

28. direct global radiance Direct and Global Components: Interreflections surface i camera source j BRDF and geometry PROCAMS Shree K. Nayar,

29. High Frequency Illumination Pattern surface camera source fraction of activated source elements + i PROCAMS Shree K. Nayar,

30. High Frequency Illumination Pattern surface fraction of activated source elements camera source + - i PROCAMS Shree K. Nayar, PROCAMS Shree K. Nayar,

31. Separation from Two Images directglobal PROCAMS Shree K. Nayar,

32. Other Global Effects Subsurface Scattering Volumetric Scattering Diffusion PROCAMS Shree K. Nayar,

33. Diffuse Interreflections Specular Interreflections Volumetric Scattering Subsurface Scattering Diffusion PROCAMS Shree K. Nayar,

34. Scene DirectGlobal PROCAMS Shree K. Nayar,

35. Real World Examples: Can You Guess the Images? PROCAMS Shree K. Nayar,

36. Eggs: Diffuse Interreflections DirectGlobal PROCAMS Shree K. Nayar,

37. Wooden Blocks: Specular Interreflections DirectGlobal PROCAMS Shree K. Nayar,

38. Novel Images PROCAMS Shree K. Nayar,

39. Photometric Stereo using Direct Images Bowl Shape Source 1 Source 2 Source 3 Nayar et al., 1991 Direct Global PROCAMS Shree K. Nayar,

40. Peppers: Subsurface Scattering DirectGlobal PROCAMS Shree K. Nayar,

41. Novel Images PROCAMS Shree K. Nayar,

42. Hand DirectGlobal Skin: Hanrahan and Krueger 93, Uchida 96, Haro 01, Jensen et al. 01, Cula and Dana 02, Igarashi et al. 05, Weyrich et al. 05 PROCAMS Shree K. Nayar,

43. Separation from a Single Image PROCAMS Shree K. Nayar,

44. Face DirectGlobal Sum PROCAMS Shree K. Nayar,

45. Skin Tone Control Skin Color and Lipids: Tsumura et al. 03 PROCAMS Shree K. Nayar,

46. Summary Coded Structured Light Shifted Sinusoids Depth from Projector Defocus (Minimum Images: 3) Direct-Global Separation (Minimum Images: 2) PROCAMS Shree K. Nayar,

47. Open Questions What is Recoverable with a Projector and a Camera ? How Many Illumination Patterns? What are the Illumination Patterns? PROCAMS Shree K. Nayar,

48. Cameras for ProjectorsProjectors for Cameras Convergence of Cameras and Projectors Projectors in 2006 Size Power Cost x 1/10 PROCAMS Shree K. Nayar,