1. Spherical Convolution in Computer Graphics and Vision Ravi Ramamoorthi Columbia Vision and Graphics Center Columbia University SIAM Imaging Science Conference: May 17, 2006

2. Outline  Motivation and Practical Problems  Spherical Convolution and Applications  A Theory of Spherical Harmonic Identities  Signal Processing for Visual Appearance

3. Real-Time Rendering Motivation: Interactive rendering with natural illumination and realistic, measured materials

4. Inverse Rendering Photographs Geometric model Inverse Rendering Algorithm BRDF

5. Inverse Rendering Geometric model Forward Rendering Algorithm BRDF Novel lighting Rendering Photographs

6. Direct Object Relighting ? Unknown Lighting Unknown BRDF

7. Lighting-Insensitive Recognition Illuminate subject from many incident directions Space of images as lighting is varied

8. Checking Image Consistency  Easy to tamper / splice images  Image processing software widely available  In news reporting and other applications  Need to detect tampering or photomontage  Verify image consistency  Try to check consistency of lighting, shading

9. Checking Image Consistency

10. Without knowing illumination and reflectance:  How do we know if two objects / people in a photograph are lit consistently?  How do we detect inconsistencies or tampering?

11. Outline  Motivation and Practical Problems  Spherical Convolution and Applications  A Theory of Spherical Harmonic Identities  Signal Processing for Visual Appearance

12. Environment Maps Miller and Hoffman, 1984 Later, Greene 86, Cabral 87, 99,…

13. Irradiance Environment Maps Incident Radiance (Illumination Environment Map) Irradiance Environment Map R N

14. Computing Irradiance  Classically, hemispherical integral for each pixel  Lambertian surface is like a low pass filter  Frequency-space analysis (spherical harmonics) Incident Radiance Irradiance

15. Spherical Harmonics -2012 0 1 2...2...

16. Analytic Irradiance Formula Lambertian surface is low-pass filter 0 01 2 Basri & Jacobs 01 Ramamoorthi & Hanrahan 01a

17. 9 Parameter Approximation -201 2 0 1 2 Exact image Order 2 9 terms RMS Error = 1% For any illumination, average error < 2% [Basri, Jacobs 01] Ramamoorthi and Hanrahan 01b

18. Real-Time Rendering Simple procedural rendering method (no textures)  Requires only matrix-vector multiply and dot-product  In software or NVIDIA vertex programming hardware Widely used in Games (AMPED for Microsoft Xbox), Movies (Pixar, Framestore CFC, …)

19. Computer Vision Complex Illumination Low Dimensional Subspace  Lighting Insensitive Recognition (Basri and Jacobs 01, Lee et al. 01, Ramamoorthi 02, …)  Photometric stereo, shape acquisition

20. Convolution for General Materials Ramamoorthi and Hanrahan 01 Frequency: product Spatial: integral Spherical Harmonics

21. Related Theoretical Work  Qualitative observation of reflection as convolution: Miller & Hoffman 84, Greene 86, Cabral et al. 87,99  Reflection as frequency-space operator: D’Zmura 91  Lambertian reflection is convolution: Basri Jacobs 01 Our Contributions  Explicitly derive frequency-space convolution formula  Formal quantitative analysis in general 3D case  Apply to real-time, inverse rendering, computer vision

22. Natural Lighting, Realistic Materials Ramamoorthi and Hanrahan 02

23. Inverse Rendering 3 photographs of cat sculpture Complex unknown illumination Geometry known Estimate microfacet BRDF and distant lighting

24. New View, Lighting Photograph Rendering Ramamoorthi and Hanrahan, 01c

25. Outline  Motivation and Practical Problems  Spherical Convolution and Applications  A Theory of Spherical Harmonic Identities  Signal Processing for Visual Appearance Mahajan, Ramamoorthi, Curless ECCV 06

26. Lighting 1 Lighting 2 Material 1Material 2 Two Objects – Two Lightings

27. Lighting 1 Lighting 2 Material 1Material 2 Two Objects – Two Lightings

28. Lighting 1 Lighting 2 Material 1Material 2 Two Objects – Two Lightings

29. Lighting 1 Lighting 2 Material 1Material 2 Two Objects – Two Lightings Independent of Lighting and BRDF

30. Lighting 1 Lighting 2 Material 1Material 2 Image Estimation Framework ?

31. Lighting 1 Lighting 2 Material 1Material 2 Image Estimation Framework ?

32. Lighting 1 Lighting 2 Material 1Material 2 Image Estimation Framework ?

33. Lighting 1 Lighting 2 Material 1Material 2 Image Estimation – BRDF Transfer ? BRDF Transfer Function

34. Lighting 1 Lighting 2 Material 1Material 2 Image Estimation – Lighting Transfer ?

35. Lighting 1 Lighting 2 Material 1Material 2 Image Estimation – Lighting Transfer ?

36. Lighting 1 Lighting 2 Material 1Material 2 Image Estimation – Lighting Transfer ? Light Transfer Function

37. Image Estimation Object 1 Object 2 Lighting 1 Our MethodActual Lighting 2 ? No BRDF and lighting known or estimated

38. Image Consistency Checking Spliced Image

39. Image Consistency Checking Spliced Image

40. Image Consistency Checking Single Image Identity diffuse + specular case Two Lightings – Same Reflectance Identity Two Materials – Two Lightings identity Tampered Cat Tampered CatUntampered Cat

41. Outline  Motivation and Practical Problems  Spherical Convolution and Applications  A Theory of Spherical Harmonic Identities  Signal Processing for Visual Appearance

42. Signal Processing for Appearance Signal Processing widely applicable visual appearance  Convolution relation for cast shadows [Soler and Sillion 98, Ramamoorthi et al. 04]  Convolution with glows for participating media (mist, fog, haze) [Sun et al. 05]  Signal-Processing analysis of light field and reflectance [Chai et al. 00, Zickler et al. 06]  Triple Product Integrals [Ng et al. 04]  First Order Analysis [Ramamoorthi et al. 06]

43. Video clip 1 Real Time Rendering of Scattering

44. Video clip 1 Reflectance Sharing

45. Acknowledgements  Collaborators  Dhruv Mahajan  Brian Curless  Pat Hanrahan  Sameer Agarwal (helpful discussions)  Funding: NSF and Sloan Foundation http://www.cs.columbia.edu/~ravir

46. Questions