Spherical Convolution in Computer Graphics and Vision Ravi Ramamoorthi Columbia Vision and Graphics Center Columbia University SIAM Imaging Science Conference: May 17, 2006

Outline  Motivation and Practical Problems  Spherical Convolution and Applications  A Theory of Spherical Harmonic Identities  Signal Processing for Visual Appearance

Real-Time Rendering Motivation: Interactive rendering with natural illumination and realistic, measured materials

Inverse Rendering Photographs Geometric model Inverse Rendering Algorithm BRDF

Inverse Rendering Geometric model Forward Rendering Algorithm BRDF Novel lighting Rendering Photographs

Direct Object Relighting ? Unknown Lighting Unknown BRDF

Lighting-Insensitive Recognition Illuminate subject from many incident directions Space of images as lighting is varied

Checking Image Consistency  Easy to tamper / splice images  Image processing software widely available  In news reporting and other applications  Need to detect tampering or photomontage  Verify image consistency  Try to check consistency of lighting, shading

Checking Image Consistency

Without knowing illumination and reflectance:  How do we know if two objects / people in a photograph are lit consistently?  How do we detect inconsistencies or tampering?

Outline  Motivation and Practical Problems  Spherical Convolution and Applications  A Theory of Spherical Harmonic Identities  Signal Processing for Visual Appearance

Environment Maps Miller and Hoffman, 1984 Later, Greene 86, Cabral 87, 99,…

Irradiance Environment Maps Incident Radiance (Illumination Environment Map) Irradiance Environment Map R N

Computing Irradiance  Classically, hemispherical integral for each pixel  Lambertian surface is like a low pass filter  Frequency-space analysis (spherical harmonics) Incident Radiance Irradiance

Spherical Harmonics

Analytic Irradiance Formula Lambertian surface is low-pass filter Basri & Jacobs 01 Ramamoorthi & Hanrahan 01a

9 Parameter Approximation Exact image Order 2 9 terms RMS Error = 1% For any illumination, average error < 2% [Basri, Jacobs 01] Ramamoorthi and Hanrahan 01b

Real-Time Rendering Simple procedural rendering method (no textures)  Requires only matrix-vector multiply and dot-product  In software or NVIDIA vertex programming hardware Widely used in Games (AMPED for Microsoft Xbox), Movies (Pixar, Framestore CFC, …)

Computer Vision Complex Illumination Low Dimensional Subspace  Lighting Insensitive Recognition (Basri and Jacobs 01, Lee et al. 01, Ramamoorthi 02, …)  Photometric stereo, shape acquisition

Convolution for General Materials Ramamoorthi and Hanrahan 01 Frequency: product Spatial: integral Spherical Harmonics

Related Theoretical Work  Qualitative observation of reflection as convolution: Miller & Hoffman 84, Greene 86, Cabral et al. 87,99  Reflection as frequency-space operator: D’Zmura 91  Lambertian reflection is convolution: Basri Jacobs 01 Our Contributions  Explicitly derive frequency-space convolution formula  Formal quantitative analysis in general 3D case  Apply to real-time, inverse rendering, computer vision

Natural Lighting, Realistic Materials Ramamoorthi and Hanrahan 02

Inverse Rendering 3 photographs of cat sculpture Complex unknown illumination Geometry known Estimate microfacet BRDF and distant lighting

New View, Lighting Photograph Rendering Ramamoorthi and Hanrahan, 01c

Outline  Motivation and Practical Problems  Spherical Convolution and Applications  A Theory of Spherical Harmonic Identities  Signal Processing for Visual Appearance Mahajan, Ramamoorthi, Curless ECCV 06

Lighting 1 Lighting 2 Material 1Material 2 Two Objects – Two Lightings

Lighting 1 Lighting 2 Material 1Material 2 Two Objects – Two Lightings

Lighting 1 Lighting 2 Material 1Material 2 Two Objects – Two Lightings

Lighting 1 Lighting 2 Material 1Material 2 Two Objects – Two Lightings Independent of Lighting and BRDF

Lighting 1 Lighting 2 Material 1Material 2 Image Estimation Framework ?

Lighting 1 Lighting 2 Material 1Material 2 Image Estimation Framework ?

Lighting 1 Lighting 2 Material 1Material 2 Image Estimation Framework ?

Lighting 1 Lighting 2 Material 1Material 2 Image Estimation – BRDF Transfer ? BRDF Transfer Function

Lighting 1 Lighting 2 Material 1Material 2 Image Estimation – Lighting Transfer ?

Lighting 1 Lighting 2 Material 1Material 2 Image Estimation – Lighting Transfer ?

Lighting 1 Lighting 2 Material 1Material 2 Image Estimation – Lighting Transfer ? Light Transfer Function

Image Estimation Object 1 Object 2 Lighting 1 Our MethodActual Lighting 2 ? No BRDF and lighting known or estimated

Image Consistency Checking Spliced Image

Image Consistency Checking Spliced Image

Image Consistency Checking Single Image Identity diffuse + specular case Two Lightings – Same Reflectance Identity Two Materials – Two Lightings identity Tampered Cat Tampered CatUntampered Cat

Outline  Motivation and Practical Problems  Spherical Convolution and Applications  A Theory of Spherical Harmonic Identities  Signal Processing for Visual Appearance

Signal Processing for Appearance Signal Processing widely applicable visual appearance  Convolution relation for cast shadows [Soler and Sillion 98, Ramamoorthi et al. 04]  Convolution with glows for participating media (mist, fog, haze) [Sun et al. 05]  Signal-Processing analysis of light field and reflectance [Chai et al. 00, Zickler et al. 06]  Triple Product Integrals [Ng et al. 04]  First Order Analysis [Ramamoorthi et al. 06]

Video clip 1 Real Time Rendering of Scattering

Video clip 1 Reflectance Sharing

Acknowledgements  Collaborators  Dhruv Mahajan  Brian Curless  Pat Hanrahan  Sameer Agarwal (helpful discussions)  Funding: NSF and Sloan Foundation

Questions