A Signal-Processing Framework for Inverse Rendering Ravi Ramamoorthi Pat Hanrahan Stanford University

Photorealistic Rendering Materials/Lighting (Texture Reflectance[BRDF] Lighting) Realistic input models required Arnold Renderer: Marcos Fajardo Rendering Algorithm 80’ s,90’ s : Physically based Geometry 70’ s, 80’ s : Splines 90’ s : Range Data

FlowchartFlowchart Photographs Geometric model Inverse Rendering Algorithm Lighting BRDF

FlowchartFlowchart Photographs Geometric model Forward Rendering Algorithm Lighting BRDF Rendering

FlowchartFlowchart Photographs Geometric model Forward Rendering Algorithm BRDF Novel lighting Rendering

AssumptionsAssumptions Known geometry Distant illumination Homogenous isotropic materials Convex curved surfaces: no shadows, interreflection Later, practical algorithms: relax some assumptions Known geometry Distant illumination Homogenous isotropic materials Convex curved surfaces: no shadows, interreflection Later, practical algorithms: relax some assumptions

Inverse Rendering Miller and Hoffman 84 Marschner and Greenberg 97 Sato et al. 97 Dana et al. 99 Debevec et al. 00 Marschner et al. 00 Sato et al. 99 Nishino et al. 01 Lighting BRDF Known Unknown Known Textures are a third axis

Inverse Problems: Difficulties Angular width of Light Source Surface roughness Ill-posed (ambiguous)

ContributionsContributions 1.Formalize reflection as convolution 2.Signal-processing framework 3.Analyze well-posedness of inverse problems 4.Practical algorithms 1.Formalize reflection as convolution 2.Signal-processing framework 3.Analyze well-posedness of inverse problems 4.Practical algorithms

Reflection as Convolution (2D) L Lighting Reflected Light Field BRDF L

Reflection as Convolution (2D) Fourier analysis L L Frequency: product Spatial: integral

Spherical Harmonic Analysis 2D: 3D:

Insights: Signal Processing Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution

Insights: Signal Processing Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution Filter is Delta function : Output = Signal Mirror BRDF : Image = Lighting [Miller and Hoffman 84] Mirror BRDF : Image = Lighting [Miller and Hoffman 84] Image courtesy Paul Debevec

Insights: Signal Processing Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution Signal is Delta function : Output = Filter Point Light Source : Images = BRDF [Marschner et al. 00] Point Light Source : Images = BRDF [Marschner et al. 00]

Amplitude Frequency Roughness Phong, Microfacet Models Mirror Illumination estimation ill-posed for rough surfaces

Inverse Lighting Well-posed unless denominator vanishes BRDF should contain high frequencies : Sharp highlights Diffuse reflectors low pass filters: Inverse lighting ill-posed Well-posed unless denominator vanishes BRDF should contain high frequencies : Sharp highlights Diffuse reflectors low pass filters: Inverse lighting ill-posed Given: B,ρ find L

Inverse BRDF Well-posed unless L lm vanishes Lighting should have sharp features (point sources, edges) BRDF estimation ill-conditioned for soft lighting Well-posed unless L lm vanishes Lighting should have sharp features (point sources, edges) BRDF estimation ill-conditioned for soft lighting Directional Source Area source Same BRDF Given: B,L find ρ

Factoring the Light Field Light Field can be factored Up to global scale factor Assumes reciprocity of BRDF Can be ill-conditioned Analytic formula in paper Light Field can be factored Up to global scale factor Assumes reciprocity of BRDF Can be ill-conditioned Analytic formula in paper Given: B find L and ρ 4D2D3D More knowns (4D) than unknowns (2D/3D) More knowns (4D) than unknowns (2D/3D)

Practical Issues Incomplete sparse data (few photographs) Difficult to compute frequency spectra Concavities: Self Shadowing and Interreflection Spatially varying BRDFs: Textures Incomplete sparse data (few photographs) Difficult to compute frequency spectra Concavities: Self Shadowing and Interreflection Spatially varying BRDFs: Textures

Practical Issues Incomplete sparse data (few photographs) Difficult to compute frequency spectra Concavities: Self Shadowing and Interreflection Spatially varying BRDFs: Textures Issues can be addressed; can derive practical algorithms Dual spatial (angular) and frequency-space representation Simple extensions for shadowing, textures Incomplete sparse data (few photographs) Difficult to compute frequency spectra Concavities: Self Shadowing and Interreflection Spatially varying BRDFs: Textures Issues can be addressed; can derive practical algorithms Dual spatial (angular) and frequency-space representation Simple extensions for shadowing, textures

Algorithm Validation Photograph KdKd 0.91 KsKs 0.09 μ 1.85  0.13 “True” values

Algorithm Validation Renderings Unknown lighting Photograph Known lighting KdKd KsKs μ  “True” values Image RMS error 5%

Inverse BRDF: Spheres Bronze Delrin PaintRough Steel Photographs Renderings (Recovered BRDF)

Complex Geometry 3 photographs of a sculpture Complex unknown illumination Geometry known Estimate microfacet BRDF and distant lighting

ComparisonComparison Photograph Rendering

New View, Lighting Photograph Rendering

Textured Objects Photograph Rendering

SummarySummary Reflection as convolution Signal-processing framework Formal study of inverse rendering Practical algorithms Reflection as convolution Signal-processing framework Formal study of inverse rendering Practical algorithms

Implications and Future Work Frequency space analysis of reflection Well-posedness of inverse problems Perception, human vision Forward rendering [Friday] Complex uncontrolled illumination Frequency space analysis of reflection Well-posedness of inverse problems Perception, human vision Forward rendering [Friday] Complex uncontrolled illumination

AcknowledgementsAcknowledgements Marc Levoy Szymon Rusinkiewicz Steve Marschner John Parissenti, Jean Gleason Scanned cat sculpture is “Serenity” by Sue Dawes Hodgson-Reed Stanford Graduate Fellowship NSF ITR grant # : “Interacting with the Visual World” Paper Website: Marc Levoy Szymon Rusinkiewicz Steve Marschner John Parissenti, Jean Gleason Scanned cat sculpture is “Serenity” by Sue Dawes Hodgson-Reed Stanford Graduate Fellowship NSF ITR grant # : “Interacting with the Visual World” Paper Website:

The End

Related Work Qualitative observation of reflection as convolution: Miller & Hoffman 84, Greene 86, Cabral et al. 87,99 Reflection as frequency-space operator: D’Zmura 91 Lambertian reflection is convolution: Basri Jacobs 01 Our Contributions Explicitly derive frequency-space convolution formula Formal Quantitative Analysis in General 3D Case Qualitative observation of reflection as convolution: Miller & Hoffman 84, Greene 86, Cabral et al. 87,99 Reflection as frequency-space operator: D’Zmura 91 Lambertian reflection is convolution: Basri Jacobs 01 Our Contributions Explicitly derive frequency-space convolution formula Formal Quantitative Analysis in General 3D Case

Spherical Harmonics (3D)

Dual Representation Dual Representation Diffuse BRDF: Filter width small in frequency domain Specular: Filter width small in spatial (angular) domain Practical Representation: Dual angular, frequency-space Diffuse BRDF: Filter width small in frequency domain Specular: Filter width small in spatial (angular) domain Practical Representation: Dual angular, frequency-space B B d diffuse B s,slow slow specular (area sources) B s,fast fast specular (directional) = + + FrequencyAngular Space

Inverse Lambertian Sum l=2Sum l=4 True Lighting Mirror Teflon

Other Papers Linked to from website for this paper Theory Flatland or 2D using Fourier analysis [SPIE 01] Lambertian: radiance from irradiance [JOSA 01] Application to other areas Forward Rendering (Friday) [SIGGRAPH 01] Lighting variability object recognition [CVPR 01] Linked to from website for this paper Theory Flatland or 2D using Fourier analysis [SPIE 01] Lambertian: radiance from irradiance [JOSA 01] Application to other areas Forward Rendering (Friday) [SIGGRAPH 01] Lighting variability object recognition [CVPR 01]