1. A Signal-Processing Framework for Forward and Inverse Rendering COMS 6998-3, Lecture 8

2. Illumination Illusion People perceive materials more easily under natural illumination than simplified illumination. Images courtesy Ron Dror and Ted Adelson

3. Illumination Illusion People perceive materials more easily under natural illumination than simplified illumination. Images courtesy Ron Dror and Ted Adelson

4. Material Recognition Photographs of 4 spheres in 3 different lighting conditions courtesy Dror and Adelson

5. Estimating BRDF and Lighting Photographs Geometric model Inverse Rendering Algorithm Lighting BRDF

6. Estimating BRDF and Lighting Photographs Geometric model Forward Rendering Algorithm Rendering Lighting BRDF

7. Estimating BRDF and Lighting Photographs Geometric model Forward Rendering Algorithm Rendering Novel lighting BRDF

8. Inverse Problems: Difficulties Angular width of Light Source Surface roughness Ill-posed (ambiguous)

9. Motivation Understand nature of reflection and illumination Applications in computer graphics Real-time forward rendering Inverse rendering

10. Contributions of Thesis 1.Formalize reflection as convolution 2.Signal-processing framework 3.Practical forward and inverse algorithms

11. Outline Motivation Reflection as Convolution Preliminaries, assumptions Reflection equation, Fourier analysis (2D) Spherical Harmonic Analysis (3D) Signal-Processing Framework Applications Summary and Implications

12. Assumptions Known geometry Laser range scanner Structured light

13. Assumptions Known geometry Convex curved surfaces: no shadows, interreflection Complex geometry: use surface normal

14. Assumptions Known geometry Convex curved surfaces: no shadows, interreflection Distant illumination Photograph of mirror sphere Illumination: Grace Cathedral courtesy Paul Debevec

15. Assumptions Known geometry Convex curved surfaces: no shadows, interreflection Distant illumination Homogeneous isotropic materials Isotropic Anisotropic

16. Assumptions Known geometry Convex curved surfaces: no shadows, interreflection Distant illumination Homogeneous isotropic materials Later, practical algorithms: relax some assumptions

17. Reflection Lighting L Reflected Light Field BRDF B 

18. Reflection as Convolution (2D) Lighting Reflected Light Field BRDF L B B L L

19. Reflection as Convolution (2D) Lighting Reflected Light Field BRDF L B L B

20. Reflection as Convolution (2D) L B L B

21. Convolution x Signal f(x) Filter g(x) Output h(u) u

22. Convolution x Signal f(x) Filter g(x) Output h(u) u u1u1

23. Convolution x Signal f(x) Filter g(x) Output h(u) u u2u2

24. Convolution x Signal f(x) Filter g(x) Output h(u) u u3u3

25. Convolution x Signal f(x) Filter g(x) Output h(u) u Fourier analysis

26. Reflection as Convolution (2D) Frequency: product Spatial: integral Fourier analysis R. Ramamoorthi and P. Hanrahan “Analysis of Planar Light Fields from Homogeneous Convex Curved Surfaces under Distant Illumination” SPIE Photonics West 2001: Human Vision and Electronic Imaging VI pp 195-208 L B L B

27. Related Work Qualitative observation of reflection as convolution: Miller & Hoffman 84, Greene 86, Cabral et al. 87,99 Reflection as frequency-space operator: D’Zmura 91 Lambertian reflection is convolution: Basri Jacobs 01 Our Contributions Explicitly derive frequency-space convolution formula Formal quantitative analysis in general 3D case

28. Spherical Harmonics -2012 0 1 2...2...

29. Spherical Harmonic Analysis 2D: 3D:

30. Outline Motivation Reflection as Convolution Signal-Processing Framework Insights, examples Well-posedness of inverse problems Applications Summary and Implications

31. Insights: Signal Processing Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution

32. Insights: Signal Processing Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution Filter is Delta function : Output = Signal Mirror BRDF : Image = Lighting [Miller and Hoffman 84] Image courtesy Paul Debevec

33. Insights: Signal Processing Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution Signal is Delta function : Output = Filter Point Light Source : Images = BRDF [Marschner et al. 00]

34. Amplitude Frequency Roughness Phong, Microfacet Models Mirror Illumination estimation ill-posed for rough surfaces Analytic formulae in R. Ramamoorthi and P. Hanrahan “A Signal-Processing Framework for Inverse Rendering” SIGGRAPH 2001 pp 117-128

35. Lambertian Incident radiance (mirror sphere) Irradiance (Lambertian) N 0 01 2 R. Ramamoorthi and P. Hanrahan “On the Relationship between Radiance and Irradiance: Determining the Illumination from Images of a Convex Lambertian Object” Journal of the Optical Society of America A 18(10) Oct 2001 pp 2448-2459 R. Basri and D. Jacobs “Lambertian Reflectance and Linear Subspaces” ICCV 2001 pp 383-390

36. Inverse Lighting Well-posed unless denominator vanishes BRDF should contain high frequencies : Sharp highlights Diffuse reflectors low pass filters: Inverse lighting ill-posed Given: B,ρ find L

37. Inverse BRDF Well-posed unless L lm vanishes Lighting should have sharp features (point sources, edges) BRDF estimation ill-conditioned for soft lighting Directional Source Area source Same BRDF Given: B,L find ρ

38. Factoring the Light Field Light Field can be factored Up to global scale factor Assumes reciprocity of BRDF Can be ill-conditioned Analytic formula derived Given: B find L and ρ 4D2D3D More knowns (4D) than unknowns (2D/3D)

39. Outline Motivation Reflection as Convolution Signal-Processing Framework Applications Forward rendering (convolution) Inverse rendering (deconvolution) Summary and Implications

40. Computing Irradiance Classically, hemispherical integral for each pixel Lambertian surface is like low pass filter Frequency-space analysis Incident Radiance Irradiance

41. 9 Parameter Approximation Exact image Order 0 1 term (constant) RMS error = 25 % -2012 0 1 2

42. 9 Parameter Approximation Exact image Order 1 4 terms (linear) RMS Error = 8% -2012 0 1 2

43. 9 Parameter Approximation Exact image Order 2 9 terms (quadratic) RMS Error = 1% For any illumination, average error < 2% [Basri Jacobs 01] -2012 0 1 2

44. Comparison Incident illumination 300x300 Irradiance map Texture: 256x256 Hemispherical Integration 2Hrs Irradiance map Texture: 256x256 Spherical Harmonic Coefficients 1sec

45. Inverse Rendering Miller and Hoffman 84 Marschner and Greenberg 97 Sato et al. 97 Dana et al. 99 Debevec et al. 00 Marschner et al. 00 Sato et al. 99 Lighting BRDF Known Unknown Known Textures are a third axis

46. Contributions Complex illumination Factorization of BRDF, lighting (find both) New representations and algorithms Formal study of inverse problems (well-posed?)

47. Complications Incomplete sparse data (few photographs) Concavities: Self Shadowing Spatially varying BRDFs

48. Complications Challenge: Incomplete sparse data (few photographs) Difficult to compute frequency spectra Solution: Use parametric BRDF model Dual angular and frequency space representation

49. Algorithm Validation Photograph KdKd 0.91 KsKs 0.09 μ 1.85  0.13 “True” values

50. Algorithm Validation Renderings Unknown lighting Photograph Known lighting KdKd 0.910.890.87 KsKs 0.090.110.13 μ 1.851.781.48  0.130.120.14 “True” values Image RMS error 5%

51. Inverse BRDF: Spheres Bronze Delrin PaintRough Steel Photographs Renderings (Recovered BRDF)

52. Complications Challenge: Complex geometry with concavities Self shadowing Solution: Use associativity of convolution Blur lighting, treat specular BRDF term as mirror Single ray for shadowing, easy in ray tracer

53. Complex Geometry 3 photographs of a sculpture Complex unknown illumination Geometry known Estimate microfacet BRDF and distant lighting

54. Comparison Photograph Rendering

55. New View, Lighting Photograph Rendering

56. Complications Challenge: Spatially varying BRDFs Solution: Use textures to modulate BRDF parameters

57. Textured Objects Photograph Rendering

58. Summary Reflection as convolution Frequency-space analysis gives many insights Practical forward and inverse algorithms Signal-Processing: A useful paradigm for forward and inverse rendering in graphics and vision