1. Computational Fundamentals of Reflection COMS 6998-3, Lecture 7 0 01 2

2. Motivation Understand intrinsic computational structure of reflection and illumination Necessary for many applications in computer graphics (cannot solve by brute force!!) Real-time forward rendering IBR sampling rates, dimensionality explosion Inverse rendering and inverse problems in general Computer vision: complex lighting, materials

3. Real-Time Rendering Demo Motivation: Interactive rendering with complex natural illumination and realistic, measured BRDFs

4. Questions Images are view-dependent (4D quantity) Can we find low-dimensional structure to capture view-dependence?

5. Space of images as lighting varies Illuminate subject from many incident directions

6. Example Images Images from Debevec et al. 00

7. Principal Component Analysis Try to approximate with low dimensional subspace Linear combination of few principal components =.5+.5+ … =.7+.3+ … Principal component images

8. Lighting Variability Theory Infinite number of light directions, one coefficient/direction Space of images infinite dimensional [Belhumeur 98] Empirical [Hallinan 94, Epstein 95] Diffuse objects: 5D subspace suffices No satisfactory theoretical explanation of observations

9. Complex Light Transport Shadows high frequency Analysis possible? Low-dim. structure? Real-time complex lights? Agrawala, Ramamoorthi, Heirich, Moll SIGGRAPH 00

10. Challenges Illumination complexity Material (BRDF)/view complexity Transport complexity (shadows, interreflection) Fundamental questions Theoretical analysis of intrinsic complexity Sampling rates and resolutions Efficient practical algorithms

11. Outline Lighting variability in appearance [PAMI Oct, 2002] View variability real-time rendering [SIGGRAPH 02] Visibility/shadows [In Progress]

12. Lighting variability analysis 1.Frequency space analytic PCA construction 2.Mathematical derivation of principal components 3.Explain empirical results quantitatively Dimensionality of approximating subspace Forms of principal components Relative importance of principal components

13. Assumptions Single view of single object Lambertian Distant illumination Discount texture Discount concavities: interreflection, cast shadows Consider attached shadows (backfacing normals)

14. Definitions Irradiance (image) Radiance (light) Normal Lambertian = half-cosine

15. Previous Theoretical Work Discount attached shadows [Shashua 97, …] Resulting 3D subspace does not fully explain data Analytic PCA (without shadows) [Zhao & Yang 99] Irradiance (image) Radiance (light) Normal Lambertian = half-cosine

16. Spherical Harmonics -2012 0 1 2...2...

17. Spherical Harmonic Expansion Expand lighting (L), irradiance (E) in basis functions =.67+.36+ …

18. Lambertian BRDF Expansion -2012 0 1 2 Lambertian coefficients

19. Analytic Irradiance Formula Lambertian surface acts like low-pass filter 0 01 2 Basri & Jacobs 01 Ramamoorthi & Hanrahan 01

20. 9 Parameter Approximation Exact image Order 2 9 terms RMS Error = 1% For any illumination, average error < 2% [Basri Jacobs 01] -2012 0 1 2

21. Open Questions Relationship between spherical harmonics, PCA 9D approximation > 5D empirical subspace Key insight: Consider approximations over visible normals (upper hemisphere), not entire sphere

22. Intuition: Backwards Half-Cosine -2012 0 1 2 1 parameter irrelevant 8 or fewer params enough

23. Intuition: dimensionality reduction Start with 9D space, remove dimensions Mean (constant term) subtracted Backwards half-cosine x, xz very similar y, yz very similar Left with 5D subspace -2012 0 1 2

24. Results: Image of a Sphere Principal components (eigenvectors) mix (linear combinations of) spherical harmonics Results agree with experiment [Epstein 95] We predict: 3 eigenvectors = 91% variance, 5 give 96% Empirical : 3 eigenvectors = 94% variance, 5 give 98% % VAF (eigenvalue) 43%24% 2%

25. Results: Human Face Numerically compute orthogonality matrix Specific distribution of surface normals important Symmetries in sphere broken (faces are elongated) Principal components somewhat different from sphere % VAF 42%33%16% 4%2%

26. Results: Human Face Prediction: Principal components have specific forms Empirical : [Hallinan 94] Frontal lighting, side, above/below, extreme side, corner Frontal % VAF 42%33%16% 4%2% SideAbove/Below Extreme sideCorner

27. Results: Human Face Prediction: Space is close to 5D 3 principal components = 91% variance, 5 components = 97% Empirical : [Epstein 95] 3 principal components = 90% variance, 5 components = 94% Frontal % VAF 42%33%16% 4%2% SideAbove/Below Extreme sideCorner

28. Results: Human Face Prediction: groups of principal components Group 1: first two (frontal and side) Group 2: next three [with above/below always 3 rd ] Empirical: [Hallinan 94] Two groups [first two (frontal,side) and next three] Within group, %VAF close, may exchange places Frontal % VAF 42%33%16% 4%2% SideAbove/Below Extreme sideCorner

29. Summary: Lighting Analysis Analytic PCA construction with attached shadows Spherical harmonic analysis: Orthogonality matrix Mathematically derive principal components Qualitative, quantitative agreement with experiment Extend 9D Lambertian model to single view case

30. Implications Lighting Analysis Attached shadows nearly free: 5D subspace enough Mathematical derivation of principal components Basis functions for subspace methods for recognition,… Graphics applications: Image-Based, inverse rendering Complex illumination in computer vision

31. Outline Lighting variability in appearance [PAMI Oct, 2002] View variability real-time rendering [SIGGRAPH 02] Visibility/shadows [In Progress]

32. Reflection Equation 2D Environment Map

33. Reflection Equation 2D Environment Map N L BRDF

34. Reflection Equation 4D Orientation Light Field 2D Environment Map Previous Work: Blinn & Newell 76, Miller & Hoffman 84, Greene 86, Kautz & McCool 99, Cabral et al. 99, … BRDF N L

35. Goals Efficiently precompute and represent OLF Real-time rendering with OLF

36. Questions Parameterization and structure of OLF Structure leads to representation Computation and rendering of OLF

37. OLF Parameterization N L N V

38. N V N V R Reparameterize by reflection vector

39. OLF Parameterization Captures structure of BRDF (and hence OLF) better Reflective BRDFs become low-dimensional N V N V R Reparameterize by reflection vector

40. OLF Structure 2D view array of reflection maps 2D image array of view maps

41. OLF Structure: Phong 2D view array of reflection maps 2D image array of view maps Environment Map Phong Reflection Map (blurred environment map) Same reflection map for all views

42. OLF Structure: Phong Same reflection map for all views View x View y

43. OLF Structure: Phong Same reflection map for all views View maps constant for each R View x View y

44. OLF Structure: Phong Same reflection map for all views View maps constant for each R View x View y Reflection x Reflection y

45. OLF Structure: Lafortune View y Single 2D reflection map no longer sufficient But variation with viewing direction is slow View x

46. OLF Structure: Lafortune View maps vary slowly Reflection x Reflection y View y View x

47. A Simple Factorization View x View y Reflection y Reflection x *

48. Spherical Harmonic Reflection Map View-dependent reflection (cube)map Encode view maps with low-order spherical harmonics

49. Prefiltering Directly compute SHRM from Lighting, BRDF Convolution easier to compute in frequency domain Input Lighting and BRDF Spherical Harmonic coeffs. Convolution SHRM

50. Prefiltering 3 to 4 orders of magnitude faster (< 1 s compared to minutes or hours) Detailed analysis, algorithms, experiments in paper Input Lighting and BRDF Spherical Harmonic coeffs. Convolution SHRM

51. Number of terms: CURET Analysis for all 61 samples [full bar chart in paper] For essentially all materials, 9-16 terms in SHRM suffice

52. Demo

53. Summary view variability Theoretical, empirical analysis of sampling rates and resolutions Frequency space analysis directly on lighting, BRDF Low order expansion suffices for essentially all BRDFs Spherical Harmonic Reflection Maps Hybrid angular-frequency space Compact, efficient, accurate Easy to analyze errors, determine number of terms Fast computation using convolution

54. Implications Frequency space methods for rendering Global illumination Fast computation of surface light fields Compression for optimal factored representations PCA on SHRMs Theoretical analysis of sampling rates, resolutions General framework for sampling in image-based rendering

55. Outline Lighting variability in appearance [PAMI Oct, 2002] View variability real-time rendering [SIGGRAPH 02] Visibility/shadows [In Progress]

56. Visibility complexity (high freq)

57. But Sparse (< 4%)

58. Questions on Visibility Theory Locally low-dimensional subspaces? Intrinsic complexity of binary function? Practical Real-time rendering with complex soft shadows, changing illumination for lighting design, simulation Efficient encoding/decoding (wavelets, PCA, dictionaries, hierarchical?) In progress….

59. Overall Summary Many applications in graphics cannot be solved by brute force Real-time rendering IBR sampling rates, dimensionality explosion Inverse rendering, inverse problems Computer vision: complex lighting, materials Need fundamental understanding of nature of reflection/lighting Illumination complexity Material (view) complexity Transport complexity

60. Overall Summary Theoretical analysis tools Signal processing, sampling theory Low-dimensional subspaces Information theory, information-based complexity? Practical algorithms Real-time rendering with complex lights,view, transport? Lighting, Material design? Exploit theoretical analysis (sampling rates, forward/inverse duality, angular/frequency/sparsity duality, subspace results, differential analysis, perception)