1. All-Frequency Shadows Using Non-linear Wavelet Lighting Approximation Ren Ng Stanford Ravi Ramamoorthi Columbia SIGGRAPH 2003 Pat Hanrahan Stanford

2. Light on Stone River (Goldsworthy)

3. From Frank Gehry Architecture, Ragheb ed. 2001 Lighting Design

4. Existing Fast Shadow Techniques We know how to render very hard and very soft shadows Sloan, Kautz, Snyder 2002 Shadows from smooth lighting (precomputed radiance transfer) Sen, Cammarano, Hanrahan, 2003 Shadows from point-lights (shadow maps, volumes)

5. Soft Lighting Teapot in Grace Cathedral

6. All-Frequency Lighting Teapot in Grace Cathedral

7. Formulation Geometry relighting: assume diffuse Image relighting: fix view point T could include global effects

8. Relighting as Matrix-Vector Multiply

9. Input Lighting (Cubemap Vector) Output Image (Pixel Vector) Transport Matrix

10. Ray-Tracing Matrix Columns

11. Only works for image relighting

12. Light-Transport Matrix Rows

15. Rasterizing Matrix Rows Pre-computing rows n Rasterize visibility hemicubes with graphics hardware n Read back pixels and weight by reflection function

16. Matrix Multiplication is Enormous Dimension n 512 x 512 pixel images n 6 x 64 x 64 cubemap environments Full matrix-vector multiplication is intractable n On the order of 10 10 operations per frame

17. Sparse Matrix-Vector Multiplication Choose data representations with mostly zeroes Vector: Use non-linear wavelet approximation on lighting Matrix: Wavelet-encode transport rows

18. Non-linear Wavelet Light Approximation Wavelet Transform

19. Non-linear Wavelet Light Approximation Non-linear Approximation Retain 0.1% – 1% terms

20. Why Non-linear Approximation? Linear n Use a fixed set of approximating functions n Precomputed radiance transfer uses 25 - 100 of the lowest frequency spherical harmonics Non-linear n Use a dynamic set of approximating functions (depends on each frame’s lighting) n In our case: choose 10’s - 100’s from a basis of 24,576 wavelets Idea: Compress lighting by considering input data

21. Why Wavelets? Wavelets provide dual space / frequency locality n Large wavelets capture low frequency, area lighting n Small wavelets capture high frequency, compact features In contrast n Spherical harmonics n Perform poorly on compact lights n Pixel basis n Perform poorly on large area lights

22. Choosing Non-linear Coefficients Three methods of prioritizing 1. Magnitude of wavelet coefficient n Optimal for approximating lighting 2. Transport-weighted n Biases lights that generate bright images 3. Area-weighted n Biases large lights

23. Sparse Matrix-Vector Multiplication Choose data representations with mostly zeroes Vector: Use non-linear wavelet approximation on lighting Matrix: Wavelet-encode transport rows Choose data representations with mostly zeroes Vector: Use non-linear wavelet approximation on lighting Matrix: Wavelet-encode transport rows

24. Matrix Row Wavelet Encoding

25. Extract Row

26. Matrix Row Wavelet Encoding Wavelet Transform

27. Matrix Row Wavelet Encoding Wavelet Transform

28. Matrix Row Wavelet Encoding Wavelet Transform

29. Matrix Row Wavelet Encoding Wavelet Transform

30. Matrix Row Wavelet Encoding Store Back in Matrix 0 ’ ’ 0 0

31. Matrix Row Wavelet Encoding 0 ’ ’ 0 0 Only 3% – 30% are non-zero

32. Total Compression Lighting vector compression n Highly lossy n Compress to 0.1% – 1% Matrix compression n Essentially lossless encoding n Represent with 3% – 30% non-zero terms Total compression in sparse matrix-vector multiply n 3 – 4 orders of magnitude less work than full multiplication

33. Error Analysis Compare to linear spherical harmonic approximation n [Sloan, Kautz, Snyder, 2002] Measure approximation quality in two ways: n Error in lighting n Error in output image pixels Main point (for detailed natural lighting) n Non-linear wavelets converge exponentially faster than linear harmonics

34. Lighting Error

35. Error in Lighting: St Peter’s Basilica Approximation Terms Relative L 2 Error (%) Sph. Harmonics Non-linear Wavelets

36. Output Image Comparison Top: Linear Spherical Harmonic Approximation Bottom: Non-linear Wavelet Approximation 252002,00020,000

37. Error in Output Image: Plant Scene Approximation Terms Relative L 2 Error (%) Sph. Harmonics Non-linear Wavelets

38. Results SIGGRAPH 2003 video

39. Summary A viable representation for all-frequency lighting n Sparse non-linear wavelet approximation n 100 – 1000 times information reduction Efficient relighting from detailed environment maps

40. Triple Product Wavelet Integrals For All-Frequency Relighting Ren Ng Stanford Ravi Ramamoorthi Columbia SIGGRAPH 2004 Pat Hanrahan Stanford

41. Precomputed Relighting Techniques Low-Frequency Lighting, Dynamic View [Sloan et al 2002, 2003] All-Frequency Lighting, Fixed View [Ng et al 2003]

42. Formulation Simplifications: 1. Incorporate cosine term into BRDF 2. Assume the lighting is distant, so L depends only on ω 3. Uniform BRDF over the surface 4. Expressed in the global frame; BRDF has to be rotated For a fixed point,

43. Double product Let

44. Double Product Integral Relighting Changing Surface Position Changing Camera Viewpoint Lighting Transport

45. Problem Characterization 6D Precomputation Space n Distant Lighting(2D) n Surface (2D) n View(2D) With ~ 100 samples per dimension n Total of ~ 10 12 samples

46. Factorization Approach 6D Transport ~ 10 12 samples ~ 10 8 samples 4D Visibility4D BRDF * =

47. Triple Product Integral Relighting

48. Triple Product Integrals

49. Basis Requirements 1.Need few non-zero “tripling” coefficients 2.Need sparse basis coefficients

50. 1. Number Non-Zero Tripling Coeffs Basis Choice Number Non-Zero General (e.g. PCA) O (N 3)O (N 3) Pixels O (N)O (N) Fourier Series O (N 2)O (N 2) Sph. Harmonics O (N 5 / 2)O (N 5 / 2) Haar Wavelets O ( N log N )

51. Basis Requirements 1.Need few non-zero “tripling” coefficients 2.Need sparse basis coefficients

52. 2. Sparsity in Light Approximation Approximation Terms Relative L 2 Error (%) Pixels Wavelets

53. 2. Sparsity in Visibility and BRDF Visibility n ~ 40% sparse in pixels n 1 – 5 % sparse in wavelets BRDF with diffuse component n ~ 50% sparse in pixels n 0.1 – 1% sparse in wavelets

54. Summary of Basis Analysis Choose wavelet basis because 1.Few non-zero tripling coefficients 1.Sparse basis coefficients

55. Haar Tripling Coefficient Analysis * * * *

56. Visual Review of 2D Haar Basis

57. * * * = = = 1. Non-Overlapping Haar Multiplication

58. * * * 1. Zero Triple Product Integral * * *

59. * * * = = = 2. Co-Square Haar Multiplication

60. * * * * * * 2. Non-Zero Triple Product Integrals

61. = = = 3. Overlapping Haar Multiplication * * *

62. 3. More Non-Zero Triple Integrals * * * * * *

63. Haar Tripling Coefficient Theorem The integral of three Haar wavelets is non-zero iff n All three are the scaling function n All three are co-square and different n Two are identical, and the third overlaps at a coarser level

64. Theorem Consequences 1.Prove O(N log N) Haar sparsity 2.Derive O(N log N) triple product integral algorithm n Dynamic programming eliminates log N term n Final complexity is linear in number of retained basis coefficients

65. High-Level Algorithm: Precomputation for each vertex compute the visibility cubemap wavelet encode store for each viewing direction compute BRDF for that view nonlinear wavelet approximation store for each vertex compute the visibility cubemap wavelet encode store for each viewing direction compute BRDF for that view nonlinear wavelet approximation store

66. for each frame wavelet encode lighting, L for each vertex in mesh look up visibility, V look up BRDF for view, ρ integrate product of L, V, ρ set color of vertex draw colored mesh High-Level Algorithm: Relighting for each frame wavelet encode lighting, L for each vertex in mesh look up visibility, V look up BRDF for view, ρ integrate product of L, V, ρ set color of vertex draw colored mesh

67. Relit Images

68. Changing Lighting, Fixed View

69. Changing View, Fixed Lighting

70. Dynamic Lighting and View

71. Results SIGGRAPH 2004 video

72. Summary All-frequency relighting, changing view n Factor into visibility and BRDF n Fast relight in Haar basis Triple product analysis and algorithms n Analysis of several bases n Haar tripling theorem n Efficient triple product integral estimation

73. Efficient Wavelet Rotation for Environment Map Rendering Rui Wang Ren Ng David Luebke Greg Humphreys EGSR 2006

74. Triple product ωoωo x The output color of vertex x given view direction ω o BRDF : Represents lighting reflective ratio of x at given light direction ω to view direction ω o. Lighting : Represents the radiance of direction ω. Visibility : Decides whether light is visible in direction ω.

75. Triple product ωoωo x The output radiance is the integral of these functions over all directions!

76. BRDF Lighting Global frame

77. BRDF Lighting Local frame

78. Rotation n Because we can’t rotate wavelet basis easily, some functions must be sampled with different rotations in advance and use lookup at runtime. n Spherical harmonics does not have this problem because it can be rotated easily.

79. Rotation of spherical functions

80. Rotation of spherical basis source basis target basis Basis transform matrix

81. n Transform matrices have to be pre- computed at several rotations

82. Varying truncation threshold

83. Varying rotation sampling rate