All-Frequency Shadows Using Non-linear Wavelet Lighting Approximation Ren Ng Stanford Ravi Ramamoorthi Columbia Pat Hanrahan Stanford

From Frank Gehry Architecture, Ragheb ed Lighting Design

From Frank Gehry Architecture, Ragheb ed Lighting Design

Existing Fast Shadow Techniques We know how to render very hard and very soft shadows Sloan, Kautz, Snyder 2002 Shadows from smooth lighting (precomputed radiance transfer) Sen, Cammarano, Hanrahan, 2003 Shadows from point-lights (shadow maps, volumes)

All-Frequency Lighting Teapot in Grace Cathedral

Soft Lighting Teapot in Grace Cathedral

All-Frequency Lighting Teapot in Grace Cathedral

Soft Lighting Teapot in Grace Cathedral

All-Frequency Lighting Teapot in Grace Cathedral

Overview Matrix formulation of relighting n Pre-computing the light-transport matrix n Compressing the matrix multiplication Results n Demonstration n Error analysis (Comparison with PRT) Future directions n Limitations n Areas for research

Overview Matrix formulation of relighting n Pre-computing the light-transport matrix n Compressing the matrix multiplication Results n Demonstration n Error analysis (Comparison with PRT) Future directions n Limitations n Areas for research

Relighting as Matrix-Vector Multiply

Input Lighting (Cubemap Vector) Output Image (Pixel Vector) Transport Matrix

Ray-Tracing Matrix Columns

Light-Transport Matrix Rows

Rasterizing Matrix Rows Pre-computing rows n Rasterize visibility hemicubes with graphics hardware n Read back pixels and weight by reflection function

Overview Matrix formulation of relighting n Pre-computing the light-transport matrix n Compressing the matrix multiplication Results n Demonstration n Error analysis (Comparison with PRT) Future directions n Limitations n Areas for research

Overview Matrix formulation of relighting n Pre-computing the light-transport matrix n Compressing the matrix multiplication Results n Demonstration n Error analysis (Comparison with PRT) Future directions n Limitations n Areas for research

Matrix Multiplication is Enormous Dimension n 512 x 512 pixel images n 6 x 64 x 64 cubemap environments Full matrix-vector multiplication is intractable n On the order of operations per frame

Sparse Matrix-Vector Multiplication Choose data representations with mostly zeroes Vector: Use non-linear wavelet approximation on lighting Matrix: Wavelet-encode transport rows

Non-linear Wavelet Light Approximation Wavelet Transform

Non-linear Wavelet Light Approximation Non-linear Approximation Retain 0.1% – 1% terms

Why Non-linear Approximation? Linear n Use a fixed set of approximating functions n Precomputed radiance transfer uses of the lowest frequency spherical harmonics Non-linear n Use a dynamic set of approximating functions (depends on each frame’s lighting) n In our case: choose 10’s - 100’s from a basis of 24,576 wavelets Idea: Compress lighting by considering input data

Why Wavelets? Wavelets provide dual space / frequency locality n Large wavelets capture low frequency, area lighting n Small wavelets capture high frequency, compact features In contrast n Spherical harmonics n Perform poorly on compact lights n Pixel basis n Perform poorly on large area lights

Choosing Non-linear Coefficients Three methods of prioritizing 1. Magnitude of wavelet coefficient n Optimal for approximating lighting 2. Transport-weighted n Biases lights that generate bright images 3. Area-weighted n Biases large lights

Sparse Matrix-Vector Multiplication Choose data representations with mostly zeroes Vector: Use non-linear wavelet approximation on lighting Matrix: Wavelet-encode transport rows Choose data representations with mostly zeroes Vector: Use non-linear wavelet approximation on lighting Matrix: Wavelet-encode transport rows

Matrix Row Wavelet Encoding

Extract Row

Matrix Row Wavelet Encoding Wavelet Transform

Matrix Row Wavelet Encoding Wavelet Transform

Matrix Row Wavelet Encoding Wavelet Transform

Matrix Row Wavelet Encoding Wavelet Transform

Matrix Row Wavelet Encoding Store Back in Matrix 0 ’ ’ 0 0

Matrix Row Wavelet Encoding 0 ’ ’ 0 0 Only 3% – 30% are non-zero

Total Compression Lighting vector compression n Highly lossy n Compress to 0.1% – 1% Matrix compression n Essentially lossless encoding n Represent with 3% – 30% non-zero terms Total compression in sparse matrix-vector multiply n 3 – 4 orders of magnitude less work than full multiplication

Overall Relighting Algorithm Pre-compute (per scene) n Compute matrix in pixel basis n Wavelet transform rows n Quantize, store Interactive Relighting (each frame) n Wavelet transform lighting n Prioritize and retain N wavelet coefficients n Perform sparse-matrix vector multiplication

Overview Matrix formulation of relighting n Pre-computing the light-transport matrix n Compressing the matrix multiplication Results n Demonstration n Error analysis (Comparison with PRT) Future directions n Limitations n Areas for research

Overview Matrix formulation of relighting n Pre-computing the light-transport matrix n Compressing the matrix multiplication Results n Demonstration n Error analysis (Comparison with PRT) Future directions n Limitations n Areas for research

Demo

Error Analysis Compare to linear spherical harmonic approximation n [Sloan, Kautz, Snyder, 2002] Measure approximation quality in two ways: n Error in lighting n Error in output image pixels Main point (for detailed natural lighting) n Non-linear wavelets converge exponentially faster than linear harmonics

Error in Lighting: St Peter’s Basilica Approximation Terms Relative L 2 Error (%) Sph. Harmonics Non-linear Wavelets

Error in Output Image: Plant Scene Approximation Terms Relative L 2 Error (%) Sph. Harmonics Non-linear Wavelets

Output Image Comparison Top: Linear Spherical Harmonic Approximation Bottom: Non-linear Wavelet Approximation ,00020,000

Summary A viable representation for all-frequency lighting n Sparse non-linear wavelet approximation n 100 – 1000 times information reduction Efficient relighting from detailed environment maps

Future Improvements Matrix compression n Transport matrices are very large (over 1 GB) n We only compress matrix rows n Can also compress columns (images) Non-linear coefficient selection n Area-weighted priority works best n Develop theory for how to approximate input vector for minimum error in output vector

Signal Processing Taxonomy of Lighting Effects n “All-frequency” vs “Exclusively low frequency”

Future All-Frequency Lighting Research n Realistic materials with changing view n Local lighting effects Gehry’s E.M.P. Museum

Acknowledgments

Relighting with Changing View Matrix formulation still works n Output vector contains exit radiance at scene vertices rather than image pixels Main difference: Must assume diffuse surfaces

Plant Reference

Teapot Reference

Lighting Error

Light-Transport Matrix Columns

Light-Transport Matrix

Matrix Row Wavelet Encoding

Extract Row

Matrix Row Wavelet Encoding Wavelet Transform

Matrix Row Wavelet Encoding Wavelet Transform

Matrix Row Wavelet Encoding Wavelet Transform

Matrix Row Wavelet Encoding Wavelet Transform

Matrix Row Wavelet Encoding Store back in matrix

From Frank Gehry Architecture, Ragheb ed Focus on Shadows