titre

A Frequency Analysis of Light Transport Titre 2 F. Durand, MIT CSAIL N. Holzschuch, C. Soler, ARTIS/GRAVIR-IMAG INRIA E. Chan, MIT CSAIL F. Sillion, ARTIS/GRAVIR-IMAG INRIA

From [Ramamoorthi and Hanrahan 2001] Illumination effects Blurry reflections: From [Ramamoorthi and Hanrahan 2001] Good morning. The topic of my talk is illumination and light transport. You all know some interesting illumination effects: for example, a glossy reflection function blurs the environment, as you can see here. On a specular receiver, you can identify all the features of the environment, while on glossy receivers, the environment becomes less and less identifiable. You can notice that the identifiable features on the receivers become larger and larger as the receivers become more diffuse.

Illumination effects Shadow boundaries: Point light source Similarly, shadow boundaries can be either vary sharp, for example in the case of point light sources, or when the shadow casting object touches the receiver. They can also be blurry, when the shadow casting objects are small with respect to the light source, or far from the receiver. In certain circumstances, the shadow boundary disappears completely. Point light source Area light source © U. Assarsson 2005.

Illumination effects Indirect lighting is usually blurry: Another interesting illumination effect concerns indirect lighting. If you separate the indirect component of illumination, it is usually much more blurry than the direct illumination itself. Here, you can see a room which is almost entirely lit by indirect lighting. Complete lighting

Indirect lighting only Illumination effects Indirect lighting is usually blurry: Another interesting illumination effect concerns indirect lighting. If you separate the indirect component of illumination, it is usually much more blurry than the direct illumination itself. Here, you can see a room which is almost entirely lit by indirect lighting. Direct lighting only Indirect lighting only

Frequency aspects of light transport Blurriness = frequency content Sharp variations: high frequency Smooth variations: low frequency All effects are expressed as frequency content: Diffuse shading: low frequency Shadows: introduce high frequencies Indirect lighting: tends to be low frequency Actually, all these effects can be expressed as frequency content. The size of the features is related to the Fourier spectrum of the illumination function. When illumination contains small features, it is said to be of high frequency, and inversely, if it contains only large features, it is said to be low frequency. All the effects you have seen can be expressed as frequency content. For example, diffuse shading is low frequency, shadows and occlusions introduce higher frequencies in the model, and indirect lighting is usually a low frequency phenomenon.

Problem statement How does light interaction in a scene explain the frequency content? Theoretical framework: Understand the frequency spectrum of the radiance function From equations of light transport Diapo très importante. Pause de 5 secondes après chaque phrase. Goal, not problem Our goal is to understand the relationship between this frequency content and the light interactions in the scene. To put things differently, we will study how the light is transported and interacts in a scene, and deduct from that the frequency spectrum of the radiance function. Our starting point will therefore be the mathematical equations of light transport, and our end point will be the Fourier spectrum of the illumination in the scene.

Unified framework: Spatial frequency (e.g. shadows, textures) Angular frequency (e.g. blurry highlight) Bien souligner que c’est U-NI-FIE. In this talk, we are going to derive a u-ni-fied framework for both spatial and angular frequencies. We will show in which circumstances high frequencies appear in the spatial or angular domains, and in which circumstances high frequencies can disappear. We will also show that in certain circumstances, spatial frequencies can be turned into angular frequencies, or the opposite.

Disclaimer: not Fourier optics We do not consider wave optics, interference, diffraction Only geometrical optics Change slide at ‘NOT’ From [Hecht]

Disclaimer: not Fourier optics We do not consider wave optics, interference, diffraction Only geometrical optics Before discussing relevant work, I must point out that although we are talking about frequency content, we are not interested in the wave nature of light. We do not discuss wave optics, interference, diffraction… only geometrical optics. Bien qu’on parle de frequences, on ne parle pas de wave effects From [Hecht]

Overview Previous work Our approach: Case studies: Local light field Transformations on local light field Case studies: Diffuse soft shadows Adaptive shading sampling Conclusions and future directions My talk will be organized like this: First, I will review some of the previous research. Then, I will present our framework for frequency analysis of light transport. Our framework is based on an object called the “local light field”, which I will first present in great details. Then, I will present the effects on this local light field of some of the operators of light transport. We will see what happens when light travels in free space, when it is occluded, when it is reflected by a curved surface, and finally what is the effect of the BRDF. Our framework has many interesting consequences; I will discuss two of them in greater details. First, we will see how it can be applied to the case of diffuse soft shadows, and how it explains the frequency effects we usually observe with many small occluders. Next, I will present a small application that uses our framework for adaptive shading sampling, speeding up illumination computations. Finally, I will conclude and present some future directions of research.

Overview Previous work Our approach: Case studies: Local light field Transformations on local light field Case studies: Diffuse soft shadows Adaptive shading sampling Conclusions and future directions And let’s start with previous work.

Previous work Vast body of literature: Light field sampling Perceptually-based rendering Wavelets for Computer Graphics Irradiance caching Photon mapping … We focus on frequency analysis in graphics: Reflection as a convolution Much effort has already been dedicated to frequency effects in light transport and illumination computations. For example, all multi-scale and wavelets methods can be seen as attempts to obtain the ideal sampling rate. Similarly, irradiance caching and photon mapping methods seeks to exploit the well-known frequency contents of lighting effects, adapting the sampling rate and the sampling method to the frequency. For example, photon mapping uses a different photon map for indirect lighting (which is low frequency) and for caustics (which are usually high frequency). I will focus more closely on two topics that are more closely related to our analysis: light field sampling and reflection as a convolution.

Light field sampling [Chai et al. 00, Isaksen et al. 00, Stewart et al. 03] Light field spectrum as a function of object distance No BRDF, occlusion ignored A first related research topic is light field sampling. The authors tried to adapt the sampling of the light field, to obtain an ideal sampling. They found that the optimal sampling rate was proportional to the inverse of object distance. Their work is more limited than ours, since they only consider the effects of distance, and ignore occlusion or BRDF effects. From [Chai et al. 2000]

Signal processing for reflection [Ramamoorthi & Hanrahan 01, Basri & Jacobs 03] Reflection on a curved surface is a convolution Direction only In a separate work, it has been shown that reflection on a curved surface is a convolution between the incoming light signal and the BRDF of the receiver. This explains why the incoming signal can be blurred on glossy and diffuse receivers. Their work is more limited than ours, since they only consider angular frequencies; in this example, the object being reflected on the ball is assumed to be at infinity. There is no possibility to combine angular frequencies with spatial frequencies, or to go from one to the other. From [Ramamoorthi and Hanrahan 2001]

Overview Previous work Our approach: Case studies: Local light field Transformations on local light field Case studies: Diffuse soft shadows Adaptive shading sampling Conclusions and future directions Now, let’s see our approach.

Our approach Light sources are input signal Interactions are filters/transforms Transport Visibility BRDF Etc. In our approach we define each operation that can happen to the light in a scene as a transform or a black box that acts on the signal. By combining the different transforms, we are able to predict the frequency content of the output signal. In this chain of transforms, the input signal is given by the light source, of course.

Our approach Light sources are input signal Interactions are filters/transforms Transport Visibility BRDF Etc. Light source signal In our approach we define each operation that can happen to the light in a scene as a transform or a black box that acts on the signal. By combining the different transforms, we are able to predict the frequency content of the output signal. In this chain of transforms, the input signal is given by the light source, of course.

Our approach Light sources are input signal Interactions are filters/transforms Transport Visibility BRDF Etc. Light source signal Transport In our approach we define each operation that can happen to the light in a scene as a transform or a black box that acts on the signal. By combining the different transforms, we are able to predict the frequency content of the output signal. In this chain of transforms, the input signal is given by the light source, of course.

Our approach Light sources are input signal Interactions are filters/transforms Transport Visibility BRDF Etc. Light source signal Transport In our approach we define each operation that can happen to the light in a scene as a transform or a black box that acts on the signal. By combining the different transforms, we are able to predict the frequency content of the output signal. In this chain of transforms, the input signal is given by the light source, of course. Signal 1

Our approach Light sources are input signal Interactions are filters/transforms Transport Visibility BRDF Etc. Light source signal Transport Occlusion In our approach we define each operation that can happen to the light in a scene as a transform or a black box that acts on the signal. By combining the different transforms, we are able to predict the frequency content of the output signal. In this chain of transforms, the input signal is given by the light source, of course. Signal 2

Our approach Light sources are input signal Interactions are filters/transforms Transport Visibility BRDF Etc. Light source signal Transport Occlusion Transport In our approach we define each operation that can happen to the light in a scene as a transform or a black box that acts on the signal. By combining the different transforms, we are able to predict the frequency content of the output signal. In this chain of transforms, the input signal is given by the light source, of course. Signal 3

Our approach Light sources are input signal Interactions are filters/transforms Transport Visibility BRDF Etc. Light source signal Transport Occlusion Transport In our approach we define each operation that can happen to the light in a scene as a transform or a black box that acts on the signal. By combining the different transforms, we are able to predict the frequency content of the output signal. In this chain of transforms, the input signal is given by the light source, of course. BRDF Signal 4

Local light field 4D light field, around a central ray Central ray

Local light field 4D light field, around a central ray We study its spectrum during transport

Local light field 4D light field, around a central ray We study its spectrum during transport

Local light field 4D light field, around a central ray We study its spectrum during transport

Local light field We give explanations in 2D Local light field is therefore 2D See paper for extension to 3D In this presentation, for the sake of simplicity, I will give explanations only in a 2 dimensional case. In that case, the local light field is also a two-dimensional object. I refer you to the paper for the extension of our framework to a 3 dimensional scene.

Local light field parameterization Space and angle angle space Central ray

Local light field representation Density plot: Area light source The local light field will be represented as a density plot, with the spatial dimension on the horizontal axis, and the angular dimension on the vertical axis. Cas particulier. For example. Faïllenaïte For example, here you have the local light field of an area spot light, which occupies an area in space. Angle Space

Local light field Fourier spectrum We are interested in the Fourier spectrum of the local light field Also represented as a density plot We are also interested in the Fourier spectrum of the local light field. This spectrum will also be represented as a density plot, with the spatial frequencies on the horizontal axis and the angular frequencies on the vertical axis. Pause Next slide

Local light field Fourier spectrum Spectrum of an area light source: Angular frequency Here, you can see the Fourier spectrum of the spot light in the previous example. Its Fourier spectrum in the spatial dimension is a sine cardinal, and its Fourier spectrum in the angular dimension is another Gaussian. You can see here that the example contains high spatial frequencies, corresponding to the sharp cutoff at the spatial boundary, and a small number of angular frequencies. In this presentation, you will see many plots like this one, so it is interesting to pause and look at it more closely. What it represents is a combination of spatial and angular frequencies. Something that occurs far on the horizontal axis corresponds to a high spatial frequency, that is a sharp feature. Something that occurs at the center of the horizontal axis corresponds to a low spatial frequency, that is a blurry feature. Same for the vertical axis and angular frequency. Here, the frequencies are mostly on the horizontal axis, because we have an area spot light, sharp features in the spatial dimension, blurry features in the angular dimension. Spatial frequency

Fourier analysis 101 Spectrum corresponds to blurriness: Sharpest feature has size 1/Fmax Convolution theorem: Multiplication  convolution Classical spectra: Box  sinc Dirac  constant Before we start with the main part of the paper, I think it is worth remembering some of the most important points of Fourier analysis. First of all, the Fourier spectrum of a function corresponds to the feature size in the function. The smallest feature in the function is of size 1/Fmax, where Fmax is the largest frequency in the Fourier spectrum. Obviously, the function should be sampled at intervals of 1/Fmax. Second, the Fourier analysis is closely related with the integral tool called convolution. When you multiply two functions, the Fourier spectrum of their product is the convolution of their spectra. Inversely, when you take the convolution of two functions, the Fourier spectrum of the result is the multiplication of their spectra. Finally, it is worth remembering that the Fourier transform of a box function is a sine cardinal (and reciprocally, the Fourier transform of a sine cardinal is a box function), and the Fourier transform of a constant function is a Dirac, and reciprocally.

Overview Previous work Our approach: Case studies: Local light field Transformations on local light field Case studies: Diffuse soft shadows Adaptive shading sampling Conclusions and future directions Now, let’s see our approach.

Overview Previous work Our approach: Case studies Local light field Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions Now, let’s see how our algorithm applies to the main operations of light transport in a scene: transport, occlusion, curvature and BRDF.

Example scene Blockers Receiver Light source EXAMPLE Scene We will be considering a sample scene, with an area light source, travel in free space, a set of blockers, another travel in free space after the blockers and finally reflection by a receiver. Receiver Light source

Overview Previous work Our approach: Case studies Local light field Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions Now, let’s see how our algorithm applies to the main operations of light transport in a scene: transport, occlusion, curvature and BRDF.

Transport in free space Shear Angle Angle Space Space As you will see in the following video, transport in free space corresponds to a shear in the spatial dimension: the angular dimension is left unchanged, and the spatial content is moved in the angular dimension. The effect on the Fourier spectrum is also a shear, but in the other dimension.

Transport in free space Shear Angle Angle Space Space As you will see in the following video, transport in free space corresponds to a shear in the spatial dimension: the angular dimension is left unchanged, and the spatial content is moved in the angular dimension. The effect on the Fourier spectrum is also a shear, but in the other dimension. Shear Angular freq. Angular freq. Spatial frequency Spatial frequency

Transport  Shear This is consistent with light field spectra [Chai et al. 00, Isaksen et al. 00] This result is consistent with previous studies on light field parameterization, which found that the optimal sampling rate for a light field was related to the inverse of the distance to the object. From [Chai et al. 2000]

Overview Previous work Our approach: Case studies Local light field Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions Now, let’s see how our algorithm applies to the main operations of light transport in a scene: transport, occlusion, curvature and BRDF.

Occlusion: multiplication Occlusion is a multiplication in ray space Convolution in Fourier space Creates new spatial frequency content Related to the spectrum of the blockers As you will see in the following video, occlusion corresponds to a multiplication in ray space. As we have seen, this is equivalent to a convolution in Fourier space. The main result is to create new spatial frequency content. The higher the frequency content in the blocker spectrum, the higher the spatial frequencies in the signal after occlusion.

Overview Previous work Our approach: Case studies Local light field Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions Now, let’s see how our algorithm applies to the main operations of light transport in a scene: transport, occlusion, curvature and BRDF.

Overview Previous work Our approach: Case studies Local light field Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions Now, let’s see how our algorithm applies to the main operations of light transport in a scene: transport, occlusion, curvature and BRDF.

BRDF integration Outgoing light: Integration of incoming light times BRDF BRDF Incoming light Outgoing light

BRDF integration Ray-space: convolution Fourier domain: multiplication Outgoing light: convolution of incoming light and BRDF For rotationally-invariant BRDFs Fourier domain: multiplication Outgoing spectrum: multiplication of incoming spectrum and BRDF spectrum For well-behaved BRDFs, this corresponds to a convolution. The outgoing light is equal to the convolution of the incoming light and the BRDF of the receiver. As we have seen, this means that the spectrum of the outgoing light is the product of the spectrum of the incoming light and the spectrum of the BRDF. If the BRDF is spatially constant, then the convolution and the multiplication only take place in the angular dimension.

BRDF in Fourier: multiplication  = Le dire plus, que c’est bandwidth-limiting. BRDF is bandwidth-limiting in angle

 = Example: diffuse BRDF Convolve by constant: multiply by Dirac Only spatial frequencies remain  =

Overview Previous work Our approach: Case studies Local light field Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions Now, let’s see how our algorithm applies to the main operations of light transport in a scene: transport, occlusion, curvature and BRDF.

Curved receiver Reduce to the case of a planar surface: “Unroll” the curved receiver Equivalent to changing angular content: Linear effect on angular dimension No effect on spatial dimension Shear in the angular dimension

Transforms: summary Radiance/Fourier Effect Transport Shear Occlusion Multiplication/Convolution Adds spatial frequencies BRDF Convolution/Multiplication Removes angular frequencies Curvature

More effects in paper Cosine term (multiplication/convolution) Fresnel term (multiplication/convolution) Texture mapping (multiplication/convolution) Central incidence angle (scaling) Separable BRDF Spatially varying BRDF (semi-convolution) …and extension to 3D

Overview Previous work Our approach: Case studies: Local light field Transformations on local light field Case studies: Diffuse soft shadows Adaptive shading sampling Conclusions and future directions Now, let’s see our approach.

Overview Previous work Our approach: Case studies: Local light field Transformations on local light field Case studies: Diffuse soft shadows Adaptive shading sampling Conclusions and future directions Now, let’s see our approach.

Diffuse soft shadows Decreasing blockers size: First high-frequencies increase Then only low frequency Non-monotonic behavior

Diffuse soft shadows (2) Occlusion : convolution in Fourier creates high frequencies Blockers scaled down  spectrum scaled up Fourier space Wv (angle) Wx (space) blocker spectrum

Diffuse soft shadows (3) Transport after occlusion: Spatial frequencies moved to angular dimension Diffuse reflector: Angular frequencies are cancelled Faire apparaitre les machins ronds Ronds correspondents à lobes qui se voient Disparition des lobes liés à la distance parcourue.

Diffuse soft shadows (3) Transport after occlusion: Spatial frequencies moved to angular dimension Diffuse reflector: Angular frequencies are cancelled

Overview Previous work Our approach: Case studies: Local light field Transformations on local light field Case studies: Diffuse soft shadows Adaptive shading sampling Conclusions and future directions Now, let’s see our approach. Ah-DAH-ptive

Adaptive shading sampling Monte-Carlo ray tracing Blurry regions need fewer shading samples Another application we designed is… The idea is that blurry regions should get less shading samples.

Adaptive shading sampling Per-pixel prediction of max. frequency (bandwidth) Based on curvature, BRDF, distance to occluder, etc. No spectrum computed, just estimate max frequency Per-pixel bandwidth criterion

Adaptive shading sampling Per-pixel prediction of max. frequency (bandwidth) Based on curvature, BRDF, distance to occluder, etc. No spectrum computed, just estimate max frequency Shading samples

Adaptive sampling Adaptive 20,000 samples

Uniform sampling 20,000 samples

Overview Previous work Our approach: Case studies: Local light field Transformations on local light field Case studies: Diffuse soft shadows Adaptive shading sampling Conclusions and future directions Now, let’s see our approach.

Conclusions Unified framework: Explains interesting lighting effects: For frequency analysis of radiance In both space and angle Simple mathematical operators Extends previous analyses Explains interesting lighting effects: Soft shadows, caustics Proof-of-concept: Adaptive sampling

Future work More experimental validation on synthetic scenes Extend the theory: Bump mapping, microfacet BRDFs, sub-surface scattering… Participating media Applications to rendering: Photon mapping Spatial sampling for PRT Revisit traditional techniques Applications to vision and shape from shading

Acknowledgments Jaakko Lehtinen Reviewers of the MIT and ARTIS graphics groups Siggraph reviewers This work was supported in part by: NSF CAREER award 0447561 Transient Signal Processing for Realistic Imagery NSF CISE Research Infrastructure Award (EIA9802220) ASEE National Defense Science and Engineering Graduate fellowship INRIA Équipe associée Realreflect EU IST project MIT-France

Penser à mettre à jour le plan

Solar oven Curved surface In: parallel light rays Out: focal point

Other bases? We’re not using Fourier as a function basis Don’t recommend it, actually Just used for analysis, understanding, predictions Results are useable with any other base: Wavelets, Spherical Harmonics, point sampling, etc Max. frequency translates in sampling rate Analysis relies on Fourier properties: Especially the convolution theorem

Why Local Light Field? Linearization:   tan  Curvature Local information is what we need: Local frequency content, for local sampling Local properties of the scene (occluders, curv.)

Extension to 3D It works. See paper:

Reflection on a surface: Full summary Angle of incidence Curvature Cosine/Fresnel term Mirror re-parameterization BRDF

Reflection on a surface: Full summary Angle of incidence: scaling Curvature: shear in angle Cosine/Fresnel term: multiplication/convolution Mirror re-parameterization BRDF: convolution/multiplication

Application: 3D scene

Application: 3D scene

Application: 3D scene

Application: 3D scene

Application: 3D scene