1. titre

2. A Frequency Analysis of Light Transport
Titre 2
F. Durand, MIT CSAIL
N. Holzschuch, C. Soler, ARTIS/GRAVIR-IMAG INRIA
E. Chan, MIT CSAIL
F. Sillion, ARTIS/GRAVIR-IMAG INRIA

3. From [Ramamoorthi and Hanrahan 2001]
Illumination effects
Blurry reflections:
From [Ramamoorthi and Hanrahan 2001]
Good morning. The topic of my talk is illumination and light transport. You all know some interesting illumination effects: for example, a glossy reflection function blurs the environment, as you can see here. On a specular receiver, you can identify all the features of the environment, while on glossy receivers, the environment becomes less and less identifiable. You can notice that the identifiable features on the receivers become larger and larger as the receivers become more diffuse.

4. Illumination effects Shadow boundaries: Point light source
Similarly, shadow boundaries can be either vary sharp, for example in the case of point light sources, or when the shadow casting object touches the receiver.
They can also be blurry, when the shadow casting objects are small with respect to the light source, or far from the receiver. In certain circumstances, the shadow boundary disappears completely.
Point light source
Area light source
© U. Assarsson 2005.

5. Illumination effects Indirect lighting is usually blurry:
Another interesting illumination effect concerns indirect lighting. If you separate the indirect component of illumination, it is usually much more blurry than the direct illumination itself. Here, you can see a room which is almost entirely lit by indirect lighting.
Complete lighting

6. Indirect lighting only
Illumination effects
Indirect lighting is usually blurry:
Another interesting illumination effect concerns indirect lighting. If you separate the indirect component of illumination, it is usually much more blurry than the direct illumination itself. Here, you can see a room which is almost entirely lit by indirect lighting.
Direct lighting only
Indirect lighting only

7. Frequency aspects of light transport
Blurriness = frequency content
Sharp variations: high frequency
Smooth variations: low frequency
All effects are expressed as frequency content:
Diffuse shading: low frequency
Shadows: introduce high frequencies
Indirect lighting: tends to be low frequency
Actually, all these effects can be expressed as frequency content. The size of the features is related to the Fourier spectrum of the illumination function. When illumination contains small features, it is said to be of high frequency, and inversely, if it contains only large features, it is said to be low frequency. All the effects you have seen can be expressed as frequency content.
For example, diffuse shading is low frequency, shadows and occlusions introduce higher frequencies in the model, and indirect lighting is usually a low frequency phenomenon.

8. Problem statement
How does light interaction in a scene explain the frequency content?
Theoretical framework:
Understand the frequency spectrum of the radiance function
From equations of light transport
Diapo très importante.
Pause de 5 secondes après chaque phrase.
Goal, not problem
Our goal is to understand the relationship between this frequency content and the light interactions in the scene.
To put things differently, we will study how the light is transported and interacts in a scene, and deduct from that the frequency spectrum of the radiance function.
Our starting point will therefore be the mathematical equations of light transport, and our end point will be the Fourier spectrum of the illumination in the scene.

9. Unified framework: Spatial frequency (e.g. shadows, textures)
Angular frequency (e.g. blurry highlight)
Bien souligner que c’est U-NI-FIE.
In this talk, we are going to derive a u-ni-fied framework for both spatial and angular frequencies. We will show in which circumstances high frequencies appear in the spatial or angular domains, and in which circumstances high frequencies can disappear.
We will also show that in certain circumstances, spatial frequencies can be turned into angular frequencies, or the opposite.

10. Disclaimer: not Fourier optics
We do not consider wave optics, interference, diffraction
Only geometrical optics
Change slide at ‘NOT’
From [Hecht]

11. Disclaimer: not Fourier optics
We do not consider wave optics, interference, diffraction
Only geometrical optics
Before discussing relevant work, I must point out that although we are talking about frequency content, we are not interested in the wave nature of light. We do not discuss wave optics, interference, diffraction… only geometrical optics.
Bien qu’on parle de frequences, on ne parle pas de wave effects
From [Hecht]

12. Overview Previous work Our approach: Case studies:
Local light field
Transformations on local light field
Case studies:
Diffuse soft shadows
Adaptive shading sampling
Conclusions and future directions
My talk will be organized like this:
First, I will review some of the previous research.
Then, I will present our framework for frequency analysis of light transport. Our framework is based on an object called the “local light field”, which I will first present in great details. Then, I will present the effects on this local light field of some of the operators of light transport. We will see what happens when light travels in free space, when it is occluded, when it is reflected by a curved surface, and finally what is the effect of the BRDF.
Our framework has many interesting consequences; I will discuss two of them in greater details. First, we will see how it can be applied to the case of diffuse soft shadows, and how it explains the frequency effects we usually observe with many small occluders. Next, I will present a small application that uses our framework for adaptive shading sampling, speeding up illumination computations.
Finally, I will conclude and present some future directions of research.

13. Overview Previous work Our approach: Case studies:
Local light field
Transformations on local light field
Case studies:
Diffuse soft shadows
Adaptive shading sampling
Conclusions and future directions
And let’s start with previous work.

14. Previous work Vast body of literature:
Light field sampling
Perceptually-based rendering
Wavelets for Computer Graphics
Irradiance caching
Photon mapping …
We focus on frequency analysis in graphics:
Reflection as a convolution
Much effort has already been dedicated to frequency effects in light transport and illumination computations. For example, all multi-scale and wavelets methods can be seen as attempts to obtain the ideal sampling rate. Similarly, irradiance caching and photon mapping methods seeks to exploit the well-known frequency contents of lighting effects, adapting the sampling rate and the sampling method to the frequency. For example, photon mapping uses a different photon map for indirect lighting (which is low frequency) and for caustics (which are usually high frequency).
I will focus more closely on two topics that are more closely related to our analysis: light field sampling and reflection as a convolution.

15. Light field sampling
[Chai et al. 00, Isaksen et al. 00, Stewart et al. 03]
Light field spectrum as a function of object distance
No BRDF, occlusion ignored
A first related research topic is light field sampling. The authors tried to adapt the sampling of the light field, to obtain an ideal sampling. They found that the optimal sampling rate was proportional to the inverse of object distance.
Their work is more limited than ours, since they only consider the effects of distance, and ignore occlusion or BRDF effects.
From [Chai et al. 2000]

16. Signal processing for reflection
[Ramamoorthi & Hanrahan 01, Basri & Jacobs 03]
Reflection on a curved surface is a convolution
Direction only
In a separate work, it has been shown that reflection on a curved surface is a convolution between the incoming light signal and the BRDF of the receiver. This explains why the incoming signal can be blurred on glossy and diffuse receivers. Their work is more limited than ours, since they only consider angular frequencies; in this example, the object being reflected on the ball is assumed to be at infinity. There is no possibility to combine angular frequencies with spatial frequencies, or to go from one to the other.
From [Ramamoorthi and Hanrahan 2001]

17. Overview Previous work Our approach: Case studies:
Local light field
Transformations on local light field
Case studies:
Diffuse soft shadows
Adaptive shading sampling
Conclusions and future directions
Now, let’s see our approach.

18. Our approach Light sources are input signal
Interactions are filters/transforms
Transport
Visibility
BRDF
Etc.
In our approach we define each operation that can happen to the light in a scene as a transform or a black box that acts on the signal. By combining the different transforms, we are able to predict the frequency content of the output signal.
In this chain of transforms, the input signal is given by the light source, of course.

19. Our approach Light sources are input signal
Interactions are filters/transforms
Transport
Visibility
BRDF
Etc.
Light source signal
In our approach we define each operation that can happen to the light in a scene as a transform or a black box that acts on the signal. By combining the different transforms, we are able to predict the frequency content of the output signal.
In this chain of transforms, the input signal is given by the light source, of course.

20. Our approach Light sources are input signal
Interactions are filters/transforms
Transport
Visibility
BRDF
Etc.
Light source signal
Transport
In our approach we define each operation that can happen to the light in a scene as a transform or a black box that acts on the signal. By combining the different transforms, we are able to predict the frequency content of the output signal.
In this chain of transforms, the input signal is given by the light source, of course.

21. Our approach Light sources are input signal
Interactions are filters/transforms
Transport
Visibility
BRDF
Etc.
Light source signal
Transport
In our approach we define each operation that can happen to the light in a scene as a transform or a black box that acts on the signal. By combining the different transforms, we are able to predict the frequency content of the output signal.
In this chain of transforms, the input signal is given by the light source, of course.
Signal 1

22. Our approach Light sources are input signal
Interactions are filters/transforms
Transport
Visibility
BRDF
Etc.
Light source signal
Transport
Occlusion
In our approach we define each operation that can happen to the light in a scene as a transform or a black box that acts on the signal. By combining the different transforms, we are able to predict the frequency content of the output signal.
In this chain of transforms, the input signal is given by the light source, of course.
Signal 2

23. Our approach Light sources are input signal
Interactions are filters/transforms
Transport
Visibility
BRDF
Etc.
Light source signal
Transport
Occlusion
Transport
In our approach we define each operation that can happen to the light in a scene as a transform or a black box that acts on the signal. By combining the different transforms, we are able to predict the frequency content of the output signal.
In this chain of transforms, the input signal is given by the light source, of course.
Signal 3

24. Our approach Light sources are input signal
Interactions are filters/transforms
Transport
Visibility
BRDF
Etc.
Light source signal
Transport
Occlusion
Transport
In our approach we define each operation that can happen to the light in a scene as a transform or a black box that acts on the signal. By combining the different transforms, we are able to predict the frequency content of the output signal.
In this chain of transforms, the input signal is given by the light source, of course.
BRDF
Signal 4

25. Local light field
4D light field, around a central ray
Central ray

26. Local light field 4D light field, around a central ray
We study its spectrum during transport

27. Local light field 4D light field, around a central ray
We study its spectrum during transport

28. Local light field 4D light field, around a central ray
We study its spectrum during transport

29. Local light field We give explanations in 2D
Local light field is therefore 2D
See paper for extension to 3D
In this presentation, for the sake of simplicity, I will give explanations only in a 2 dimensional case. In that case, the local light field is also a two-dimensional object. I refer you to the paper for the extension of our framework to a 3 dimensional scene.

30. Local light field parameterization
Space and angle
angle
space
Central ray

31. Local light field representation
Density plot:
Area light source
The local light field will be represented as a density plot, with the spatial dimension on the horizontal axis, and the angular dimension on the vertical axis.
Cas particulier. For example. Faïllenaïte
For example, here you have the local light field of an area spot light, which occupies an area in space.
Angle
Space

32. Local light field Fourier spectrum
We are interested in the Fourier spectrum of the local light field
Also represented as a density plot
We are also interested in the Fourier spectrum of the local light field. This spectrum will also be represented as a density plot, with the spatial frequencies on the horizontal axis and the angular frequencies on the vertical axis.
Pause
Next slide

33. Local light field Fourier spectrum
Spectrum of an area light source:
Angular frequency
Here, you can see the Fourier spectrum of the spot light in the previous example. Its Fourier spectrum in the spatial dimension is a sine cardinal, and its Fourier spectrum in the angular dimension is another Gaussian.
You can see here that the example contains high spatial frequencies, corresponding to the sharp cutoff at the spatial boundary, and a small number of angular frequencies.
In this presentation, you will see many plots like this one, so it is interesting to pause and look at it more closely. What it represents is a combination of spatial and angular frequencies. Something that occurs far on the horizontal axis corresponds to a high spatial frequency, that is a sharp feature. Something that occurs at the center of the horizontal axis corresponds to a low spatial frequency, that is a blurry feature. Same for the vertical axis and angular frequency. Here, the frequencies are mostly on the horizontal axis, because we have an area spot light, sharp features in the spatial dimension, blurry features in the angular dimension.
Spatial frequency

34. Fourier analysis 101 Spectrum corresponds to blurriness:
Sharpest feature has size 1/Fmax
Convolution theorem:
Multiplication  convolution
Classical spectra:
Box  sinc
Dirac  constant
Before we start with the main part of the paper, I think it is worth remembering some of the most important points of Fourier analysis.
First of all, the Fourier spectrum of a function corresponds to the feature size in the function. The smallest feature in the function is of size 1/Fmax, where Fmax is the largest frequency in the Fourier spectrum. Obviously, the function should be sampled at intervals of 1/Fmax.
Second, the Fourier analysis is closely related with the integral tool called convolution. When you multiply two functions, the Fourier spectrum of their product is the convolution of their spectra. Inversely, when you take the convolution of two functions, the Fourier spectrum of the result is the multiplication of their spectra.
Finally, it is worth remembering that the Fourier transform of a box function is a sine cardinal (and reciprocally, the Fourier transform of a sine cardinal is a box function), and the Fourier transform of a constant function is a Dirac, and reciprocally.

35. Overview Previous work Our approach: Case studies:
Local light field
Transformations on local light field
Case studies:
Diffuse soft shadows
Adaptive shading sampling
Conclusions and future directions
Now, let’s see our approach.

36. Overview Previous work Our approach: Case studies
Local light field
Transformations on local light field
Transport
Occlusion
BRDF
Curvature
Case studies
Conclusions and future directions
Now, let’s see how our algorithm applies to the main operations of light transport in a scene: transport, occlusion, curvature and BRDF.

37. Example scene Blockers Receiver Light source EXAMPLE Scene
We will be considering a sample scene, with an area light source, travel in free space, a set of blockers, another travel in free space after the blockers and finally reflection by a receiver.
Receiver
Light source

38. Overview Previous work Our approach: Case studies
Local light field
Transformations on local light field
Transport
Occlusion
BRDF
Curvature
Case studies
Conclusions and future directions
Now, let’s see how our algorithm applies to the main operations of light transport in a scene: transport, occlusion, curvature and BRDF.

39. Transport in free space
Shear
Angle
Angle
Space
Space
As you will see in the following video, transport in free space corresponds to a shear in the spatial dimension: the angular dimension is left unchanged, and the spatial content is moved in the angular dimension.
The effect on the Fourier spectrum is also a shear, but in the other dimension.

40. Transport in free space
Shear
Angle
Angle
Space
Space
As you will see in the following video, transport in free space corresponds to a shear in the spatial dimension: the angular dimension is left unchanged, and the spatial content is moved in the angular dimension.
The effect on the Fourier spectrum is also a shear, but in the other dimension.
Shear
Angular freq.
Angular freq.
Spatial frequency
Spatial frequency

42. Transport  Shear
This is consistent with light field spectra [Chai et al. 00, Isaksen et al. 00]
This result is consistent with previous studies on light field parameterization, which found that the optimal sampling rate for a light field was related to the inverse of the distance to the object.
From [Chai et al. 2000]

43. Overview Previous work Our approach: Case studies
Local light field
Transformations on local light field
Transport
Occlusion
BRDF
Curvature
Case studies
Conclusions and future directions
Now, let’s see how our algorithm applies to the main operations of light transport in a scene: transport, occlusion, curvature and BRDF.

44. Occlusion: multiplication
Occlusion is a multiplication in ray space
Convolution in Fourier space
Creates new spatial frequency content
Related to the spectrum of the blockers
As you will see in the following video, occlusion corresponds to a multiplication in ray space. As we have seen, this is equivalent to a convolution in Fourier space. The main result is to create new spatial frequency content. The higher the frequency content in the blocker spectrum, the higher the spatial frequencies in the signal after occlusion.

46. Overview Previous work Our approach: Case studies
Local light field
Transformations on local light field
Transport
Occlusion
BRDF
Curvature
Case studies
Conclusions and future directions
Now, let’s see how our algorithm applies to the main operations of light transport in a scene: transport, occlusion, curvature and BRDF.

48. Overview Previous work Our approach: Case studies
Local light field
Transformations on local light field
Transport
Occlusion
BRDF
Curvature
Case studies
Conclusions and future directions
Now, let’s see how our algorithm applies to the main operations of light transport in a scene: transport, occlusion, curvature and BRDF.

49. BRDF integration Outgoing light:
Integration of incoming light times BRDF
BRDF
Incoming light
Outgoing light

50. BRDF integration Ray-space: convolution Fourier domain: multiplication
Outgoing light: convolution of incoming light and BRDF
For rotationally-invariant BRDFs
Fourier domain: multiplication
Outgoing spectrum: multiplication of incoming spectrum and BRDF spectrum
For well-behaved BRDFs, this corresponds to a convolution. The outgoing light is equal to the convolution of the incoming light and the BRDF of the receiver.
As we have seen, this means that the spectrum of the outgoing light is the product of the spectrum of the incoming light and the spectrum of the BRDF.
If the BRDF is spatially constant, then the convolution and the multiplication only take place in the angular dimension.

51. BRDF in Fourier: multiplication =
Le dire plus, que c’est bandwidth-limiting.
BRDF is bandwidth-limiting in angle

52.  = Example: diffuse BRDF Convolve by constant: multiply by Dirac
Only spatial frequencies remain  =

53. Overview Previous work Our approach: Case studies
Local light field
Transformations on local light field
Transport
Occlusion
BRDF
Curvature
Case studies
Conclusions and future directions
Now, let’s see how our algorithm applies to the main operations of light transport in a scene: transport, occlusion, curvature and BRDF.

54. Curved receiver Reduce to the case of a planar surface:
“Unroll” the curved receiver
Equivalent to changing angular content:
Linear effect on angular dimension
No effect on spatial dimension
Shear in the angular dimension

56. Transforms: summary Radiance/Fourier Effect Transport Shear Occlusion
Multiplication/Convolution
Adds spatial frequencies
BRDF
Convolution/Multiplication
Removes angular frequencies
Curvature

57. More effects in paper Cosine term (multiplication/convolution)
Fresnel term (multiplication/convolution)
Texture mapping (multiplication/convolution)
Central incidence angle (scaling)
Separable BRDF
Spatially varying BRDF (semi-convolution)
…and extension to 3D

58. Overview Previous work Our approach: Case studies:
Local light field
Transformations on local light field
Case studies:
Diffuse soft shadows
Adaptive shading sampling
Conclusions and future directions
Now, let’s see our approach.

59. Overview Previous work Our approach: Case studies:
Local light field
Transformations on local light field
Case studies:
Diffuse soft shadows
Adaptive shading sampling
Conclusions and future directions
Now, let’s see our approach.

60. Diffuse soft shadows Decreasing blockers size:
First high-frequencies increase
Then only low frequency
Non-monotonic behavior

61. Diffuse soft shadows (2)
Occlusion : convolution in Fourier
creates high frequencies
Blockers scaled down  spectrum scaled up
Fourier space
Wv (angle)
Wx (space)
blocker spectrum

62. Diffuse soft shadows (3)
Transport after occlusion:
Spatial frequencies moved to angular dimension
Diffuse reflector:
Angular frequencies are cancelled
Faire apparaitre les machins ronds
Ronds correspondents à lobes qui se voient
Disparition des lobes liés à la distance parcourue.

63. Diffuse soft shadows (3)
Transport after occlusion:
Spatial frequencies moved to angular dimension
Diffuse reflector:
Angular frequencies are cancelled

64. Overview Previous work Our approach: Case studies:
Local light field
Transformations on local light field
Case studies:
Diffuse soft shadows
Adaptive shading sampling
Conclusions and future directions
Now, let’s see our approach.
Ah-DAH-ptive

65. Adaptive shading sampling
Monte-Carlo ray tracing
Blurry regions need fewer shading samples
Another application we designed is…
The idea is that blurry regions should get less shading samples.

66. Adaptive shading sampling
Per-pixel prediction of max. frequency (bandwidth)
Based on curvature, BRDF, distance to occluder, etc.
No spectrum computed, just estimate max frequency
Per-pixel bandwidth criterion

67. Adaptive shading sampling
Per-pixel prediction of max. frequency (bandwidth)
Based on curvature, BRDF, distance to occluder, etc.
No spectrum computed, just estimate max frequency
Shading samples

68. Adaptive sampling
Adaptive
20,000 samples

69. Uniform sampling
20,000 samples

70. Overview Previous work Our approach: Case studies:
Local light field
Transformations on local light field
Case studies:
Diffuse soft shadows
Adaptive shading sampling
Conclusions and future directions
Now, let’s see our approach.

71. Conclusions Unified framework: Explains interesting lighting effects:
For frequency analysis of radiance
In both space and angle
Simple mathematical operators
Extends previous analyses
Explains interesting lighting effects:
Soft shadows, caustics
Proof-of-concept:
Adaptive sampling

72. Future work More experimental validation on synthetic scenes
Extend the theory:
Bump mapping, microfacet BRDFs, sub-surface scattering…
Participating media
Applications to rendering:
Photon mapping
Spatial sampling for PRT
Revisit traditional techniques
Applications to vision and shape from shading

73. Acknowledgments Jaakko Lehtinen
Reviewers of the MIT and ARTIS graphics groups
Siggraph reviewers
This work was supported in part by:
NSF CAREER award Transient Signal Processing for Realistic Imagery
NSF CISE Research Infrastructure Award (EIA )
ASEE National Defense Science and Engineering Graduate fellowship
INRIA Équipe associée
Realreflect EU IST project
MIT-France

74. Penser à mettre à jour le plan

75. Solar oven
Curved surface
In: parallel light rays
Out: focal point

76. Other bases? We’re not using Fourier as a function basis
Don’t recommend it, actually
Just used for analysis, understanding, predictions
Results are useable with any other base:
Wavelets, Spherical Harmonics, point sampling, etc
Max. frequency translates in sampling rate
Analysis relies on Fourier properties:
Especially the convolution theorem

77. Why Local Light Field? Linearization:
  tan 
Curvature
Local information is what we need:
Local frequency content, for local sampling
Local properties of the scene (occluders, curv.)

78. Extension to 3D
It works. See paper:

79. Reflection on a surface: Full summary
Angle of incidence
Curvature
Cosine/Fresnel term
Mirror re-parameterization
BRDF

80. Reflection on a surface: Full summary
Angle of incidence: scaling
Curvature: shear in angle
Cosine/Fresnel term: multiplication/convolution
Mirror re-parameterization
BRDF: convolution/multiplication

81. Application: 3D scene

82. Application: 3D scene

83. Application: 3D scene

84. Application: 3D scene

85. Application: 3D scene