1. Frequency Domain Normal Map Filtering
Charles Han
Bo Sun
Ravi Ramamoorthi
Eitan Grinspun
Columbia University
[authors if not mentioned]

2. Normal Mapping (Blinn 78)
At almost 30 years old now, normal mapping is a technique that I’m sure many of you have come across, but since there are several slight variations on the method I’ll just briefly go over it here to establish the form that we will be using in this talk.
As an example shape, let’s use this simple plane. Now, you might be thinking that this plane is boring to look at, and that is mostly due to the fact, that the surface normal is the same everywhere, and we illustrate that here using a schematic view at the bottom. Now, one interesting trick we can use is to

3. Normal Mapping (Blinn 78) Specify surface normals
define a function over the surface that specifies a hopefully more interesting pattern of surface normals.
And this function is what we’ll refer to as the normal map.
Now, if we use these mapped normals for rendering, we can get a much more interesting appearance.
in this example I’ve also added a color map to emphasize the effect.
Geometrically, we still have just a plane, but we’ve created the illusion of much more geometry for just the cost of representing the normal map.
In practice we usually just store the normal map in a texture, so this is a very efficient technique, which has made it popular over the years..
And nowadays, with the development of programmable shaders, this has become increasingly…

4. Normal Mapping
It’s been quite popular over the years… And nowadays with the development of programmable GPU shaders, normal mapping has become especially popular in real-time applications such as games and visualization.

5. A Problem… Multiple normals per pixel Undersampling Filtering needed ?
However, there is a problem, and To illustrate this, let’s use our same normal map from before, and you can’t see just yet but we’ve now applied it to the famous stanford bunny model…
Now so far we’ve had a unique normal mapped to each screen pixel, but watch what happens as we zoom out…
We now have multiple normals per pixel, and so we begin to undersample… terrible aliasing
So It becomes clear that some sort of low-pass filtering is going to be needed.
And this far from being a new problem…. In fact, Blinn even mentions it in the original bumpmapping paper ?

6. Supersampling Correct results Too slow
The solution that he offered then is to supersample the until we’ve captured all of the normal information we need. Now, while this does give us correct results, it can get arbitrarily costly and is really not a practical solution in the context of realtime rendering.

7. MIP mapping Pre-filter Normals do not interpolate linearly
Blurring of details
so now let’s consider the standard filtering method for realtime, which is to prefilter the normal map at different scales and to store the results in a mip map. However, unlike colors, surface normals do not contribute linearly to the rendered color and the standard linear filtering will not produce the correct
result. as you can see in this example, we just end up blurring out all our detais until we arrive at an entirely smooth surface.

8. Comparison supersampled MIP mapped
And you can really see that here in a side-by-side comparison
supersampled
MIP mapped

9. Representation a single vector is not enough
how do we represent multiple surface normals?
What we have here is essentially an issue of representation.
A single normal vector cannot, in the general case, describe the often complex distribution of normals it is trying to represent
So the question then becomes…

10. no general solution Previous Work Gaussian Distributions
(Olano and North 97)
(Schilling 97)
(Toksvig 05)
Mixture Models
(Fournier 92)
(Tan, et.al. 05)
3D Gaussian
2D covariance matrix
1D Gaussian
mixture of Phong lobes
mixture of 2D Gaussians
Now, while single gaussians are easy to fit, they are often inadequate for describing more complex distributions.
For these, others have proposed methods that rely on mixtures of multiple lobes
In a classic paper, Fournier represented groups of normals using phong lobes.
And finally, a recent method by Tan etal uses gaussian mixture models.
Now you’ll notice that each of these methods chooses a particular approximation, implicitly limiting the types of distributions that they can handle.
Moreover, each of these papers also offers a rendering algorithm tailored to its specific normal map representation, and usually also to a specific BRDF…
So, what we see here is the lack of a general solution.
and that has been the state of normal map filtering to date.
no general solution

11. Our Contributions Theoretical Framework New normal map representations
Normal Distribution Function (NDF)
Linear averaging for filtering
Convolution for rendering
Unifies previous works
New normal map representations
Spherical harmonics
von Mises-Fisher Distribution
Simple, efficient rendering algorithms
[new formal convoution formula]
First, as opposed to the situation we saw with a single normal, NDFs can be linearly averaged, permitting us to use it as a correct normal map representation. Also, we’ll see that rendering an NDF amounts to convolving it with a BRDF and lighting, and this result will allow us to separate the notions of normal and BRDF in the context of rendering, something that could not be done before in normal map filtering.
Finally, our theory does not contradict previous work; in retrospect, we can in fact understand existing methods as special cases within our framework.
the understanding we gain from this analysis will inform our decisions in
Choosing new representations, namely the spherical harmonics, which are now ubiquitous in rendering, and the
Von MFD, which to our knowledge is being using here for the first time in graphics.
And we choose these representations b/c at the end of the day they lead to powerful and easily implemented rendering algorithms

12. Normal Distribution Function (NDF)
Describes normals within region
Defined on the unit sphere
Integrates to one
Extended Gaussian Image (Horn 84)
We use an NDF to describe the normals in a given region of the normal map.
It is a scalar-valued function whose domain is the unit sphere.
Being a distribution function, its integral over the sphere is 1.
To be very formal about it, the NDF is the composition of a uniform area sampling function with the Gauss Map, or as Horn has called it, the Extended Gaussian Image.
Now, that probably doesn’t make much sense as is, but fortunately the NDF is better understood by example, so let’s take a look at a few…

13. Normal Distribution Function
The simplest example would be a completely flat normal map. Since we have only one
normal direction in the entire normal map, the NDF in this case is a delta function centered about that normal.
..and I'll pause here to fully explain this NDF visualization, which you'll be seeing more of in this talk. The circle you see here is a top-down view of the unit sphere, with brighter regions corresponding to higher NDF values.
and although the domain is technically the entire sphere, for most normal maps we can focus on just the upper hemisphere.
normal map
NDF

14. Normal Distribution Function
Now here's a slightly more involved normal map, with this sort of tiled pattern.
Most of the normals still point upwards, which shows up in the NDF as the bright region at the top of the sphere. But now, we also have four peaks at
an angle from the upward direction, which come from the sides of the tiles. Also, note that this normal map has a slightly rougher texture to it, since
we've added some noise to the normals. this shows up in the NDF as a broadening of the peaks.
normal map
NDF

15. Normal Distribution Function
.. and just as one last example, we have here our ridged normal map from before, and its corresponding NDF. again we have a good deal of normals
pointing in the upward direction, and also a good spread of normals off to either side, as a result of the rounded bumps.
normal map
NDF

16. Normal Distribution Function
now so far we've looked at the NDFs of entire normal maps, but the NDF is
actually defined over any given region of the normal map. So, for example,
if we take just this highlighted part as the region of interest, then
the corresponding normal map would look like this, with just upward and
rightward-sloping normals.
normal map
NDF

17. NDF Filtering normal map
And this leads us into the subject of filtering with NDFs. So, let’s set aside the NDF for this region for a moment
normal map

18. NDF Filtering normal map
And now look at the NDF for this adjacent region here, which we can see is mostly upward and left-sloping…
normal map

19. NDF Filtering NDF averaging is linear Store NDFs in MIP map
and finally, let's consider the union of these two regions, and its NDF.
we find that it is in fact the linear average of the NDFs from the two subregions.
and this leads us to an important property of NDFs:
NDF averaging is a linear operation. recall that this was exactly the problem that we had with surface normals, which
do not interpolate linearly.
so if we store in each MIP map texel the NDF of its corresponding region,
we can faithfully represent the normal map at all scales.
Now that we've found a good representation for our normal map, it remains
to understand how to render with it.
NDF averaging is linear
Store NDFs in MIP map

20. Rendering Radially symmetric BRDFs Lambertian: Blinn-Phong:
normal,
Radially symmetric BRDFs
Lambertian:
Blinn-Phong:
Torrance-Sparrow:
Factored:
rendered image
Now, let's go all the way back to the basics of rendering. And so here we use the familiar reflectance equation. To find the color of a given
pixel, we multiply all incoming lights with the BRDF. And we limit ourselves to radially symmetric BRDFs, which take as argument the angle
omega dot n, where n is our shading normal, and omega depends on the specific BRDF.
Now, this equation only works as long as we have a well-defined normal for each pixel…, so now let's move to our situation from before, where we had many normals visible in each screen pixel.
pixel value
lights
BRDF

21. Supersampling samples supersampled image Effective BRDF
The correct result,
as you may remember, was obtained by supersampling, where we sample the image at a subpixel resolution. for each of the N samples, we simply render using the same equation as before, then average the results to find our final pixel value....
and we can write that in summation notation.
now, notice that this expression is linear in the BRDF term, so we can perform some simple algebraic regrouping and isolate just the BRDFs.
and look at what we've done. we now have what is basically the reflectance equation again, but with a sort of pseudo-BRDF that depends on all of the
normals that we've sampled. and we'll refer to this quantity as the Effective BRDF of the pixel.
Effective BRDF

22. Effective BRDF samples NDF,
Looking at our equation for the effective BRDF, we notice that it is simply a weighted sum of BRDFs evaluated at the sampled normals.
Now, we can’t know ahead of time what omega will be, so unfortunately this expression still won’t work for us in the context of prefiltering. However, what we do know ahead of time is the NDF of the pixel, which, as we said, can be precomputed and stored in a MIP map. If we consider and evaluate the BRDF at all possible normals, in the limit this sum becomes an integral, with the NDF giving the appropriate weight.
And now we have an expression that tells us how to render using our NDFs… but it gets even better.

23. Spherical Convolution
Form studied in lighting
(Basri and Jacobs 01)
(Ramamoorthi and Hanrahan 01)
Effective BRDF = convolution of NDF & BRDF
Equations of this form have been studied before in lighting, and they have been shown to be in fact convolutions on the sphere.
In our case, this means that the Effective BRDF is in fact the convolution of the NDF w/ the BRDF.

24. Spherical Convolution
We can also understand this idea visually.
Here we see one of our NDFs from before, and also a BRDF.
If we center the BRDF at all points on the sphere and multiply with the NDF, the resulting spherical function is the effective BRDF.
And this is an important result, because it gives us a general solution to normal map filtering. We can use any normal map, and any BRDF, so long as we know how to convolve them.
Effective BRDF
NDF
BRDF

25. Previous Work Gaussian Distributions Mixture Models Our Work
Olano and North (97)
Schilling (97)
Toksvig (05)
Mixture Models
Fournier (92)
Tan, et.al. (05)
Our Work
3D Gaussian
2D covariance matrix
1D Gaussian
mixture of Phong lobes
mixture of 2D Gaussians
NDF representations
And now, let’s take a quick look at the methods we saw before.
we see that their various representations are in fact different approximations Or representations of the NDF.
Also, in retrospect, their respective rendering algorithms can be understood to be
Pre-baked convolutions of their specific NDF model with their specific choice of BRDF.
Moving forward, I will now introduce two new representations based on the theory I just laid out.
Since we’re speaking of spherical convolutions, a natural first choice for our representation would be the spherical harmonics..
spherical harmonics
von Mises-Fisher mixtures

26. Spherical Harmonics Analogous to Fourier basis Convolution formula:
Being the spherical analogue to the Fourier basis, the spherical harmonics
are naturally suited to performing convolutions on the sphere.
Specifically, our convolution for the effective BRDF becomes a simple product in frequency space.
And I should note that I’ve left out a normalizing term here for readability.
Using this simple product formula, we can implement it directly in a shader program, which we’ll walk through now

27. BRDF Coefficients Arbitrary BRDFs Cheaply represented
Analytic: compute in shader
Measured: store on GPU
Easily changed at runtime
First, we have the BRDF coefficients, which you will notice are completely decoupled from the NDF.
So, we can use any BRDF as long as we know its SH coefficients, and these are very inexpensive to represent.
For many BRDFs, analytic formulae are available, and we just compute the coefficients on the fly.
If an analytic form is not available, such as with measured materials, we can simply store the coefficients in GPU registers…
Also, note that this efficiency allows us to even change the BRDF at runtime.

28. NDF Coefficients Store in MIP mapped textures
Finest-level NDFs are delta functions, so:
Use standard linear filtering
Next we have the NDF, which as we said, we’ll store in a MIP map..
And actually we need a mip map for each harmonic lm.
At the finest level texels, note that we have a delta function ndf, since we have only one normal per texel; projecting into SH then
Just amounts to evaluating the harmonic at that normal.
For coarser-level texels, we can get away with the standard linear MIP map filtering, since the SH are a fixed basis.
Note that we have completely avoided any sort of offline fitting.

29. Effective BRDF Coefficients
Product of NDF, BRDF coefficients
Proceed as usual
Finally, we get the effective BRDF coefficients at runtime by performing the convolution product of the BRDF and NDF.
And we can treat this effective BRDF as we would any other BRDF; spherical harmonics are now in fairly common usage,
so we can now plug these coefficients into any existing spherical harmonic lighting algorithm.

30. And here’s our implementation in action.
We’re using here measured materials from the matusik database, and this isn’t fancy editing; we are in fact
Changing the BRDF on the fly, and note that we maintain proper rendering and filtering at all scales.

31. Limitations Storage cost of NDF Limited to low frequencies
One texture per coefficient
O( ) cost
Limited to low frequencies
Now, although they offer a simple and general solution, there are some limitations to the spherical harmonics.
First of all, we need to store a full MIP map for each coefficient, and since the number of coefficients grows quadratically with the order,
this can get expensive in terms of texture memory.
And this relates closely to a known problem with spherical harmonic methods in general,
Which is that they are limited to low frequencies in practice.
To address this, we now introduce a different NDF representation

32. von Mises-Fisher Distribution (vMF)
more concentrated
less concentrated
Spherical analogue to Gaussian
Desirable properties
Spherical domain
Distribution function
Radially symmetric
The vonmf distribution, or vmf for short.
This is an object that we borrow from the field of directional statistics, and it is roughly analogous to the gaussian distribution on a sphere.
And the vmf has several properties that make it useful as an NDF representation..
First, it lives in the spherical domain, and integrates to 1, which as you will recall are exactly the properties that we
Defined for an NDF. Also, the vmf is radially symmetric about its central direction, making it essentially one-dimensional.
This will allow us to convolve with it much faster, and to use it in higher-frequency situations.

33. Mixtures of vMFs NDF 1 2 3 4 5 6 number of vMFs
Now, in general, a single vMF will be a poor representative for most NDFs that we come across.
However, by using a linear combination of a small number of vmfs, we can suitably approximate most NDFs.
Here we see that this fairly complex NDF can be represented using about 5 or 6 lobes.
Of course, this leaves the question of how to find the parameters for our vMFs, and for this we use a great tool…
number of vMFs

34. Expectation Maximization (EM)
From machine learning
Used in (Tan et.al. 05)
Fit model parameters to data
data
NDF
model
vMF Mixture
Expectation Maximization, or EM, which is a popular method from statistics and machine learning.
For normal map filtering, it was also used by tan and others for finding GMMs.
The main purpose of EM is for fitting model parameters to observed data, which is exactly what we need.
In our case, the data points are the fine-level surface normals represented in the NDF, and the model will be a mixture of vMFs.
Unlike previous methods, our EM works directly in the natural spherical domain of the NDF. EM

35. Rendering Convolution Extensions to EM Spherical harmonic coefficients
Analytic convolution formula
Extensions to EM
Aligned lobes (Tan et.al. 05)
Colored lobes
NDF
rendered image
Finally, once we have our vMF mixtures, the last remaining question is how to render with them.
One approach is to project our vMFs into spherical harmonics and to convolve as we did before.
However, for some BRDFs, we have derived a closed-form convolution formula which allows us to skip the frequency space product altogether… and in practice this means that we can attain very-high frequency BRDFs.
Also, we make a number of extensions to the EM algorithm to enhance rendering. First, we align the lobes of neighboring vmf mixtures, and this basically permits to better leverage standard graphics hardware at render time. Also, we extend EM to include colors, allowing us to assign an RGB color to each vMF lobe.
And although I’ve breezed through these points, full mathematical details can be found in our paper.

38. Now here’s a slightly more involved example.
We have dinosaurs at different scales, but they are all properly filtered, all under dynamic lighting and view.
And we can have dynamic brdfs as well… so here he take the specular, wet appearance, and dry it out by making the brdf more diffuse.

40. Conclusion Summary Future directions Theoretical Framework
New NDF representations
Practical rendering algorithms
Future directions
Offline rendering, PRT
Further applications for vMFs
Shadows, parallax, inter-reflections, etc.
In summary,
we’ve demonstrated a new theory of normal map filtering that has explained previous works, while
enabling us to discover new methods. And our results are immediately usable, and in fact you can find example code for all of our algorithms
on our website.
And for future work, We suspect that our theory will also have implications in related areas such as offline rendering and PRT.
Also, we expect that the von mises fisher distribution will find use in other areas in graphics.
we’d like to incorporate some of the local effects that we’ve ignored, such as shadows, occlusion, interref. And so on.
So in conclusion,
normal map filtering has been an open problem for three decades now, and
we believe that our work here is a major step in finally solving it.

41. Thanks! http://www.cs.columbia.edu/cg/normalmap
Tony Jebara, Aner Ben-Artzi, Peter Belhumeur,
Pat Hanrahan, Shree Nayar, Evgueni Parilov,
Makiko Yasui, Denis Zorin, and nVidia.