1. Fu-Chung Huang Ravi Ramamoorthi University of California, Berkeley
Sparsely Precomputing the Light Transport Matrix for Real Time Rendering
Fu-Chung Huang
Ravi Ramamoorthi
University of California, Berkeley
(Hi, Good morning everyone, thanks for coming to my talk this early )
My paper is: Sparsely …..
I’m Fu-Chung from UC Berkeley

2. Precomputed Radiance Transfer (PRT)
is a new recent development for real time rendering.
Natural environment lighting, intricate shadows, and
Glossy materials by pre-computation.
The method makes possible rapid prototyping of lighting design, for games and movies.
Although we can achieve interactive rate rendering, the time spent in pre-computation is very long.
(click)
Like in this case, the pre-computation takes 13 hours, and the time becomes a bottleneck prevents people using it.
Our goal in this paper is to accelerate the pre-computation.
So we want to define the problem first.
[Sloan et al. 02] [Ng et al. 03, 04] [Liu et al. 04] [Wang et al. 04]
Precomputation Time: 13 hours 22 mins

3. The Problem Reflectance equation Separable Dynamic lighting
Ng et al. 04
Reflectance equation
Separable
Dynamic lighting
Static geometry
Lighting
Visibility
BRDF
So to render this nice scene, we have to consider the reflectance equation
If we want to render the red dot B(x), whose value is given by the integral of
(surrounding L) (the V in all direction) and (BRDF)
-----
(click) The key insight here is that light is dynamically change,
and is separable from the rest in the equation.
(click) So we can combine the information only relevant to the static scene,
by pre-computing the V and BRDF into
(click) the transport function T.
Then in real time, we just need to multiply the lighting with the transport function.
(click) Notice that by the nature of precomputation, we are also limited to static scene.
Beside this, what is the problem of pre-computing the transport function T ?
Limited to
static scene

4. Storage and Time Matrix Compression 50K vert. (rows)
24K directions (cols)
1.2B Rays
Compression
10x to 50x
Precomputation time is not reduced!
Clustered
PCA
Assume in a typical scene, we have 50k vertices,
Each with 24k angular direction to sample
Then we have 1.2 B rays to samples, and 1.2B cell to store.
(click) Then the immediate problem is how to compress the data
(click) For example, people use wavelet transformation
To reduce the coeffs from several thousands to a few hundreds,
And they can be further quantized to 6-8bits.
---
(click) On the other direction, with the *spatial coherences*,
We can cluster vertices into smaller group, and compress them with very few PCA bases.
(click) These combined give 10-50x compression rate, However, these requires additional time to compute?
(click) and notice that:
Even we reduce the storage, pre-computation time is not reduced actually!
So the Q is CAN we find some coherence at sampling stage to reduce sampling time?
Wavelet +
Quantization

5. Precomputation Time Buddha Scene 5x speed up in precomputation
The answer is yes…
We can render with the same quality, with reduced pre-computation time!
In this example, we can sample the scene with 2 hour and 30m
Achieving 5x speed up in precomputation.
Sparse Sampling
Full Sampling
2 hours 36 mins
13 hours 22 mins

6. Precomputation Time Bunny Scene 4x speed up precomputation
Here is another example with bunny model,
again we achieve around 4x speedup in precomputation
So before we start our algorithm,
I want to first review some important literature,
Sparse Sampling
Full Sampling
3 hours 25mins
13 hours 8 mins

7. Outline Related Work Motivation / Introduction Algorithm Results
Conclusion / Future Work

8. Related Work Precomputation based rendering
[Nimeroff et al. 94] [Dorsey et al. 95]
Focus only on added functionality
[Wang et al. 04,06]
[Sloan et al. 02,03]
[Ng et al. 03,04]
The idea of PRT can be traced back to 15 years ago, and is formally introduced by Sloal et al.
They use SH to achieve real-time Low-Freq shadow,
and further they propose CPCA to compress the data
(click) Ng: use non-linear Wavelet APPROX to achieve all-frequency shadow,
and later introduce the notion of triple product integral.
(click) Want: show that glossy material is also possible for all frequency shadow PRT
Notice that these are just *representative seminal paperS*.
There are still a lot of work presented in SIGGRAPH/EGSR these years
OK, since these are nice work, then what is the problem with them?
(click) well, these methods keep adding new features for real-time rendering,
But the sampling time is still not reduced
Low frequency shadows
CPCA compression
High frequency shadows
Triple product integral
Glossy materials

9. Related Work Compressive sensing Not applicable to PRT
[Candes et al. 06] [Candes and Tao 06]
Sampling Rate = k logN for k-sparse signal
Not applicable to PRT
No random pattern sampling in virtual scene
Must sample one ray at a time
[Peers et al. 09]
[Sen and Darabi. 09]
Compressive sensing is an interesting research topic to sparsely sample the data.
Canades et al have shown that a signal with length N
can be *recovered* with klogN samples,
if it has a k-sparse representation
(click) The result is applied to appearance acquisition, by
Peers et al. And Sen and Darabi
Sen and Darabi also has a new paper this year showing up later
In this section
(click) However, the method is not …
because there is no random sampling pattern in virtual scene,
Where we use ray-tracing to trace one direction at a time
-----
So probably we need something that applies to point samples

10. Related Work Row-column sampling
Column selection is fixed across all rows R C A
B = CA+R
Row-column sampling is a method that we can use,
since it simply sample certain row/column cells ,
then recover/approximate the entire matrix
(click) Here is the work by Hasan et al.
(click) They sparsely sample the matrix for many light contribution P.
(click) They first sample some rows, and
(click) by finding clusters, they use cluster centers to represent the matrix
(click) Wang et al. use NYSTROM method sparse sample the matrix
(click) With certain rows and column sampled,
(click) They can fully recover the matrix
These methods are very close reference to our work,
(click) But they use the fixed column selection across all rows.
[Hasan et al. 07]
[Wang et al. 09]

11. Related Work Hierarchical and Sparse Sampling Adaptive Methods
[Kontkanen et al. 06]
[Hasan et al. 07]
[Lehtinen et al. 08]
[Krivanek and Gautron 09]
Adaptive Methods
[Guo 98]
[Krivanek et al. 04]
[Krivanek et al. 04]
[Krivanek and Gautron 09]
Finally I want to reference some other work in offline rendering,
In the sense of sparse-sampling-interpolation,
Our method is also related to irradiance caching, And some other work
However, we focus on drastically changing high frequency shadows,
While these method focused on smooth indirect illumination,
(click)
Finally the adaptive remeshing can be seen as compliment way to our method.
These methods explore the spatial coherence, but the angular coherence is not utilized

12. Outline Algorithm Motivation / Introduction Related Work Results
Conclusion / Future Work
So after reviewing some previous work, lets talk about our method

13. Algorithm Outline Overview Dense Vertex Sampling
Sparse Vertex Sampling
Integrating Clustered PCA
I will start with simple overview

14. Overview - 2 Phase Sampling
Dense vertex
Spatial: 20%~25%
Sparse vertex
Spatial: 75%~80%
Angular: ~30%
Our method use a two-phase sampling strategy
It separate the scene in 2 kinds of vertex
(click) The first kind is call dense vertex,
we sample their angular direction fully or densely
(click) And for the second kind called sparse vertex,
(click) we sample their angular direction sparsely,
(click) by using information from neighboring dense vertices, we can reconstruct it later
(click) Around 20-25% of the vertices are dense,
whose angular sampling rate are around 30%
(click) The rest 75-80% are sparse,
their angular sampling rate accounts for only 5-7%
If we view the problem in a row-column sampling way
Angular: 5%~7%

15. Overview - 2 Phase Sampling
Row-column sense
Angular: ~30%
Spatial:
20%~25%
=6%~7%
Dense Vertex
Spatial:
75%~80%
=4%~5%
Sparse Vertex
Then given a matrix,
(click) we first sample 20-25% rows for dense vertex
(click) with the column sampling rate ~30%
(click) which account for 6-7% total cost
(click) for the rest 75-80% rows are sparse vertices
(click) we sample columns with 5-7%,
(click) with total cost 4-5%
----
In next few slides I will explain the sampling for dense vertex
Angular: 5%~7%

16. Algorithm Outline Dense Vertex Sampling Overview
Where?
How?
What?
Sparse Vertex Sampling
Integrating Clustered PCA
So in this section, I will describe for dense vertex:
Where to choose them
How to choose?
And what angular direction to trace ray

17. Dense Vertex Distribution
Observation from CPCA
Non-uniform cluster sizes
Large cluster: low rank
Small cluster: high rank
Non-uniform sampling
But how?
We begin by the observation from CPCA,
And you can see that the clusters are not uniform in size.
(click) after further inspection, we find out large cluster has low rank,
and small cluster has high rank
(click) this give us hint that we should have more samples at high rank area
So how to sample?
Without prior knowledge, we don’t have the rank information in advance?

18. Dense Vertex Spatial Sampling
Sampling by exploration
1st iteration
Uniform
Local rank -> probability
2nd iteration
And so on….
Zoom-Up
High rank area
Here we get the information during the sampling iterations.
(click) 1st we have uniform sampling, using these samples to calculate local rank.
We use the rank to assign probability for the next iteration sampling
(click) *Notice the red box*, which is the high rank area,
so that this area is more likely to be sampled.
(click) In second iteration, we again calculate rank for next iteration probability
And so on.
(click**) here is the zoom up,
We can see samples concentrate to high rank regions.
After I describe how to sample spatially,
I will describe what angular direction to sample
[2nd iteration]
[3rd iteration]
[4th iteration]
[1st iteration]

19. Dense Adaptive Angular Sampling
1st pass: regularly
2nd pass: adaptively
If values are inconsistent
50%-70% savings
We use adaptive angular sampling to find features
(click)We first sample regularly at lower resolution
(click) For all other directions,
(click) we check the sampled value in the window
(click) if these value are inconsistent,
(click) we simply sample that direction
(click) and by this we can sample directions around the feature
(click) this gives around 50~70% saving

20. Sparse Vertex Sampling
Algorithm Outline
Overview
Dense Vertex Sampling
Sparse Vertex Sampling
Angular Sampling
Local Reconstruction
Integrating Clustered PCA
So after sampling the dense vertex
in spatial and angular domain,
then I want to show how to deal with the sparse vertex

21. Sparse Vertex Angular Sampling
Remember in overview…
The sparse set of important angular features
In overview, we’ve shown that for sparse vertex,
(click) we sample their angular direction sparsely,
(click) and use neighboring dense vertices for reconstruction,
this section I will show how to do that
(click) The first problem is to determine … How can we start?
(click) Basically we can start from the neighboring dense vertices
(click) Here from previous feature adaptive sampling, we know where the features are…
(click) So we will select from the union of all these feature direction.
But there are too many of them,
(click) So we use variance to cut-off unimportant features.
(click) and here is the final selection
So once we’ve selected the sparse set of angular features, I’ll describe how to reconstruct the transport function
Weighted selection, 5%~7%
Neighboring angular features
The light transport from its
neighboring dense vertices
Union of these features,
but too many
Variances of neighboring
angular features

22. Sparse Vertex Reconstruction
Reconstructed
light transport ?
Sampled directions as constraints
Here we use a very simple linear system for reconstruction
The assumption is that: the transport function should be some linear combination from the neighboring dense vertices.
Such that
(click)the sampled directions should
(click)Equals to the linear combination of the
*selected directions from dense neighbors*
(click)So by solving for alpha, we can recover the light transport.
The final Q is how many neighbors should we use?

23. Sparse Vertex Reconstruction
How many neighbors?
In reality
L1 sparse solver
[Kim et al. 07]
No exact radius needed
(click) Here we’ve experimented with increasing radius to include more neighbors
(click) Ideally, if we have full information
on how to interpolate the sparse vertex
the green curve is something we would expect!!!!
(click) In reality, we’ve got blue curve with least squares method,
with increasing error,
Because it over fit to the sparse constraints.
(click) Inspired by recent development in sparse reconstruction,
we used L1 solver, and have lower error rate for the red curve
And because of the sparse nature, we don’t need to specify the exact radius.

24. Integrating Clustered PCA
Algorithm Outline
Overview
Dense Vertex Sampling
Sparse Vertex Sampling
Integrating Clustered PCA
So here after the sampling for D and S V
we further found that the current algorithm fit nicely with CPCA

25. Clustered PCA Incrementally adding bases
[Sloan et al. 03]
Avoid local minimum
Linear combination assumption!
run clustering on
*dense* vertices 1.
for each PCA basis
for each LBG iteration
end for
run clustering on
*all* vertices
The original CPCA algorithm is proposed by sloan et al 03.
In order to avoid local minimum,
they use incremental bases addition.
(click) Their method runs a nested loop on *all vertices*, and becomes very expensive.
(click) Our observation is that sparse V are linear comb from D V,
(click) So we only run the nested loop for dense vertex,
And you can see the clustering for dense vertex on the left
Then we assign the rest sparse vertices to their nearest cluster,
And run the clustering just once
The result is shown on the left
The result give us great amount of speed up,
that we will show later
Assign sparse vertex
to nearest cluster 2.
Run inner loop for all vertices 3.

26. Outline Results Motivation / Introduction Related Work Algorithm
Error analysis
Performance
Conclusion / Future Work
Here I will show the error comparison and performance

27. Analysis: Angular Sampling
Adaptive sampling
Non-adaptive angular sampling
We first compare our angular sampling strategy with related previous work
So Here is Wang et al
(click) And here is Hasan et al
(click) Both methods non-adaptive… Which means their angular selection is fixed,
(click) So a lot of samples are wasted.
(click) our method use adaptive sampling,
(click) where samples are sent to really important regions
[Ours]
[Wang et al. 09]
[Hasan et al. 07]
Sampling important
directions
A lot of directional samples are wasted

28. Analysis: L2 Error for Bunny
35K + 29K vertices
Scanned
High error at seams
For the error analysis,
The bunny is a scanned model, vertices are distributed evenly.
Notice that there are some high error rate at the seams
But in general, error is low everywhere and un-noticeable

29. Analysis: L2 Error for Horse
8.5K + 29K vertices
Low resolution
Man made
On the other hand, we have a fairly simplified mesh.
This is a low-resolution and man-made model,
which means vertex distribution is not uniform.
A miss in important region can be large.
However the error is still low in general
and not noticeable in the rendering

30. Performance Precomputation time only
Rendering is real-time with the same quality
Model
Horse
Dancer
Bunny
CPCA
New
CPCA Speed Up
52m 4m
12.6x
45m 3m
81m 6m
12.8x
Full Sampling
Sparse Sampling
Sampling
Rate
Speed Up
4h 2m
54m
~11%
4.47x
5h 40m
1h 15m
4.77x
13h 8m
3h 26m
3.83x
Now I want to show the table for precomputation time.
The rendering is real-time with the same quality
(click) So for the fully sampled version, it took 4-13 hours to sample
(click) Our sparse method only samples 11%, the time reduced to 1-3 hours.
(click) Achieving 3-4x speed up
(click) here is the time for CPCA, we achieve 12x speed up.

31. Performance for Glossy Objects
In-Out Factorization for glossy BRDFs
[Liu et al. 04] [Wang et al. 04]
Model
Size
Full
Sampling
Sparse Sampling
Rate
Speed Up
Armadillo
55K
10h 7m
2h 17m
11.43%
4.75x
Buddha
13h 22m
2h 36m
11.11%
5.12x
Dragon
10h 31m
2h 1m
11.16%
5.22x
Bench
50K
7h 7m
1h 56m
18.54%
3.67x
Here is another performance table for glossy objects
We use in-out Fact. for glossy BRDF
(click) the size of the scene range from 50-55K vertices,
(click) the precomputation time range from 7-13 hours
(click) again with our method, we can sample in about 2 hours
(click)
Notice that the bench scene has lots of high frequency features,
so we sent more samples for this particular scene.
(click) finally we can still do 3-5x faster.

32. Precomputation Time Bench Scene 3.6x speed up in precomputation
This is the bench scene I’ve just mentioned
The is one extreme scene that we have for stressed test.
Notice that there are a lot of high frequency features,
Vide-dependent high light, and intricate shadows,
Although this is a very difficult scene for sparse sampling,
we can still precompute within 2 hours,
Resulting in 3.6x sped up
Sparse Sampling
Full Sampling
1 hours 56mins
7 hours 7 mins

33. Detailed Timing Here is the system time breakdown,
(click) most cost are spent on ray-tracing
And you can see our overhead is actually minimum
In addition, although in our case 10% sampling accounts for 5x speedup,
these are rays sent to important region, that actually need more attention,
As our algorithm is designed.

34. Conclusion PRT research on real-time functionality
Precomputation is often the bottleneck
Adaptive and sparse sampling
Exploit both spatial and angular coherence
Accelerated Clustered PCA compression
Sparse precomputation possible
5x speed-up
12x speed-up computing CPCA
So to conclude my talk,
The current PRT research still focus on new functionalities
But the precomputation is still the bottle neck prevent people using it
(click) In this paper we propose an adaptive and sparse sampling scheme
That take advantage in both S and A coherence
Also we’ve show that CPCA can be integrated with the new scheme
(click)
Finally we show 5x speed up in sampling
And 12x speed up in compression.

35. Future Work GPU acceleration New capabilities
Interactive precomputation
New capabilities
Rapid prototyping
Lighting design
Dynamic scenes
General theory of sparse sampling
Avoid heuristic parameter tuning
Broader context
Appearance acquisition
Offline rendering
For the future work,
Since most of our system time are still ray-tracing,
GPU can help to accelerate that,
and we can expected an interactive precomputation
(click) For the new capability, the fast precomputation enables
a faster Rapid prototyping…
Also temporal coherence can be utilized for dynamic scene
(click) In addition, we would like to see some
general theory for sparse sampling in the rendering community,
Since most parameters are heuristically tuned
(click) In a broader context, sparse sampling is also important in
Appearance acquisition and offline rendering
And we would like to see more of them.

36. The End Acknowledgements Ryan S. Overbeck Anonymous reviewers
NSF CAREER Grant IIS
ONR PECASE N
Intel
NVIDIA
Adobe
Pixar