1. Advanced Lighting With Spherical Harmonics

2. What Is the Algorithm

3. Similar to Light Propagation Volumes

4. Light Propagation Volumes From Crytek and Used in Crysis 2 on Pc, Xbox 360, PS3 Uses Reflective Shadow Maps And Propagation Volumes to Speed up Rendering

5. Goal Global Illumination The Goal is to have light “Bounce” off of Surfaces as it does in the Real World It allows the environment to look much more realistic It has the greatest effect in environments with low ambient light

6. Instant Radiosity The father of Reflective Shadow Mapping The idea is that lights will “project” more lights onto other surfaces Has led to many other techniques

7. Instant Radiosity The “projected” lights will light other surfaces The “Bounce” has been achieved

8. But This is incredibly slow Time to calculate becomes, number of lights multiplied by the number of projected lights to the power of the number of bounces multiplied by the number of surfaces affected Plus the position of the projected lights is determined by ray tracing

9. Reflective Shadow Mapping Start by Shadow Mapping

10. Shadow Mapping Create a Z-buffer or depth buffer from the perspective of the light Using this information you can determine if a point of a surface is in shadow or not

11. Reflective Shadow Mapping Transform Z-buffer to world space and create lights at the Z- buffer locations But we need more information So when we render the initial Z- buffer we also render colour and surface normal

12. Reflective Shadow Mapping Using the colour we can determine what colour the newly created light will be Using the surface normal we can determine what direction the new light should face

13. Reflective Shadow Mapping Removes the Overhead of Raytracing to find new light positions But Does not remove the Overhead of Rendering hundreds or even thousands of lights Adds the Overhead of Rendering many Z-buffers

14. Imperfect Shadow Mapping

15. Same steps as Reflective Shadow Mapping But When we are creating a Reflective shadow map we create many Z-buffers in many different render targets many of which are of very low resolution Low resolution means less info of the geometry is needed

16. Imperfect Shadow Mapping Instead render all the small depth buffers in one large buffer The graphics card has to change state less When rendering the Z-buffers use Stochastic depth Sampling

17. Stochastic Sampling We only need a little information we can figure the rest out So we take less depth samples What we end up with is a depth texture with “holes” in it These holes are where we did not take a sample

18. Imperfect Shadow Mapping Essentially tries to remove the cost of rendering lots of shadow maps Works well for diffuse surfaces (Regular) Does Not work well for glossy surfaces

19. Many LODs

20. A newer technique then Imperfect shadow mapping Again similar to reflective shadow mapping But uses a different way of reducing the cost of rendering many depth textures

21. Many LODs Both Imperfect shadow mapping and Reflective shadow mapping use level of detail to reduce the cost of rendering depth textures But traditional LOD is good for displaying high quality And Bad at displaying low quality

22. Many LODs To solve this many LODs uses a different method It renders points instead of triangles The points are rendered from a Bounding Sphere Hierarchy of the geometry Think about it the depth texture only cares about the shape of the object and the depth textures could be as small as 16 * 16 pixels or even smaller

23. Many LODs With such a low resolution triangle rendering becomes inefficient it requires a pixel to be shaded dozens of times more than they need to be If a triangle is smaller than the size of a pixel then that pixel is being written to more than four times Point rendering will only render that pixel once

24. Many LODs Reduces the cost of rendering depth textures about as much as Imperfect shadow mapping But maintains greater quality Works well displaying diffuse surfaces Works well displaying glossy surfaces

25. Many LODs Also is used for rendering lots of small reflections And refractions Both of which also require very small buffers for which point rendering becomes much more efficient than triangle rendering

26. Temporal Coherence

27. Lights don’t move around all that often So why do we need to calculate the positions, colour and direction of the extra lights every frame Why not keep information from previous frames

28. Temporal Coherence Due to floating point precision error a slow moving reflective shadow map will “jitter” Temporal Coherence will stop that And lower the average cost of rendering

29. Temporal Coherence Will reduce average cost of rendering depth textures Will either increase the memory requirement or will lower the quality of the result Will not reduce the cost of lighting

30. Temporal Coherence You can either store results from previous frames (increasing memory requirements) Or you can decide to only calculate the depth textures every X frames (reducing the quality)

31. But All of these extra techniques focus on reducing the cost of calculating the depth textures This lowers the Overhead of the creation of the “bounce” lights None of these techniques reduce the Overhead of rendering these lights

32. Light Propagation Volumes

33. Aims to reduce the Overhead of rendering the lights Uses reflective shadow mapping to find locations for the first set of “bounce” lights But instead of creating lights add a virtual point light to a volume

34. A Volume you say

35. Volume Imagine a 3D grid and at each cell on that grid you determine what light enters and exits it You could use that grid to determine what light hits a surface within that grid

36. Volume The only way to truly know what light enters each cell is to render the entire environment into a cube map Meaning for every cell on the grid you have to render the entire environment again And then when you want to use that data for each pixel you are shading you have to read each and every pixel and average out the results

37. Volume So instead of averaging out half an entire cube map per pixel we use Spherical Harmonics We then have four numbers per colour channel so the equivalent of three pixels And there is a simple directX call that will convert our cube map into the Spherical Harmonics

38. Volume However we still have to render the entire environment into each grid cell So instead we forget about getting perfectly accurate light flow we just worry about the lights that are generated from the reflective shadow map We still want to end up with spherical harmonics at the end but we don’t want to have to render cube maps

39. Volume So instead of cube maps we create three Volume textures (think of them as 3d arrays of floats) And perform a pixel shader on them Then perform the math to create a light in Spherical harmonics and then add it to the grid

40. Volume We have then essentially rendered to spherical harmonics But lights will now only affect surfaces in the same cell of the volume as they are in So we must Propagate

41. Propagate If we use the values in the neighbouring cells we can calculate what light from that cell should flow into this cell If we do this multiple times then the light will flow as far as it normally would

42. Propagate As we propagate the light we can also check if there is any geometry that would block the flow between two cells Giving you a shadowing effect And if you knew the average surface normal for that geometry then you can “bounce” the light into a corresponding cell This bounce would also have an altered colour based off of the average colour of the blocking geometry

43. Propagate The get the blocking geometry we use the geometry shader and compare each triangles world space position against the positions of the cells of the volume This is done on the geometry shader and not the vertex shader because the vertex shader does not give a surface normal

44. Ok so we have a Light Volume

45. Sampling When we have to shade a surface we now have the lighting info (and we didn’t have to render the entire world multiple times) Find the closest cell to the point you are shading (pixel shader) grad the spherical harmonic coefficients and.....

46. Sampling What does the Spherical harmonics actually give us? The Spherical harmonics is a low frequency representation of the light coming from all directions Think of being in a blurry glass ball that would be a good representation of what the Spherical harmonics gives us

47. Sampling So all we need to know is what light is coming in the direction of the surface normal we are shading Or for specular the reflection angle based on viewing angle and surface normal

48. So why is this faster than lighting normally

49. Light Propagation Volumes When rendering many lights we normally use deferred rendering In light propagation volumes if we wanted to add another light all we have to do is add it to the list and then on the GPU calculate Spherical harmonics in the pixel shader and that is the only extra overhead of adding an extra light (we do have the base cost of propagation though)

50. Light Propagation Volumes In deferred lighting if we want to add a new light then for every surface it touches per pixel we need to calculate its effect and add it to the light currently effecting that pixel With light propagation volumes adding a new light is basically only writing to three pixels where as deferred could potentially cover hundreds of pixels

51. Light Propagation Volumes Since we now have a 3d volume telling us how light is travelling in our scene we can also do other effects very easily such as True specular lighting: normaly the specular materials are faked like in old games like doom 3. true specular is used in battlefield 3 (they also use a grid of spherical harmonic coefficients to do their lighting)

52. Light Propagation Volumes Also we can do light volumes (also in Battlefield 3) which is the beam of light you see in a really dusty room We can also properly light particles (also used in Battlefield 3) if the smoke is thicker then less light would pass through it and more will bounce off it

53. Light Propagation Volumes A issue with light propagation volumes is that is it an approximation of the lighting and so it will never result in the perfect raytraced lighting But most people would not be able to tell the difference The error of the approximation will increase if the resolution of the volume texture is low

54. Light Propagation Volumes Also if the resolution of the reflective shadow maps is low then more error is introduced The other issue is that the Spherical Harmonic coefficients are a low frequency approximation of light flow Only low frequency details are captured Greater resolution of the grid means higher frequency details are captured but a lot of overhead is introduced

55. Light Propagation Volumes So what crytek did was use an average resolution volume and for high frequency details they use screen space ambient occlusion This gives the scene the micro shadows that it would other wise miss

56. Screen Space Ambient Occlusion A post process pixel shader technique First used in crysis It has as many different variants as there are different kinds of Anti Aliasing But the general idea is to use the depth buffer to determine how much geometry is occluding that pixel from its surroundings

57. Screen Space Ambient Occlusion In recent games the shader will also take into account the surface normal so the scene looks more detailed The performance limiting factor is the number of samples each pixel takes from the depth buffer Some games make the SSAO buffer a quarter of the size to increase performance

58. Screen Space Ambient Occlusion Also one of the gears of war games uses temporal coherence to build up the SSAO over multiple frames A general rule is that roughly 100 samples will be needed for good results (which is completely unpractical) But instead the games that use SSAO take between 8 and 16 samples and blur the results

59. Screen Space Directional Occlusion

60. Instead of doing 8-16 samples we do 4- 10 and add a Raytrace in the direction of the surface normal Overhead ends up being just a little more than Screen Space Ambient Occlusion But normal mapping the result is automatic and you can grab the colour from the raytrace and the light that has hit that surface

61. Screen Space Directional Occlusion This can be used to replace the high frequency light bounces It is not 100% accurate but it is effective at faking the extra precision For real time rendering it is a fair compromise Plus just about every triple A game uses it of late (the list of game is to big for this presentation)

62. My Prototype I am not doing the exact same thing as Crytek or Dice

63. My Prototype I recently worked on Spelunking Club Session in unity I was asked if I could make the glowing dance floors cast light onto other objects Propagation volumes is perfect for that it could hand the large number of lights needed to approximate a surface casting light

64. My Prototype So I decided to prototype the propagation volumes part without the reflective shadow map part Then just place lots of lights on the dance floor prefab And pass those into the volume

65. Problems though Volume textures were introduced in directX 10, Unity only supports directX 9 To add occlusion during propagation you need the geometry shader, the geometry shader was introduced in directX 10...Dam

66. Problems though The Volume texture can be replaced by offsetting your logic on a large 2D texture But the benefit of the built in Volume texture is that when you sample the volume texture it returns a filtered result (the average of a bunch of pixels) If I wrote my own it would add more complexity and would take more time but it is not insurmountable

67. Problems though Geometry shader is needed for the occlusion of light You could instead do this on the vertex shader right?

68. WRONG!

69. Problems though The vertex shader does not give us a surface normal On directX 9 and opengl equivalent have low performance for texture fetch on the vertex shader (which was fixed in directX 10) So we could not even check the colour of the surface So I decided that occlusion was not feasible

70. How Mine works Create three RGBA render targets, RGBA gives us four floats which is equivalent to four coefficients per colour channel The three targets each represent a different colour (RGB) The coefficients are zonal harmonics Which means that they are aligned around an axis (usually Z)

71. How Mine Works Add the information for each light to an array that is sent to a pixel shader that runs on these render targets Problem The number of lights in the scene varies but you cannot pass a non fixed length array to a shader in unity

72. How Mine Works Solution Unity uses the cg runtime (Nvidia’s shader language) The cg runtime allows dynamic recompilation of shaders This means we can recreate the shader at runtime

73. How Mine Works This is done by physically changing the text of the shader and then recompiling This sounds hefty but we only need to change a few numbers (length of the fixed length array) And we only need to do this when a new light is created or a light is destroyed

74. How Mine Works To inject we create a Spherical Harmonic representation of the light colour facing in all directions Then create a spherical harmonic representation visibility which is a cone

75. How Mine Works Then rotate that visibility function to the direction of the light we are simulating To combine these we perform the spherical harmonic dot product on them This gives us the Spherical Harmonic representation of this light

76. How Mine Works Combining this with the other lights from that cell will give the full light from that cell To propagate we have to avoid light flowing backward into its original cell which would cause brightening artefacts So we need to propagate only the light going in the direction of the cell we are propagating to

77. How Mine Works To do this we dot product the visibility to that cell by the light in that cell and then propagate This ensures light never flows back Then finally to sample we write a unity surface shader which is put on all surfaces in the world

78. How Mine Works With the light coefficients we do a simple dot product between the coefficients and the surface normal to get the light flowing in that direction