1. Rendering With Coherent Layers Jed LengyelJohn Snyder Microsoft Research Jed LengyelJohn Snyder Microsoft Research SIGGRAPH 97

2. Traditional Pipeline Renders 3D scene to an image. Renders 3D scene to an image.

3. Traditional Pipeline

4. Problems Does not exploit temporal coherenceDoes not exploit temporal coherence Spatial and temporal resolutions are globalSpatial and temporal resolutions are global Limited integration of 2D elements in 3DLimited integration of 2D elements in 3DProblems Does not exploit temporal coherenceDoes not exploit temporal coherence Spatial and temporal resolutions are globalSpatial and temporal resolutions are global Limited integration of 2D elements in 3DLimited integration of 2D elements in 3D

5. Traditional Pipeline

6. How can we improve this?

7. Layered Pipeline Add a 2D image warp.

8. Layered Pipeline Sprite A sprite is a warped image with alpha.

9. Layer Layered Pipeline Each layer produces a sprite.

10. Layered Pipeline The scene is factored into layers.

11. Flyby Film

12. Spatial Resolution

13. Previous Work Image-caching [Shade96, Schaufler96]Image-caching [Shade96, Schaufler96] Composition architectures (with z per pixel) [Duff85, Molnar92, Regan94, Mark97]Composition architectures (with z per pixel) [Duff85, Molnar92, Regan94, Mark97] Image-based rendering [Chen93, Chen95, McMillan95]Image-based rendering [Chen93, Chen95, McMillan95] Regulation [Funkhouser93, Maciel95]Regulation [Funkhouser93, Maciel95] Shading factorization [Cook84, Hanrahan90, Segal92, Dorsey95, Guenter95, Meier96]Shading factorization [Cook84, Hanrahan90, Segal92, Dorsey95, Guenter95, Meier96] Image-caching [Shade96, Schaufler96]Image-caching [Shade96, Schaufler96] Composition architectures (with z per pixel) [Duff85, Molnar92, Regan94, Mark97]Composition architectures (with z per pixel) [Duff85, Molnar92, Regan94, Mark97] Image-based rendering [Chen93, Chen95, McMillan95]Image-based rendering [Chen93, Chen95, McMillan95] Regulation [Funkhouser93, Maciel95]Regulation [Funkhouser93, Maciel95] Shading factorization [Cook84, Hanrahan90, Segal92, Dorsey95, Guenter95, Meier96]Shading factorization [Cook84, Hanrahan90, Segal92, Dorsey95, Guenter95, Meier96]

14. 2D Image Transform T is 3D transform to screen coordinates.

15. 2D Image Transform Extra degrees of freedom

16. 2D Image Transform Composition maps to same screen point.

17. 2D Image Transform Transform A maps geometry tightly.

18. 2D Image Transform Transform B is looser than A.

19. 2D Image Transform Spatial resolution

20. 2D Image Transform Temporal resolution

21. Factoring Geometry

22. GoodGood

23. GoodGood

24. GoodGood

25. GoodGood

26. PoorPoor

27. PoorPoor

28. PoorPoor

29. PoorPoor

30. PoorPoor

31. Depth Sorting of Layers Depth sorting in software is effective. Relatively small number of primitivesRelatively small number of primitives Exploits temporal coherenceExploits temporal coherence Depth sorting in software is effective. Relatively small number of primitivesRelatively small number of primitives Exploits temporal coherenceExploits temporal coherence

32. Factoring Shading

33. Multipass Shading [Segal92][Segal92]

35. Factoring Shading

36. Warp Choice Accuracy of image interpolation through timeAccuracy of image interpolation through time Efficient hardware implementationEfficient hardware implementation Ease of computing the warp parametersEase of computing the warp parameters Accuracy of image interpolation through timeAccuracy of image interpolation through time Efficient hardware implementationEfficient hardware implementation Ease of computing the warp parametersEase of computing the warp parameters

37. Characteristic Points Track 3D motion projected to 2D

38. Warp Calculation Characteristic points are matched

39. Warp Calculation W P 0 = P 1

40. Warp Calculation W P 0 = P 1

41. Warps Considered Pure translationPure translation Translation with isotropic scaleTranslation with isotropic scale Translation with independent scale in x and yTranslation with independent scale in x and y AffineAffine PerspectivePerspective Pure translationPure translation Translation with isotropic scaleTranslation with isotropic scale Translation with independent scale in x and yTranslation with independent scale in x and y AffineAffine PerspectivePerspective

42. Warp Comparison

43. Image Interpolation Warping intermediate images is effective Triple- or quadruple-framingTriple- or quadruple-framing Warp calculated with a small set of pointsWarp calculated with a small set of points Decouples 3D rendering from displayDecouples 3D rendering from display Warping intermediate images is effective Triple- or quadruple-framingTriple- or quadruple-framing Warp calculated with a small set of pointsWarp calculated with a small set of points Decouples 3D rendering from displayDecouples 3D rendering from display

44. Image Interpolation

45. Regulation Set independent layer quality parameters

46. Maximize fidelity, balance resources Regulator

47. Fiducials Measure fidelity of approximation GeometricGeometric PhotometricPhotometric SamplingSampling VisibilityVisibility Measure fidelity of approximation GeometricGeometric PhotometricPhotometric SamplingSampling VisibilityVisibility

48. Geometric Fiducial Compares warped and current points

49. Photometric Fiducial Samples lighting at characteristic points

50. Photometric Fiducial 2 Measures change in light position

51. Sampling Fiducial Measures distortion of image samples

52. Visibility Fiducial Counts back-to-front transitions Counts back-to-front transitions Frame 0 Frame 1

53. Barnyard Example

54. Conclusion Layers are good for real-time 3D. Effective using affine image warpEffective using affine image warp Exploit temporal coherenceExploit temporal coherence Factor shadingFactor shading Allow precise targeting of resourcesAllow precise targeting of resources 3x-10x improvement3x-10x improvement Layers are good for real-time 3D. Effective using affine image warpEffective using affine image warp Exploit temporal coherenceExploit temporal coherence Factor shadingFactor shading Allow precise targeting of resourcesAllow precise targeting of resources 3x-10x improvement3x-10x improvement