Rendering With Coherent Layers Jed LengyelJohn Snyder Microsoft Research Jed LengyelJohn Snyder Microsoft Research SIGGRAPH 97

Traditional Pipeline Renders 3D scene to an image. Renders 3D scene to an image.

Traditional Pipeline

Problems Does not exploit temporal coherenceDoes not exploit temporal coherence Spatial and temporal resolutions are globalSpatial and temporal resolutions are global Limited integration of 2D elements in 3DLimited integration of 2D elements in 3DProblems Does not exploit temporal coherenceDoes not exploit temporal coherence Spatial and temporal resolutions are globalSpatial and temporal resolutions are global Limited integration of 2D elements in 3DLimited integration of 2D elements in 3D

Traditional Pipeline

How can we improve this?

Layered Pipeline Add a 2D image warp.

Layered Pipeline Sprite A sprite is a warped image with alpha.

Layer Layered Pipeline Each layer produces a sprite.

Layered Pipeline The scene is factored into layers.

Flyby Film

Spatial Resolution

Previous Work Image-caching [Shade96, Schaufler96]Image-caching [Shade96, Schaufler96] Composition architectures (with z per pixel) [Duff85, Molnar92, Regan94, Mark97]Composition architectures (with z per pixel) [Duff85, Molnar92, Regan94, Mark97] Image-based rendering [Chen93, Chen95, McMillan95]Image-based rendering [Chen93, Chen95, McMillan95] Regulation [Funkhouser93, Maciel95]Regulation [Funkhouser93, Maciel95] Shading factorization [Cook84, Hanrahan90, Segal92, Dorsey95, Guenter95, Meier96]Shading factorization [Cook84, Hanrahan90, Segal92, Dorsey95, Guenter95, Meier96] Image-caching [Shade96, Schaufler96]Image-caching [Shade96, Schaufler96] Composition architectures (with z per pixel) [Duff85, Molnar92, Regan94, Mark97]Composition architectures (with z per pixel) [Duff85, Molnar92, Regan94, Mark97] Image-based rendering [Chen93, Chen95, McMillan95]Image-based rendering [Chen93, Chen95, McMillan95] Regulation [Funkhouser93, Maciel95]Regulation [Funkhouser93, Maciel95] Shading factorization [Cook84, Hanrahan90, Segal92, Dorsey95, Guenter95, Meier96]Shading factorization [Cook84, Hanrahan90, Segal92, Dorsey95, Guenter95, Meier96]

2D Image Transform T is 3D transform to screen coordinates.

2D Image Transform Extra degrees of freedom

2D Image Transform Composition maps to same screen point.

2D Image Transform Transform A maps geometry tightly.

2D Image Transform Transform B is looser than A.

2D Image Transform Spatial resolution

2D Image Transform Temporal resolution

Factoring Geometry

GoodGood

GoodGood

GoodGood

GoodGood

PoorPoor

PoorPoor

PoorPoor

PoorPoor

PoorPoor

Depth Sorting of Layers Depth sorting in software is effective. Relatively small number of primitivesRelatively small number of primitives Exploits temporal coherenceExploits temporal coherence Depth sorting in software is effective. Relatively small number of primitivesRelatively small number of primitives Exploits temporal coherenceExploits temporal coherence

Factoring Shading

Multipass Shading [Segal92][Segal92]

Factoring Shading

Warp Choice Accuracy of image interpolation through timeAccuracy of image interpolation through time Efficient hardware implementationEfficient hardware implementation Ease of computing the warp parametersEase of computing the warp parameters Accuracy of image interpolation through timeAccuracy of image interpolation through time Efficient hardware implementationEfficient hardware implementation Ease of computing the warp parametersEase of computing the warp parameters

Characteristic Points Track 3D motion projected to 2D

Warp Calculation Characteristic points are matched

Warp Calculation W P 0 = P 1

Warp Calculation W P 0 = P 1

Warps Considered Pure translationPure translation Translation with isotropic scaleTranslation with isotropic scale Translation with independent scale in x and yTranslation with independent scale in x and y AffineAffine PerspectivePerspective Pure translationPure translation Translation with isotropic scaleTranslation with isotropic scale Translation with independent scale in x and yTranslation with independent scale in x and y AffineAffine PerspectivePerspective

Warp Comparison

Image Interpolation Warping intermediate images is effective Triple- or quadruple-framingTriple- or quadruple-framing Warp calculated with a small set of pointsWarp calculated with a small set of points Decouples 3D rendering from displayDecouples 3D rendering from display Warping intermediate images is effective Triple- or quadruple-framingTriple- or quadruple-framing Warp calculated with a small set of pointsWarp calculated with a small set of points Decouples 3D rendering from displayDecouples 3D rendering from display

Image Interpolation

Regulation Set independent layer quality parameters

Maximize fidelity, balance resources Regulator

Fiducials Measure fidelity of approximation GeometricGeometric PhotometricPhotometric SamplingSampling VisibilityVisibility Measure fidelity of approximation GeometricGeometric PhotometricPhotometric SamplingSampling VisibilityVisibility

Geometric Fiducial Compares warped and current points

Photometric Fiducial Samples lighting at characteristic points

Photometric Fiducial 2 Measures change in light position

Sampling Fiducial Measures distortion of image samples

Visibility Fiducial Counts back-to-front transitions Counts back-to-front transitions Frame 0 Frame 1

Barnyard Example

Conclusion Layers are good for real-time 3D. Effective using affine image warpEffective using affine image warp Exploit temporal coherenceExploit temporal coherence Factor shadingFactor shading Allow precise targeting of resourcesAllow precise targeting of resources 3x-10x improvement3x-10x improvement Layers are good for real-time 3D. Effective using affine image warpEffective using affine image warp Exploit temporal coherenceExploit temporal coherence Factor shadingFactor shading Allow precise targeting of resourcesAllow precise targeting of resources 3x-10x improvement3x-10x improvement