Image-Based Rendering to Accelerate Interactive Walkthroughs Lucent Technologies Bell Labs Innovations Daniel G. Aliaga Visual Communications Research Lucent Technologies Bell Labs

E2 3D Models 1.7M tris 0.9M tris 2.0M tris 1.0M tris

E3 Why Use Images? Independent of scene complexityIndependent of scene complexity 640x480 pixels

E4 Flight Simulators Mid-1980’sMid-1980’s –E&S CT-6 one of first to use real-time photo textures Hand-selected objects:Hand-selected objects: –Terrain, trees, airplanes, buildings, etc Hz30-60 Hz –High visual fidelity

E5 Overview Replacing Geometry with ImagesReplacing Geometry with Images Displaying ImagesDisplaying Images –Texture-mapping and error metrics –Geometry and image warping –Meshes, Lightfield/Lumigraph Image PlacementImage Placement –Automatically Bounding Model Complexity –Cells and Portals ConclusionsConclusions

E6 Replacing Geometry with Images AlgorithmAlgorithm –Select subset of model –Create image of the subset –Cull subset and replace with image Why?Why? –Image displayed in (approx.) constant time –Image reused for several frames

E7 Simple Example

E8 Simple Example

E9 Simple Example

E10 Overview Replacing Geometry with ImagesReplacing Geometry with Images Displaying ImagesDisplaying Images –Texture-mapping and error metrics –Geometry and image warping –Meshes, Lightfield/Lumigraph Image PlacementImage Placement –Automatically Bounding Model Complexity –Cells and Portals ConclusionsConclusions

E11 Geometric Discontinuity If we move from the center-of-projection, discontinuities appear at the borderIf we move from the center-of-projection, discontinuities appear at the border

E12 Temporal Discontinuity While moving, if we switch between geometry and image, a sudden pop occursWhile moving, if we switch between geometry and image, a sudden pop occurs

E13 Approaches Geometric and Temporal ContinuityGeometric and Temporal Continuity –Error metrics –Geometry warping –Image warping –Lightfield/Lumigraph

E14 Error Metrics Use an error metric to control amount of discontinuityUse an error metric to control amount of discontinuity  c1c1 c2c2 [Maciel95][Shade96][Schaufler96]

E15 Error Metric Relies on “angular-deviation” measuring the visual quality of using the (same) imageRelies on “angular-deviation” measuring the visual quality of using the (same) image

E16 Video Segment I Pre-rendered ImpostorsPre-rendered Impostors –Maciel95 Dynamic Image-CachingDynamic Image-Caching –Shade96, Schaufler96

E17 Geometry Warping [ Aliaga96 ]

E18 Geometry Warping

E19 Geometry Warping

E20 Geometry Warping

E21 Geometry Warping

E22 Geometry Warping

E23 Geometry Warping

E24 Geometry Warping

E25 Geometry Warping Surrounding geometry warped (incorrect perspective) Correct perspective

E26 Geometry Warping Surrounding geometry warped to match imageSurrounding geometry warped to match image Center-of-projection Viewpoint Surrounding Geometry

E27 Video Segment II Geometry WarpingGeometry Warping –Aliaga96

E28 Image Warping Change the image itselfChange the image itself –Re-project the image to the current viewpoint [Chen93][McMillan95][Max95][Shade98] [Chen93][McMillan95][Max95][Shade98] –Display image as a (simplified, textured) mesh [Darsa97][Sillion97] [Darsa97][Sillion97]

E29 Image Warping A raster scan of each sheet produces a back-to-front ordering of warped pixelsA raster scan of each sheet produces a back-to-front ordering of warped pixels eye Reference COP

E30 Image Warping McMillan and Bishop’s Warping EquationMcMillan and Bishop’s Warping Equation ~Texture mapping Move pixels based on distance to eye x 2 =  (x 1 ) P 2 -1 (c 1 - c 2 ) + P 2 -1 P 1 x 1 x1x1 c1c1 c2c2 x2x2 d

E31 Example... Image outlined in yellowImage outlined in yellow Viewed from image’s center-of-projectionViewed from image’s center-of-projection

E32 3D Image Warp Single sample per pixelSingle sample per pixel

E33 Layered Depth Image Warp Multiple samples per pixelMultiple samples per pixel –Previous occlusions are filled-in [ Popescu98 ]

E34 Meshes (Simplified) Textured Depth Mesh(Simplified) Textured Depth Mesh –Per-pixel depth creates mesh that approximates 3D parallax effects –Image is texture-mapped onto mesh

E35 Video Segment III 3D Image Warping3D Image Warping –McMillan95 Textured MeshesTextured Meshes –Darsa97, Sillion97

E36 Lightfield/Lumigraph Flow of light at all positions and directionsFlow of light at all positions and directions –[Levoy96][Gortler96] Large number of images are used as 2D slices of a 4D light functionLarge number of images are used as 2D slices of a 4D light function (u,v) (s,t)

E37 Video Segment IV Light fieldLight field –Levoy96 LumigraphLumigraph –Gortler96

E38 Overview Replacing Geometry with ImagesReplacing Geometry with Images Displaying ImagesDisplaying Images –Texture-mapping and error metrics –Geometry and image warping –Meshes, Lightfield/Lumigraph Image PlacementImage Placement –Automatically Bounding Model Complexity –Cells and Portals ConclusionsConclusions

E39 Automatic Image-Placement As a preprocessAs a preprocess –Select geometry to replace with an image in order to limit the number of primitives to render for any frame At run timeAt run time –Display selected geometry as a (depth) image –Render remaining geometry normally

E40 Automatic Image-Placement Preprocess Run time

E41 Example Rendering Final Scene Geometry Image + =

E42 Viewpoint - Overview - Image Placement - Displaying Images - Conclusions Key Observation Example modelExample model Too much geometry in view frustumToo much geometry in view frustum

E43 Viewpoint Key Observation Geometry is replaced by image to limit the number of primitives to renderGeometry is replaced by image to limit the number of primitives to render

E44 Eye Grid viewpoint Key Observation Less geometry is in the view frustum from the eye than the one from the grid viewpointLess geometry is in the view frustum from the eye than the one from the grid viewpoint

E45 Recursive Subdivision Algorithm 2k 2k+1 2k+2 2k+1 Even to odd Odd to even

E46 Example Grid 3D grid of 1557 viewpoints Wireframe rendering

E47 Sample Path 300 Frame View Frustum Culling Images Present Polygons Power Plant, 2M tris, SGI Onyx 2 w/ IR

E48 Images Per Triangle Maximum Rendered Triangles (% of Model) Images Per Model Triangle Power Plant Torpedo Room Brooks House Pipes Best Case Worst Case

E49 Preprocessing Summary

E50 Video Segment V Automatically Bounding Geometric Complexity by Using ImagesAutomatically Bounding Geometric Complexity by Using Images –Aliaga99

E51 Cells and Portals [Airey90, Teller91, Luebke95]

E52 Portal Images [Aliaga97]

E53 Creating Portal Images portal eye Ideal portal image would be one sampled from the current eye position

E54 Creating Portal Images Reference COPs Display one of a large number of pre-computed images (~120) portal

E55 Creating Portal Images portal Reference COPs or… Warp one of a much smaller number of reference images

E56 Brooks House Frame Rendering Time (sec.) View Frustum Culling Portal Culling Static Portal Images Onyx IR

E57 Brooks House Frame Rendering Time (sec.) Portal Culling Warped Portal Images Onyx IR

E58 Video Segment VI Architectural Walkthroughs using Portal ImagesArchitectural Walkthroughs using Portal Images –Aliaga97, Rafferty98

E59 Overview Replacing Geometry with ImagesReplacing Geometry with Images Displaying ImagesDisplaying Images –Texture-mapping and error metrics –Geometry and image warping –Meshes, Lightfield/Lumigraph Image PlacementImage Placement –Automatically Bounding Model Complexity –Cells and Portals ConclusionsConclusions

E60 Image Quality What about measuring quality?What about measuring quality? – Need a perceptual quality metric! We knowWe know –Texture-mapping: bad perspective, small distortions believable (geometry warping) –IBR: correct perspective, disocclusions –Meshes: stretching of skins

E61 Limitations Diffuse illuminationDiffuse illumination –Deferred shading? Static modelsStatic models –Incremental updating? Cannot sample all visible surfacesCannot sample all visible surfaces –Smarter reconstruction/resampling? Can only sample surfaces at a fixed resolutionsCan only sample surfaces at a fixed resolutions –Multi-resolution reference images?

E62 Acknowledgments Authors of the Video SegmentsAuthors of the Video Segments ModelsModels –Discreet Logic, UNC Walkthrough Group UNC-Chapel HillUNC-Chapel Hill –Walkthrough, PixelFlow, ImageFlow NSF, NIH, DARPANSF, NIH, DARPA Lucent Technologies Bell LabsLucent Technologies Bell Labs