1. Image-Based Rendering to Accelerate Interactive Walkthroughs Lucent Technologies Bell Labs Innovations Daniel G. Aliaga Visual Communications Research Lucent Technologies Bell Labs <aliaga@bell-labs.com>

2. E2 3D Models 1.7M tris 0.9M tris 2.0M tris 1.0M tris

3. E3 Why Use Images? Independent of scene complexityIndependent of scene complexity 640x480 pixels

4. E4 Flight Simulators Mid-1980’sMid-1980’s –E&S CT-6 one of first to use real-time photo textures Hand-selected objects:Hand-selected objects: –Terrain, trees, airplanes, buildings, etc. 30-60 Hz30-60 Hz –High visual fidelity

5. E5 Overview Replacing Geometry with ImagesReplacing Geometry with Images Displaying ImagesDisplaying Images –Texture-mapping and error metrics –Geometry and image warping –Meshes, Lightfield/Lumigraph Image PlacementImage Placement –Automatically Bounding Model Complexity –Cells and Portals ConclusionsConclusions

6. E6 Replacing Geometry with Images AlgorithmAlgorithm –Select subset of model –Create image of the subset –Cull subset and replace with image Why?Why? –Image displayed in (approx.) constant time –Image reused for several frames

7. E7 Simple Example

8. E8 Simple Example

9. E9 Simple Example

10. E10 Overview Replacing Geometry with ImagesReplacing Geometry with Images Displaying ImagesDisplaying Images –Texture-mapping and error metrics –Geometry and image warping –Meshes, Lightfield/Lumigraph Image PlacementImage Placement –Automatically Bounding Model Complexity –Cells and Portals ConclusionsConclusions

11. E11 Geometric Discontinuity If we move from the center-of-projection, discontinuities appear at the borderIf we move from the center-of-projection, discontinuities appear at the border

12. E12 Temporal Discontinuity While moving, if we switch between geometry and image, a sudden pop occursWhile moving, if we switch between geometry and image, a sudden pop occurs

13. E13 Approaches Geometric and Temporal ContinuityGeometric and Temporal Continuity –Error metrics –Geometry warping –Image warping –Lightfield/Lumigraph

14. E14 Error Metrics Use an error metric to control amount of discontinuityUse an error metric to control amount of discontinuity  c1c1 c2c2 [Maciel95][Shade96][Schaufler96]

15. E15 Error Metric Relies on “angular-deviation” measuring the visual quality of using the (same) imageRelies on “angular-deviation” measuring the visual quality of using the (same) image

16. E16 Video Segment I Pre-rendered ImpostorsPre-rendered Impostors –Maciel95 Dynamic Image-CachingDynamic Image-Caching –Shade96, Schaufler96

17. E17 Geometry Warping [ Aliaga96 ]

18. E18 Geometry Warping

19. E19 Geometry Warping

20. E20 Geometry Warping

21. E21 Geometry Warping

22. E22 Geometry Warping

23. E23 Geometry Warping

24. E24 Geometry Warping

25. E25 Geometry Warping Surrounding geometry warped (incorrect perspective) Correct perspective

26. E26 Geometry Warping Surrounding geometry warped to match imageSurrounding geometry warped to match image Center-of-projection Viewpoint Surrounding Geometry

27. E27 Video Segment II Geometry WarpingGeometry Warping –Aliaga96

28. E28 Image Warping Change the image itselfChange the image itself –Re-project the image to the current viewpoint [Chen93][McMillan95][Max95][Shade98] [Chen93][McMillan95][Max95][Shade98] –Display image as a (simplified, textured) mesh [Darsa97][Sillion97] [Darsa97][Sillion97]

29. E29 Image Warping A raster scan of each sheet produces a back-to-front ordering of warped pixelsA raster scan of each sheet produces a back-to-front ordering of warped pixels eye Reference COP

30. E30 Image Warping McMillan and Bishop’s Warping EquationMcMillan and Bishop’s Warping Equation ~Texture mapping Move pixels based on distance to eye x 2 =  (x 1 ) P 2 -1 (c 1 - c 2 ) + P 2 -1 P 1 x 1 x1x1 c1c1 c2c2 x2x2 d

31. E31 Example... Image outlined in yellowImage outlined in yellow Viewed from image’s center-of-projectionViewed from image’s center-of-projection

32. E32 3D Image Warp Single sample per pixelSingle sample per pixel

33. E33 Layered Depth Image Warp Multiple samples per pixelMultiple samples per pixel –Previous occlusions are filled-in [ Popescu98 ]

34. E34 Meshes (Simplified) Textured Depth Mesh(Simplified) Textured Depth Mesh –Per-pixel depth creates mesh that approximates 3D parallax effects –Image is texture-mapped onto mesh

35. E35 Video Segment III 3D Image Warping3D Image Warping –McMillan95 Textured MeshesTextured Meshes –Darsa97, Sillion97

36. E36 Lightfield/Lumigraph Flow of light at all positions and directionsFlow of light at all positions and directions –[Levoy96][Gortler96] Large number of images are used as 2D slices of a 4D light functionLarge number of images are used as 2D slices of a 4D light function (u,v) (s,t)

37. E37 Video Segment IV Light fieldLight field –Levoy96 LumigraphLumigraph –Gortler96

38. E38 Overview Replacing Geometry with ImagesReplacing Geometry with Images Displaying ImagesDisplaying Images –Texture-mapping and error metrics –Geometry and image warping –Meshes, Lightfield/Lumigraph Image PlacementImage Placement –Automatically Bounding Model Complexity –Cells and Portals ConclusionsConclusions

39. E39 Automatic Image-Placement As a preprocessAs a preprocess –Select geometry to replace with an image in order to limit the number of primitives to render for any frame At run timeAt run time –Display selected geometry as a (depth) image –Render remaining geometry normally

40. E40 Automatic Image-Placement Preprocess Run time

41. E41 Example Rendering Final Scene Geometry Image + =

42. E42 Viewpoint - Overview - Image Placement - Displaying Images - Conclusions Key Observation Example modelExample model Too much geometry in view frustumToo much geometry in view frustum

43. E43 Viewpoint Key Observation Geometry is replaced by image to limit the number of primitives to renderGeometry is replaced by image to limit the number of primitives to render

44. E44 Eye Grid viewpoint Key Observation Less geometry is in the view frustum from the eye than the one from the grid viewpointLess geometry is in the view frustum from the eye than the one from the grid viewpoint

45. E45 Recursive Subdivision Algorithm 2k 2k+1 2k+2 2k+1 Even to odd Odd to even

46. E46 Example Grid 3D grid of 1557 viewpoints Wireframe rendering

47. E47 Sample Path 300 Frame View Frustum Culling Images Present Polygons Power Plant, 2M tris, SGI Onyx 2 w/ IR

48. E48 Images Per Triangle 0 0.001 0.002 0.003 0.004 01020304050 Maximum Rendered Triangles (% of Model) Images Per Model Triangle Power Plant Torpedo Room Brooks House Pipes Best Case Worst Case

49. E49 Preprocessing Summary

50. E50 Video Segment V Automatically Bounding Geometric Complexity by Using ImagesAutomatically Bounding Geometric Complexity by Using Images –Aliaga99

51. E51 Cells and Portals [Airey90, Teller91, Luebke95]

52. E52 Portal Images [Aliaga97]

53. E53 Creating Portal Images portal eye Ideal portal image would be one sampled from the current eye position

54. E54 Creating Portal Images Reference COPs Display one of a large number of pre-computed images (~120) portal

55. E55 Creating Portal Images portal Reference COPs or… Warp one of a much smaller number of reference images

56. E56 Brooks House 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 050100150200 Frame Rendering Time (sec.) View Frustum Culling Portal Culling Static Portal Images Onyx IR

57. E57 Brooks House 0 0.1 0.2 0.3 0.4 0.5 0100200300400500 Frame Rendering Time (sec.) Portal Culling Warped Portal Images Onyx IR

58. E58 Video Segment VI Architectural Walkthroughs using Portal ImagesArchitectural Walkthroughs using Portal Images –Aliaga97, Rafferty98

59. E59 Overview Replacing Geometry with ImagesReplacing Geometry with Images Displaying ImagesDisplaying Images –Texture-mapping and error metrics –Geometry and image warping –Meshes, Lightfield/Lumigraph Image PlacementImage Placement –Automatically Bounding Model Complexity –Cells and Portals ConclusionsConclusions

60. E60 Image Quality What about measuring quality?What about measuring quality? – Need a perceptual quality metric! We knowWe know –Texture-mapping: bad perspective, small distortions believable (geometry warping) –IBR: correct perspective, disocclusions –Meshes: stretching of skins

61. E61 Limitations Diffuse illuminationDiffuse illumination –Deferred shading? Static modelsStatic models –Incremental updating? Cannot sample all visible surfacesCannot sample all visible surfaces –Smarter reconstruction/resampling? Can only sample surfaces at a fixed resolutionsCan only sample surfaces at a fixed resolutions –Multi-resolution reference images?

62. E62 Acknowledgments Authors of the Video SegmentsAuthors of the Video Segments ModelsModels –Discreet Logic, UNC Walkthrough Group UNC-Chapel HillUNC-Chapel Hill –Walkthrough, PixelFlow, ImageFlow NSF, NIH, DARPANSF, NIH, DARPA Lucent Technologies Bell LabsLucent Technologies Bell Labs