Image-based Rendering

© 2002 James K. Hahn2 Image-based Rendering Usually based on 2-D imagesUsually based on 2-D images Pre-calculationPre-calculation –Pre-rendering (speed) –From real photographs (speed and realism) Usually for static scene and moving viewpointUsually for static scene and moving viewpoint Rendering time decoupled from scene complexityRendering time decoupled from scene complexity Usually based on 2-D imagesUsually based on 2-D images Pre-calculationPre-calculation –Pre-rendering (speed) –From real photographs (speed and realism) Usually for static scene and moving viewpointUsually for static scene and moving viewpoint Rendering time decoupled from scene complexityRendering time decoupled from scene complexity

© 2002 James K. Hahn3 2-D Techniques Warp reference image(s) to generate required imageWarp reference image(s) to generate required image Consider images as texture mapsConsider images as texture maps Use hardware for handling texturesUse hardware for handling textures Warp reference image(s) to generate required imageWarp reference image(s) to generate required image Consider images as texture mapsConsider images as texture maps Use hardware for handling texturesUse hardware for handling textures

© 2002 James K. Hahn4 Sprites Billboard: 2-D image that is handled as a 3-D objectBillboard: 2-D image that is handled as a 3-D object –E.g. image of a tree kept perpendicular to direction of view Imposters: generalization of billboardImposters: generalization of billboard –May be pre-calculated to correspond to each of its bounding box sides –Imposter corresponding to the side which face the viewpoint is used –Rendered as texture map –Warped as the viewpoint moves Billboard: 2-D image that is handled as a 3-D objectBillboard: 2-D image that is handled as a 3-D object –E.g. image of a tree kept perpendicular to direction of view Imposters: generalization of billboardImposters: generalization of billboard –May be pre-calculated to correspond to each of its bounding box sides –Imposter corresponding to the side which face the viewpoint is used –Rendered as texture map –Warped as the viewpoint moves

© 2002 James K. Hahn5 Error of planar imposters V0V0 V1V1 Impostor X’ x1x1 x2x2 As viewpoint moves from to no longer represents both andAs viewpoint moves from V 0 to V 1 X’ no longer represents both x 1 and x 2 If angle  is less than that subtended by pixel, acceptable error Amount of warp constrained No motion parallax As viewpoint moves from to no longer represents both andAs viewpoint moves from V 0 to V 1 X’ no longer represents both x 1 and x 2 If angle  is less than that subtended by pixel, acceptable error Amount of warp constrained No motion parallax 

© 2002 James K. Hahn6 Image layering “2 ½ D” rendering 3D scene is segmented into different layers – –Objects assigned to different layers roughly according to distance to viewer Rendering resources allocated to different “display memory” by spatial and/or temporal sampling rates – –Can be prioritized by distance or speed Each layer results in sprites that are then warped according to viewing direction Sprites are then composited into final output image “2 ½ D” rendering 3D scene is segmented into different layers – –Objects assigned to different layers roughly according to distance to viewer Rendering resources allocated to different “display memory” by spatial and/or temporal sampling rates – –Can be prioritized by distance or speed Each layer results in sprites that are then warped according to viewing direction Sprites are then composited into final output image

© 2002 James K. Hahn7 Using depth information Layers or sprites with depth information (not per-pixel depth) Images with z-buffer (per-pixel depth) Layered depth images (LDI) – –Single view of scene with multiple pixels along each line of sight – –Complexity a function of depth complexity (average number of surfaces that project onto a pixel) Layers or sprites with depth information (not per-pixel depth) Images with z-buffer (per-pixel depth) Layered depth images (LDI) – –Single view of scene with multiple pixels along each line of sight – –Complexity a function of depth complexity (average number of surfaces that project onto a pixel)

© 2002 James K. Hahn8 Images with z-buffer For each I(x,y) warp to I(x’,y’) as viewpoint moves to a new location – –x’, y’ a function of x, y, z, and transformation of viewpoint Image folding problem: more than one pixel in the reference image maps into a single pixel in extrapolated view Holes due to occluded point in reference image becoming visible in extrapolated view Holes due to “stretching” For each I(x,y) warp to I(x’,y’) as viewpoint moves to a new location – –x’, y’ a function of x, y, z, and transformation of viewpoint Image folding problem: more than one pixel in the reference image maps into a single pixel in extrapolated view Holes due to occluded point in reference image becoming visible in extrapolated view Holes due to “stretching”

© 2002 James K. Hahn9 Layered depth images (LDI) 3-D structure for a particular viewpoint For each pixel, store information for all surfaces that it intersects – –Color, surface normal, depth Generated by ray-tracing or warping n images (with depth information) from different viewpoints During rendering, incremental warp of each layer in back to front order 3-D structure for a particular viewpoint For each pixel, store information for all surfaces that it intersects – –Color, surface normal, depth Generated by ray-tracing or warping n images (with depth information) from different viewpoints During rendering, incremental warp of each layer in back to front order

© 2002 James K. Hahn10 View interpolation Frames required for walkthrough from – –set of reference images – –warp script that describe corresponding pixels (pixel motion) View morphing – –Generate interpolated view from reference images – –Interpolated transformation that preserve object shape – –Need to know camera parameters Frames required for walkthrough from – –set of reference images – –warp script that describe corresponding pixels (pixel motion) View morphing – –Generate interpolated view from reference images – –Interpolated transformation that preserve object shape – –Need to know camera parameters

© 2002 James K. Hahn11 Lumigraph (light field rendering) For each point in the scene, pre-calculate and store the radiance in every direction at that point Assume occluder-free space (along a ray, radiance is constant) Parameterized by 4-D function: two parallel planes with (s, t) and (u, v) parameterization Can be generated from photography by taking a picture at discrete points in (s, t) For each point in the scene, pre-calculate and store the radiance in every direction at that point Assume occluder-free space (along a ray, radiance is constant) Parameterized by 4-D function: two parallel planes with (s, t) and (u, v) parameterization Can be generated from photography by taking a picture at discrete points in (s, t)

© 2002 James K. Hahn12 Photo-modeling Generation of 3-D model from photography Allow rich textures to be used from real world Much user intervention by specifying correspondence with known geometry Generation of 3-D model from photography Allow rich textures to be used from real world Much user intervention by specifying correspondence with known geometry

© 2002 James K. Hahn13 Photographic panorama E.g. Apple Computer’s QuickTime VR Individual images stitched into cylindrical panorama Given a viewpoint, can pan in any direction in real- time E.g. Apple Computer’s QuickTime VR Individual images stitched into cylindrical panorama Given a viewpoint, can pan in any direction in real- time

© 2002 James K. Hahn14