1. Introduction to Image-Based Rendering Jian Huang, CS 594, Spring 2002 A part of this set of slides reference slides used at Standford by Prof. Pat Hanrahan and Philipp Slusallek.

2. What is Image- Based Rendering? Not just using images on geometry (akin to texture mapping) Built on desire to bypass the manual modeling phase Use images (of some kind) for modeling and rendering

3. Types of IBR Panoramas/Image Mosaics/Light Fields, Lumigraph –QuicktimeVR –Concentric Mosaics, light fields/lumigraph View Interpolation Model based methods –Depth Images –Geometry from Images

4. Plenoptic Function Plenoptic function (7D) depicts light rays passing through: –center of camera at any location (x,y,z) –at any viewing angle ( ,  ) –for every wavelength ( ) –for any time ( t )

5. Limiting Dimensions of Plenoptic Functions Plenoptic modeling (5D) : ignore time & wavelength Lumigraph/Lightfield (4D) : constrain the scene (or the camera view) to a bounding box 2D Panorama : fix viewpoint, allow only the viewing direction and camera zoom can be changed

6. Limiting Dimensions of Plenoptic Functions Concentric mosaics (3D) : index all input image rays in 3 parameters: radius, rotation angle and vertical elevation

7. Apple’s QuickTime VR OutwardInward

8. Mars Pathfinder Panorama

9. Creating a Cylindrical Panorama From www.quicktimevr.apple.com

10. Commercial Products –QuickTime VR, LivePicture, IBM (Panoramix) –VideoBrush –IPIX (PhotoBubbles), Be Here, etc.

11. Light Field and Lumigraph Take advantage of empty space to –reduce Plenoptic Function to 4D

12. Capturing Lightfields Need a 2D set of (2D) images Choices: –Camera motion: human vs. computer –Constraints on camera motion –Coverage and sampling uniformity –Aliasing

13. Point / angle Two points on a sphere Points on two planes Original images and camera positions Lightfield Parameterization

14. Two Plane Parametrization Object Focal plane (st) Camera plane (uv)

15. Reconstruction

17. Light Field Key Ideas: n4D function - Valid outside convex hull n2D slice = image - Insert to create - Extract to display nInward or outward

18. Lightfields Advantages: –Simpler computation vs. traditional CG –Cost independent of scene complexity –Cost independent of material properties and other optical effects Disadvantages: –Static geometry –Fixed lighting –High storage cost

19. Concentric Mosaics Concentric mosaics : easy to capture, small in storage size

20. Concentric Mosaics A set of manifold mosaics constructed from slit images taken by cameras rotating on concentric circles

21. Sample Images

22. Rendering a Novel View

23. Construction of Concentric Mosaics Synthetic scenes –uniform angular direction sampling –square root sampling in radial direction

24. Construction of Concentric Mosaics (2) Real scenes Bulky, costly Cheaper, easier

25. Construction of Concentric Mosaics (3) Problems with single camera: –Limited horizontal fov –Non-uniform spatial horizontal resolution Resampling necessary –bilinear is better than point sampling Video sequence can be compressed with VQ and entropy encoding (25X) Compressed stream gives 20fpx on PII300

26. Results

27. Results (2)

28. View Interpolation Sprites/Imposters with Depth –Better image warping: Wider range of reuse Backward mapping only with homograph –New mapping: Stored depth map Forward map depth map (approximate geometry) Backward mapping of color using depth information d d’

29. Mapping with Depth Forward Mapping: –Holes and aliasing I 1 d 1 (I 2 ) I 2 

30. Mapping with Depth Backward Mapping: –What is d? I 1 (I 2 ) d 2 I 2 

31. Mapping with Depth Solution: –Forward map depth –Reconstruct approximate geometry –Backward map color I 1 (I 2 ) d 2 I 2 

32. Layered Depth Images Idea: –Handle disocclusion –Store invisible geometry in depth images Data structure: –Per pixel list of depth samples –Per depth sample: RGBA Z Encoded: Normal direction, distance –Pack into cache lines

33. Layered Depth Images Computation: –Incremental warping computation –Implicit ordering information Process in up to four quadrant –Splat size computation Table lookup Fixed splat templates –Clipping of LDIs

34. Layered Depth Images

35. Model-based IBR Basic Idea: Sparse set of images [Debevec’97, Pulli’96] Overview: –Approximate Modeling Photogrammetric modeling Triangulated depth maps –View-dependent Texture Mapping Weighting Hardware accelerated rendering –Model-based Stereo Details from stereo algorithms

36. Hybrid Approach Courtesy: P. Debevec

37. Approximate Modeling User-assisted photogrammetry [Debevec ‘97]: –Based on “Structure from Motion” –Use constraints in architectural models Approach: –Simple block model –Constraints reduce DOF –Matching based on lines –Non-linear optimization –Initial Camera Positions

38. Approximate Modeling: Block Model Courtesy: P. Debevec

39. Approximate Modeling Active Light: –Calibrated camera and projector –Plane of light and triangulation –Registration of multiple views –Triangulation of point cloud Projector Camera

40. Approximate Modeling

41. Projecting Images Technique: –Known camera positions –Projective texture mapping –Shadow buffer for occlusions –Blending between textures –Filling in

42. Visibility

43. Projecting Images

44. Simple compositing vs. blending Blending: –Select “best” image closeness to viewing direction distance to border sampling density [Pulli] deletion of features Some computation in HW –Smooth transition between pixels and frames Alpha blending, soft Z-buffer, confidence

45. Projecting Images Closeness to viewing direction: –Triangulate the Hemisphere Delaunay triangulation of viewing directions Regular triangulation: label each vertex with best view –Interpolate based on barycentric coordinates

46. Blending of Textures

47. Model-Based Stereo Problems with conventional stereo algorithms: –Correspondences are difficult to find –Large disparities –Foreshortening, projective distortions Approach: –Use approximate geometry to reproject one image –Compute disparity of warped image Significant smaller disparity and foreshortening

48. Model-Based Stereo

51. Demos