1. Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi http://www.cs.columbia.edu/~cs4162

2. To Do / Motivation  Work hard on assignment 4  This last series of lectures covers (at a high level) some more advanced topics and areas of current research interest in modern rendering

3. Course Outline  3D Graphics Pipeline Rendering (Creating, shading images from geometry, lighting, materials) Modeling (Creating 3D Geometry)

4. Next few slides courtesy Paul Debevec; SIGGRAPH 99 course notes

7. Image-Based Modeling and Rendering  Generate new views of a scene directly from existing views  “Pure” IBR (such as lightfields): no geometric model of scene  Other IBR techniques try to obtain higher quality with less storage by building a model

8. IBR: Pros and Cons  Advantages  Easy to capture images: photorealistic by definition  Simple, universal representation  Often bypass geometry estimation?  Independent of scene complexity?  Disadvantages  WYSIWYG but also WYSIAYG  Explosion of data as flexibility increased  Often discards intrinsic structure of model?

11. IBR: A brief history  Texture maps, bump maps, env. maps [70s]  Poggio et al. MIT: Faces, image-based analysis/synthesis  Modern Era  Chen and Williams 93, View Interpolation [Images with depth]  Chen 95 Quicktime VR [Images from many viewpoints]  McMillan and Bishop 95 Plenoptic Modeling [Images w disparity]  Gortler et al, Levoy and Hanrahan 96 Light Fields [4D]  Shade et al. 98 Layered Depth Images [2.5D]  Debevec et al. 00 Reflectance Field [4D]  Inverse rendering methods (Sato,Yu,Marschner,Boivin,…)  Fundamentally, sampled representations in graphics

12. Outline  Overview of IBR  Basic approaches  Image Warping  Light Fields  Survey of some recent work

13. Warping slides courtesy Leonard McMillan

18. Outline  Overview of IBR  Basic approaches  Image Warping  [2D + depth. Requires correspondence/disparity]  Light Fields [4D]  Survey of some recent work

20. Plenoptic Function  L(x,y,z, , ,t, )  Captures all light flow in a scene  to/from any point (x,y,z),  in any direction ( ,  ),  at any time (t),  at any frequency ( )  Enough information to construct any image of the scene at any time (x,y,z) (,)(,)(,)(,) [Funkhouser]

21. Plenoptic Function Simplifications  Represent color as RGB: eliminate  Static scenes: ignore dependence on t  7D  3  5D

31. Lumigraph Postprocessing  Obtain rough geometric model  Chroma keying (blue screen) to extract silhouettes  Octree-based space carving  Resample images to two-plane parameterization

32. Lumigraph Rendering  Use rough depth information to improve rendering quality

33. Lumigraph Rendering  Use rough depth information to improve rendering quality

34. Lumigraph Rendering Without using geometry Using approximate geometry

35. Unstructured Lumigraph Rendering  Further enhancement of lumigraphs: do not use two-plane parameterization  Store original pictures: no resampling  Hand-held camera, moved around an environment

36. Unstructured Lumigraph Rendering  To reconstruct views, assign penalty to each original ray  Distance to desired ray, using approximate geometry  Resolution  Feather near edges of image  Construct “camera blending field”  Render using texture mapping

37. Unstructured Lumigraph Rendering Blending fieldRendering

38. Outline  Overview of IBR  Basic approaches  Image Warping  [2D + depth. Requires correspondence/disparity]  Light Fields [4D]  Survey of some recent work

39. LDIs  Layered depth images [Shade et al. 98] Geometry Camera Slide from Agrawala, Ramamoorthi, Heirich, Moll, SIGGRAPH 2000

40. LDIs  Layered depth images [Shade et al. 98] LDI

41. LDIs  Layered depth images [Shade et al. 98] LDI (Depth, Color)

43. Surface Light Fields  Miller 98, Nishino 99, Wood 00  Reflected light field (lumisphere) on surface  Explicit geometry as against light fields. Easier compress

44. Acquiring Reflectance Field of Human Face [Debevec et al. SIGGRAPH 00] Illuminate subject from many incident directions

45. Example Images Images from Debevec et al. 00

46. Conclusion (my views)  Real issue is compactness/flexibility vs. rendering speed  IBR is use of sampled representations. Easy to interpolate, fast to render. If samples images, easy to acquire.  Of course, for this course, some pretty fancy precomputed algorithms (because we want to handle complex lighting that changes)  IBR in pure form not really practical  WYSIAYG  Explosion as increase dimensions (8D transfer function)  Ultimately, compression, flexibility needs geometry/materials  But lots of recent work (some in course) begins to correct these issues  Right question is tradeoff compactness/efficiency  Factored representations  Understand sampling rates and reconstruction