1. CS 395: Adv. Computer Graphics Overview: Image-Based Modeling and Rendering Jack Tumblin jet@cs.northwestern.edu

2. GOAL: First-Class Primitive Want images as ‘first-class’ primitivesWant images as ‘first-class’ primitives –Useful as BOTH input and output –Convert to/from traditional scene descriptions Want to mix real & synthetic scenes freelyWant to mix real & synthetic scenes freely Want to extend photographyWant to extend photography –Easily capture scene: shape, movement, surface/BRDF, lighting … –Modify & Render the captured scene data --BUT----BUT-- images hold only PARTIAL scene informationimages hold only PARTIAL scene information –You can’t always get what you want” –You can’t always get what you want” –(Mick Jagger 1968)

3. for a given scene, describe:for a given scene, describe: –ALL rays through –ALL pixels, of –ALL cameras, at –ALL wavelengths, –ALL time F(x,y,z, , ,, t) “Eyeballs Everywhere” function (5 or 7-D!) Plenoptic Function (Adelson, Bergen `91) … … … … … … … … … ……

4. ‘Scene’ causes Light Field Light field: holds all outgoing light rays Shape,Position,Movement, BRDF,Texture,Scattering EmittedLight Reflected,Scattered, Light … Scene modulates outgoing light; light field captures it all.

5. A Big Light Field Question: Image entraps a partial scene description,… –Computer Vision problem: 3D->2D –Image point  scene surface point (usually) –Occlusion hides some scene surfaces –(BRDF * irradiance) tough to split apart! ? Does Plenoptic fcn. contain full scene ? –Exhaustive record of all image rays –Even SIMPLEST scene is huge, redundant, –The ‘consequences’ of all possible renderings* so

6. A Big Light Field Answer: Image entraps a partial scene description –Many-to-One map; 3D->2D –Occlusion hides some scene features –(BRDF * irradiance) tough to split! –limited resolution ? Does Plenoptic fcn. contain full scene ? Two Options for light field methods: –Find a limited subset of scene info, –Use MORE than plenoptic function data: (vary lights, etc.) !NO!

7. Shape Problems: Correspondence Can you find ray intersections? Or ray depth? Correspondence Problem: Ray colors might not match for non-diffuse materials (BRDF)

8. Shape Problems: Correspondence Can you find ray intersections? Or ray depth? Correspondence Problem: Ray colors might not match for non-diffuse materials (BRDF)

9. It gets worse… A ‘Circular problem’: PLUS! depth-of-focus, sampling, indirect illum… Shape BRDF Irradiance Surface Normal

10. 8-to-10-Dimensional Ideal? Light field(4D) + light sources(4D) + time + Light field(4D) + light sources(4D) + time + Shape,Position,Movement, BRDF,Texture,Scattering EmittedLight Reflected,Scattered, Light …

11. Linking illumination rays to light field rays might do it;Linking illumination rays to light field rays might do it; Swaps ‘correspondence’ problem for ‘massive data problem’.Swaps ‘correspondence’ problem for ‘massive data problem’. Shortcuts, simplifications?Shortcuts, simplifications? F(x c,y c,  c,  c,x l,y l  l,  l,, t) Full Plenoptic Fcn? MAYBE... camera projector

12. Practical IBMR What useful partial solutions are possible? Texture Maps++:Texture Maps++: Image(s)+Depth: (3D shell)Image(s)+Depth: (3D shell) Estimating Depth & SilhouettesEstimating Depth & Silhouettes ‘Light Probe’ measures real-world light‘Light Probe’ measures real-world light Light control measures BRDFLight control measures BRDF Hybrids: BTF, stitching, …Hybrids: BTF, stitching, …

13. Texture Maps ++ Re-use rendering results: ‘Impostors’, ‘Billboards’, ‘3D sprites’ Render portion of scene as a textureRender portion of scene as a texture Apply to mesh or plane  to C.O.P.;Apply to mesh or plane  to C.O.P.; Replace if eyepoint changes too muchReplace if eyepoint changes too much

14. Images + Depth 1 Image + Depth: a ‘thin shell’1 Image + Depth: a ‘thin shell’ –Reprojection (well known); Z-buffers can help –McMillan`95: 4-way raster ensures depth order –Problem: ‘holes’, occlusion, matching Multiple Images:Multiple Images: –LDI, LDI trees for multiresolution Limitations:Limitations: –Presumes diffuse-only environment –Depth capture tough: laser TOF reflectometer, manual scanner, structured light, or …

15. Estimating Depth, Silhouettes Mildly new IBMR methods can help… Sparse, manual image correspondences (Debevec, Seitz,)Sparse, manual image correspondences (Debevec, Seitz,) Video sequences with camera motion trackingVideo sequences with camera motion tracking Image (silhouette)-based Visual Hulls, ‘voxel carving’ (VIDEO!)Image (silhouette)-based Visual Hulls, ‘voxel carving’ (VIDEO!) Mostly a Classic Computer Vision Problem: Epipolar Geometry: reduce search for correspondencesEpipolar Geometry: reduce search for correspondences Global & local tracking & alignment methods…Global & local tracking & alignment methods…

16. Light Probe: Irradiance Estimate Place mirrored ball in scene,Place mirrored ball in scene, Photograph (careful! High contrast image!)Photograph (careful! High contrast image!) Unwrap: map image position  angleUnwrap: map image position  angle Use as illumination sourceUse as illumination source Uses:Uses: –mixing real & synthetic objects (Ward 96) –separating reflectance & illum (Yu 97) –movies, movies, movies...

17. Light Stage: Debevec2001 Carefully control incoming light direction (light stages, whirling banks of lights, etc)Carefully control incoming light direction (light stages, whirling banks of lights, etc) Images vs. light( ,  )Images vs. light( ,  ) Weighted sum of images  weighted sum of lightsWeighted sum of images  weighted sum of lights Debevec et al. 2001

18. Light Stage Methods Specular Component is PolarizedSpecular Component is Polarized Scan surface geometry (before, during)Scan surface geometry (before, during) Scattered data interpolation to approx. BRDF.Scattered data interpolation to approx. BRDF. Debevec et al. 2001

19. Light Control Methods light probe gathers illum,light probe gathers illum, light stage gathers face response to light;light stage gathers face response to light; Real face artificially inserted into real sceneReal face artificially inserted into real scene

20. Conclusion Very active areaVery active area Heavy overlap with computer vision: careful not to re-invent & re-name!Heavy overlap with computer vision: careful not to re-invent & re-name! Compute-intensive, but easily parallel; applies graphics hardware to broader probs.Compute-intensive, but easily parallel; applies graphics hardware to broader probs.