Image or Object? Michael F. Cohen Microsoft Research

Is this real? Photo by Patrick Jennings Copyright 1995, 96, 97 Whistler B. C. Canada

A mental model Photo by Patrick Jennings Copyright 1995, 96, 97 Whistler B. C. Canada ?

Sue Vision

Is this real? Michael F. Cohen, Shenchang Eric Chen, John R. Wallace, and Donald P. Greenberg 1988, A Progressive Refinement Approach to Fast Radiosity Image Generation.

Bob Graphics

Computer Graphics Image Output Model Synthetic Camera

Real Scene Computer Vision Real Cameras Model Output

Computer Graphics Meets Computer Vision

Combined Model Real Scene Real Cameras Image Output Synthetic Camera

But, vision technology falls short Model Real Scene Real Cameras Image Output Synthetic Camera

… and so does graphics. Model Real Scene Real Cameras Image Output Synthetic Camera

Image Based Rendering Real Scene Real Cameras -or- Expensive Image Synthesis Images+Model Image Output Synthetic Camera

Ray q Constant radiance time is fixed q 5D 3D position 2D direction

All Rays q Plenoptic Function all possible images too much stuff!

Line q Infinite line q 4D 2D direction 2D position

Ray q Discretize q Distance between 2 rays Which is closer together?

Image q What is an image? q All rays through a point Panorama?

Image q 2D position of rays has been fixed direction remains

Image q Image plane q 2D position

q Image plane q 2D position Image

q Light leaving towards “eye” q 2D just dual of image Object

q All light leaving object

Object q 4D 2D position 2D direction

Object q All images

Object q Computer Vision?

Plenoptic Function q 5D q Rays

Lumigraph / Lightfield q Outside convex space q 4D Stuff Empty

Lumigraph q Inside convex space q 4D Empty Stuff

Lumigraph q How to organize capture render

Lumigraph - Organization q 2D position q 2D direction s 

Lumigraph - Organization q 2D position q 2 plane parameterization s u

Lumigraph - Organization q 2D position q 2 plane parameterization u s t s,t u,v v s,t u,v

Lumigraph - Organization  Hold s,t constant  Let u,v vary q An image s,t u,v

Lumigraph - Organization q Discretization higher res near object if diffuse captures texture lower res away captures directions s,t u,v

Lumigraph - Capture q Idea 1 Move camera carefully over s,t plane Gantry see Lightfield paper s,t u,v

Lumigraph - Capture q Idea 2 Move camera anywhere Rebinning see Lumigraph paper s,t u,v

Lumigraph - Rendering q For each output pixel determine s,t,u,v either find closest discrete RGB interpolate near values s,t u,v

Lumigraph - Rendering q For each output pixel determine s,t,u,v either use closest discrete RGB interpolate near values s u

Lumigraph - Rendering q Nearest closest s closest u draw it q Blend 16 nearest quadrilinear interpolation s u

Lumigraph - Rendering q Depth Correction closest s intersection with “object” best u closest u s u

Lumigraph - Rendering q Depth Correction quadralinear interpolation new “closest” like focus s u

Lumigraph - Rendering q Fast s,t,u,v finding scanline interpolate texture mapping shear warp s u

Lumigraph - Rendering q Fast s,t,u,v finding scanline interpolate texture mapping shear warp s u s1,u1 s2,u2

Lumigraph - Rendering q Fast s,t,u,v finding scanline interpolate texture mapping shear warp s u

Lumigraph - Rendering q Fast s,t,u,v finding scanline interpolate texture mapping shear warp s u

Lumigraph - Demo q Lumigraph Lion, Fruit Bowl, Visible Woman, Path Tracing

Lightfield - Demo q Digital Michelangelo Project Marc Levoy, Stanford University Lightfield (“night”) assembled by Jon Shade

Surface Lightfields q Turn 4D parameterization around q Leverage coherence

Surface Lightfields q Wood et al, SIGGRAPH 2000

3D Representations q Image is 2D q Lumigraph is 4D q What happened to 3D? 3D Lumigraph subset Concentric mosaics

3D Lumigraph q One row of s,t plane i.e., hold t constant s,t u,v

3D Lumigraph q One row of s,t plane i.e., hold t constant thus s,u,v a “row of images” s u,v

Concentric Mosaics q Replace “row” with “circle” of images

Concentric Mosaics

q From above

Concentric Mosaics q From above

Concentric Mosaics q Panorama

2.5D Representations q Image is 2D q Lumigraph is 4D q 3D 3D Lumigraph subset Concentric mosaics q 2.5D Layered Depth Images View Dependent Surfaces

Layered Depth Image Image Depth Layered 2.5 D ?

Layered Depth Image q Rendering from LDI Incremental in LDI X and Y Guaranteed to be in back-to-front order

Layered Depth Image q Rendering from LDI Incremental in LDI X and Y Guaranteed to be in back-to-front order

Layered Depth Image (LDI)

Tiled LDIs q Multiresolution q Directionally dependent

View Dependent Surfaces

Summary q 5D: Plenoptic Function (Ray) q 4D: Lumigraph / Lightfield q 3D: Lumigraph Subset q 3D: Concentric Mosaics q 2.5D: Layered Depth Image q 2.5D: View Dependent Models q 2D: Image

Thanks q Peter-Pike Sloan (Lumigraph) q Jonathan Shade (Lightfield, LDI) q Marc Levoy, Stanford University Michaelangelo data set