CSCE 641 Computer Graphics: Image-based Rendering (cont.) Jinxiang Chai

Outline Light field rendering Plenoptic sampling (light field sampling) 3D light field (concentric mosaics) Others

Q: How many images are needed for anti-aliased light field rendering? Review: Plenoptic Sampling

Q: How many images are needed for anti-aliased light field rendering? Review: Plenoptic Sampling A: formulate this as high-dimensional signal (4D Plenoptic function) reconstruction and sampling problem

Review: Between Two Planes Z vt t v Z1Z1 Z1Z1 Z2Z2 Z2Z2

Review: Minimal Sampling Rate 1/∆t 1/∆v Image resolution Sample interval

Review: Minimal Sampling Rate 1/∆T max >=Ω v *(f/z min -f/z max ) ΩvΩv

Review: Minimal Sampling Rate Minimal sampling rate depends on: - texture of object (Ω v ) - focal length (f) - depth complexity (z min, z max ) 1/∆T max >=Ω v *(f/z min -f/z max )

3D Plenoptic Function Image/panorama is 2D Light field/lumigraph is 4D What happens to 3D?

3D Plenoptic Function Image/panorama is 2D Light field/lumigraph is 4D What happens to 3D? - 3D light field subset - Concentric mosaic [Siggraph99]

3D light field One row of s,t plane i.e., hold t constant s,t u,v

3D light field One row of s,t plane i.e., hold t constant thus s,u,v a “row of images” s u,v

Concentric mosaics [Shum and He] Polar coordinate system: - hold r constant - thus (θ,u,v)

Concentric mosaics Why concentric mosaic? - easy to capture - relatively small in storage size - inside looking out

Concentric mosaics From above How to capture images?

Concentric mosaics From above How to capture images?

Concentric mosaics From above How to render a new image?

Concentric mosaics From above How to render a new image?

Concentric mosaics From above How to render a new image? - for each ray, retrieval the closest captured rays

Concentric mosaics From above How to render a new image? - for each ray, retrieval the closest captured rays

Concentric mosaics From above How to render a new image? - for each ray, retrieval the closest captured rays

Concentric mosaics From above How to render a new image? - for each ray, retrieval the closest captured rays How about this ray?

Concentric mosaics From above object How to retrieve the closest rays?

Concentric mosaics From above object (s,t) interpolation plane How to retrieve the closest rays?

Concentric mosaics From above object (s,t) interpolation plane How to retrieve the closest rays? What’s the optimal interpolation radius?

Concentric mosaics From above object (s,t) interpolation plane How to retrieve the closest rays? What’s the optimal interpolation radius? 2r min r max /(r min +r max )

Concentric mosaics From above object (s,t) interpolation plane How to retrieve the closest rays?

Concentric mosaics From above object (s,t) interpolation plane How to retrieval the closest rays?

Concentric mosaics From above object (s,t) interpolation plane How to retrieval the closest rays?

Concentric mosaics From above object (s,t) interpolation plane How to synthesize the color of rays?

Concentric mosaics From above object (s,t) interpolation plane How to synthesize the color of rays? - bilinear interpolation

Concentric mosaics From above

Concentric mosaics From above

Concentric mosaics From above

Concentric mosaics What are limitations?

Concentric mosaics What are limitations? - limited rendering region? - large vertical distortion

Concentric mosaics What are limitations? - limited rendering region? - large vertical distortion

2.5 D representation Image is 2D Light field/lumigraph is 4D 3D - a subset of light field - concentric mosaics 2.5D - layered depth image [Shade et al, SIGGRAPH98] - view-dependent surfaces

Layered depth image [Shade et al, SIGGRAPH98] Layered depth image: - image with depths

Layered depth image [Shade et al, SIGGRAPH98] Layered depth image: - rays with colors and depths

Layered depth image [Shade et al, SIGGRAPH98] Layered depth image: (r,g,b,depth) - image with depths - rays with colors and depths

Layered depth image [Shade et al, SIGGRAPH98] Rendering from layered depth image

Layered depth image [Shade et al, SIGGRAPH98] Rendering from layered depth image - Incremental in X and Y - Guaranteed to be in back-to-front order - Forward warping one pixel with depth

Layered depth image [Shade et al, SIGGRAPH98] Rendering from layered depth image - Incremental in X and Y - Guaranteed to be in back-to-front order - Forward warping one pixel with depth

Layered depth image [Shade et al, SIGGRAPH98] Rendering from layered depth image - Incremental in X and Y - Guaranteed to be in back-to-front order - Forward warping one pixel with depth How to deal with occlusion/visibility problem?

How to form LDIs Synthetic world with known geometry and texture - from multiple depth images - modified ray tracer Real images - reconstruct geometry from multiple images (e.g., voxel coloring, stereo reconstruction) - form LDIs using multiple images and reconstructed geometry

2.5 D representation Image is 2D Light field/lumigraph is 4D 3D - a subset of light field - concentric mosaics 2.5D - layered depth image [Shade et al, SIGGRAPH98] - view-dependent surfaces

View-dependent surface representation From multiple input image - reconstruct the geometry - view-dependent texture

View-dependent surface representation From multiple input image - reconstruct the geometry - view-dependent texture

View-dependent surface representation From multiple input image - reconstruct the geometry - view-dependent texture

View-dependent surface representation From multiple input image - reconstruct the geometry - view-dependent texture

View-dependent texture mapping [Debevec et al 98]

View-dependent texture mapping Subject's 3D proxy points V C 0 C 2 C 3 C 1  0  1 D  2  3 - Virtual camera at point D - Textures from camera C i mapped onto triangle faces - Blending weights in vertex V - Angle θ i is used to compute the weight values: w i = exp(-θ i 2 /2σ 2 )

2.5 D representation Image is 2D Light field/lumigraph is 4D 3D - a subset of light field - concentric mosaics 2.5D - layered depth image [Shade et al, SIGGRAPH98] - view-dependent surfaces

Videos: view-dependent texture mapping

The Image-Based Rendering Problem Synthesize novel views from reference images Static scenes, fixed lighting Flexible geometry and camera configurations

The ULR Algorithm [Siggraph01] Designed to work over a range of image and geometry configurations Geometric Fidelity # of Images VDTM LF

The ULR Algorithm [Siggraph01] Designed to work over a range of image and geometry configurations Geometric Fidelity # of Images VDTM LF ULR

The ULR Algorithm [Siggraph01] Designed to work over a range of image and geometry configurations Designed to satisfy desirable properties Geometric Fidelity # of Images VDTM LF ULR

Desired Camera “Light Field Rendering,” SIGGRAPH ‘96 u0u0 s0s0 u s Desired color interpolated from “nearest cameras”

Desired Camera “Light Field Rendering,” SIGGRAPH ‘96 u s Desired Property #1: Epipole consistency

Desired Camera “The Scene” “The Lumigraph,” SIGGRAPH ‘96 u Potential Artifact

“The Scene” “The Lumigraph,” SIGGRAPH ‘96 Desired Property #2: Use of geometric proxy Desired Camera

“The Lumigraph,” SIGGRAPH ‘96 “The Scene” Desired Camera

“The Lumigraph,” SIGGRAPH ‘96 “The Scene” Rebinning Note: all images are resampled. Desired Camera Desired Property #3: Unstructured input images

“The Lumigraph,” SIGGRAPH ‘96 “The Scene” Desired Property #4: Real-time implementation Desired Camera

View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98 “The Scene” Occluded Out of view

Desired Camera “The Scene” Desired Property #5: Continuous reconstruction View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98

Desired Camera “The Scene” θ1θ1 θ2θ2 θ3θ3 View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98

Desired Camera “The Scene” θ1θ1 θ2θ2 θ3θ3 Desired Property #6: Angles measured w.r.t. proxy View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98

“The Scene” Desired Camera

“The Scene” Desired Property #7: Resolution sensitivity

Unstructured Lumigraph Rendering 1.Epipole consistency 2.Use of geometric proxy 3.Unstructured input 4.Real-time implementation 5.Continuous reconstruction 6.Angles measured w.r.t. proxy 7.Resolution sensitivity

Demo