1. CSCE 641 Computer Graphics: Image-based Rendering (cont.) Jinxiang Chai

2. Outline Light field rendering Plenoptic sampling (light field sampling) Layered depth image/Post-Rendering 3D WarpingPost-Rendering 3D Warping View-dependent texture mapping Unstructured lumigraph

3. Layered depth image Layered depth image [Shade et al, SIGGRAPH98] Layered depth image: - image with depths

4. Layered depth image Layered depth image [Shade et al, SIGGRAPH98] Layered depth image: - rays with colors and depths

5. Layered depth image Layered depth image [Shade et al, SIGGRAPH98] Layered depth image: (r,g,b,depth) - image with depths - rays with colors and depths

6. Layered depth image Layered depth image [Shade et al, SIGGRAPH98] Rendering from layered depth image

7. Layered depth image Layered depth image [Shade et al, SIGGRAPH98] Rendering from layered depth image - Incremental in X and Y - Forward warping one pixel with depth

8. Layered depth image Layered depth image [Shade et al, SIGGRAPH98] Rendering from layered depth image - Incremental in X and Y - Forward warping one pixel with depth

9. Layered depth image Layered depth image [Shade et al, SIGGRAPH98] Rendering from layered depth image - Incremental in X and Y - Forward warping one pixel with depth How to deal with occlusion/visibility problem?

10. How to form LDIs Synthetic world with known geometry and texture - from multiple depth images - modified ray tracer Real images - reconstruct geometry from multiple images (e.g., voxel coloring, stereo reconstruction) - form LDIs using multiple images and reconstructed geometry Kinect sensors - record both image data and depth data

11. Image-based Rendering Using Kinect Sensors Capture both video/depth data using kinect sensors Using 3D warping to render a video from a novel view point [e.g., Post-Rendering 3D Warping]Post-Rendering 3D Warping Demo: click herehere

12. Outline Light field rendering Plenoptic sampling (light field sampling) Layered depth image/Post-Rendering 3D Warping View-dependent texture mapping Unstructured lumigraph

13. View-dependent surface representation From multiple input image - reconstruct the geometry - view-dependent texture

14. View-dependent surface representation From multiple input image - reconstruct the geometry - view-dependent texture

15. View-dependent surface representation From multiple input image - reconstruct the geometry - view-dependent texture

16. View-dependent surface representation From multiple input image - reconstruct the geometry - view-dependent texture

17. View-dependent texture mapping [Debevec et al 98]

18. View-dependent texture mapping Subject's 3D proxy points V C 0 C 2 C 3 C 1  0  1 D  2  3 - Virtual camera at point D - Textures from camera C i mapped onto triangle faces - Blending weights in vertex V - Angle θ i is used to compute the weight values: w i = exp(-θ i 2 /2σ 2 )

19. Videos: view-dependent texture mapping

20. Outline Light field rendering Plenoptic sampling (light field sampling) Layered depth image/Post-Rendering 3D Warping View-dependent texture mapping Unstructured lumigraph

21. The Image-Based Rendering Problem Synthesize novel views from reference images Static scenes, fixed lighting Flexible geometry and camera configurations

22. The ULR Algorithm [Siggraph01] Designed to work over a range of image and geometry configurations Geometric Fidelity # of Images VDTM LF

23. The ULR Algorithm [Siggraph01] Designed to work over a range of image and geometry configurations Geometric Fidelity # of Images VDTM LF ULR

24. The ULR Algorithm [Siggraph01] Designed to work over a range of image and geometry configurations Designed to satisfy desirable properties Geometric Fidelity # of Images VDTM LF ULR

25. Desired Camera “Light Field Rendering,” SIGGRAPH ‘96 u0u0 s0s0 u s Desired color interpolated from “nearest cameras”

26. Desired Camera “Light Field Rendering,” SIGGRAPH ‘96 u s

27. Desired Camera “The Scene” “The Lumigraph,” SIGGRAPH ‘96 u Potential Artifact

28. “The Scene” “The Lumigraph,” SIGGRAPH ‘96 Desired Property #2: Use of geometric proxy Desired Camera

29. “The Lumigraph,” SIGGRAPH ‘96 “The Scene” Desired Camera

30. “The Lumigraph,” SIGGRAPH ‘96 “The Scene” Rebinning Note: all images are resampled. Desired Camera Desired Property #3: Unstructured input images

31. “The Lumigraph,” SIGGRAPH ‘96 “The Scene” Desired Property #4: Real-time implementation Desired Camera

32. View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98 “The Scene” Occluded Out of view

33. Desired Camera “The Scene” Desired Property #5: Continuous reconstruction View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98

34. Desired Camera “The Scene” θ1θ1 θ2θ2 θ3θ3 View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98

35. Desired Camera “The Scene” θ1θ1 θ2θ2 θ3θ3 Desired Property #6: Angles measured w.r.t. proxy View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98

36. “The Scene” Desired Camera

37. “The Scene” Desired Property #7: Resolution sensitivity

38. Demo