Image-Based Visual Hulls Wojciech Matusik Chris Buehler Leonard McMillan Wojciech Matusik Chris Buehler Leonard McMillan Massachusetts Institute of Technology Laboratory for Computer Science Ramesh Raskar Steven J. Gortler University of North Carolina at Chapel Hill Steven J. Gortler Harvard University
Motivation Real-time acquisition and rendering of dynamic scenes
Previous Work Virtualized Reality (Rander’97, Kanade’97, Narayanan’98) Virtualized Reality (Rander’97, Kanade’97, Narayanan’98) Visual Hull (Laurentini’94) Visual Hull (Laurentini’94) Volume Carving (Potmesil’87, Szeliski’93, Seitz’97) Volume Carving (Potmesil’87, Szeliski’93, Seitz’97) CSG Rendering (Goldfeather’86, Rappoport’97) CSG Rendering (Goldfeather’86, Rappoport’97) Image-Based Rendering (McMillan’95, Debevec’96, Debevec’98) Image-Based Rendering (McMillan’95, Debevec’96, Debevec’98)
Contributions View-dependent image-based visual hull representation View-dependent image-based visual hull representation Efficient algorithm for sampling the visual hull Efficient algorithm for sampling the visual hull Efficient algorithm computing visibility Efficient algorithm computing visibility A real-time system A real-time system
What is a Visual Hull?
Why use a Visual Hull? Can be computed robustly Can be computed robustly Can be computed efficiently Can be computed efficiently - =background+foregroundbackgroundforeground
Rendering Visual Hulls Reference 1 Reference 2 Desired
Build then Sample Reference 1 Reference 2 Desired
Build then Sample Reference 1 Reference 2 Desired
Build then Sample Reference 1 Reference 2 Desired
Build then Sample Reference 1 Reference 2 Desired
Build then Sample Reference 1 Reference 2 Desired
Sample Directly Reference 1 Reference 2 Desired
Sample Directly Reference 1 Reference 2 Desired
Sample Directly Reference 1 Reference 2 Desired
Sample Directly Reference 1 Reference 2 Desired
Sample Directly Reference 1 Reference 2 Desired
Sample Directly Reference 1 Reference 2 Desired
Sample Directly Reference 1 Reference 2 Desired
Sample Directly Reference 1 Reference 2 Desired
Direct Sampling Advantages Line interval intersections are robust Line interval intersections are robust Direct sampling gives us exact rendering Direct sampling gives us exact rendering Can be computed efficiently in image space Can be computed efficiently in image space
Image-Based Computation Reference 1 Reference 2 Desired
Observation Incremental computation along scanlines Incremental computation along scanlines Desired Reference
Binning Epipole Sort silhouette edges into bins Sort silhouette edges into bins
Binning Epipole Sort silhouette edges into bins Sort silhouette edges into bins
Binning Sort silhouette edges into bins Sort silhouette edges into bins Epipole Bin 1
Binning Sort silhouette edges into bins Sort silhouette edges into bins Epipole Bin 2 Bin 1
Binning Sort silhouette edges into bins Sort silhouette edges into bins Epipole Bin 3 Bin 1 Bin 2
Binning Sort silhouette edges into bins Sort silhouette edges into bins Epipole Bin 4 Bin 1 Bin 2 Bin 3
Binning Sort silhouette edges into bins Sort silhouette edges into bins Epipole Bin 5 Bin 1 Bin 2 Bin 3 Bin 4
Binning Sort silhouette edges into bins Sort silhouette edges into bins Epipole Bin 5 Bin 1 Bin 2 Bin 3 Bin 4
Scanning Epipole Bin 1
Epipole Bin 2 Scanning
Epipole Bin 2 Scanning
Epipole Bin 2 Scanning
Epipole Bin 4 Scanning
Epipole Bin 5 Scanning
Coarse-to-Fine Sampling
IBVH Results Approximately constant computation per pixel per camera Approximately constant computation per pixel per camera Parallelizes Parallelizes Consistent with input silhouettes Consistent with input silhouettes
Video of IBVH
Shading Algorithm A view-dependent strategy A view-dependent strategy
Visibility Algorithm
Visibility in 2D Desired view Reference view
Visibility in 2D Desired view Reference view Front-most Points
Visibility in 2D Desired view Reference view Visible
Visibility in 2D Desired view Reference view Coverage Mask
Visibility in 2D Desired view Reference view Coverage Mask Visible
Visibility in 2D Desired view Reference view Coverage Mask Visible
Visibility in 2D Desired view Reference view Coverage Mask VisibleNot
Visibility in 2D Desired view Reference view Coverage Mask
Visibility in 2D Desired view Reference view Coverage Mask Visible
Visibility in 2D Desired view Reference view Coverage Mask
Visibility in 2D Desired view Reference view Coverage Mask VisibleNot
Shaded Visual Hulls
System Server (4x 500 Mhz) Camera Client Camera Client Camera Client Camera Client
System Server (4x 500 Mhz) Camera Client Camera Client Camera Client Camera Client Trigger Signal
System Server (4x 500 Mhz) Camera Client Camera Client Camera Client Camera Client
System Server (4x 500 Mhz) Camera Client Camera Client Camera Client Camera Client Compressed video
System Server (4x 500 Mhz) Camera Client Camera Client Camera Client Camera Client Intersection
System Server (4x 500 Mhz) Camera Client Camera Client Camera Client Camera Client Visibility
System Server (4x 500 Mhz) Camera Client Camera Client Camera Client Camera Client Shading
More IBVH Results
Future Work 3D teleconferencing 3D teleconferencing Virtual sets Virtual sets Post-production camera effects Post-production camera effects Mixed reality Mixed reality
Summary Visual hulls with texture can provide a compelling real-time visualizations Visual hulls with texture can provide a compelling real-time visualizations Visual hulls can be computed accurately and efficiently in image space Visual hulls can be computed accurately and efficiently in image space View dependent shading with visibility View dependent shading with visibility
Acknowledgements DARPA ITO Grant F DARPA ITO Grant F A generous grant from Intel Corporation A generous grant from Intel Corporation NSF Career Awards & NSF Career Awards & Tom Buehler & Kari Anne Kjølass Tom Buehler & Kari Anne Kjølass Thanks to all members of the MIT Computer Graphics Group