1 Image-Based Visual Hulls Paper by Wojciech Matusik, Chris Buehler, Ramesh Raskar, Steven J. Gortler and Leonard McMillan [ Vortrag von Simon Dellenbach GDV Fachseminar 2001

2 Overview (1) Motivation Basics –Viewpoint Model –Visual Hull –Epipolar Geometry Creating Image-Based Visual Hulls

3 Overview (2) Rendering IBVH System Implementation Summary & Results Future Work Personal Opinion

4 Motivation (1) Traditional computer graphics, rendering.. –static synthetic scenes (CG Images) –dynamic synthetic scenes (CG Animations) –static acquired scenes (Image-Based Rendering) Acquire and render dynamic scenes in real- time: –appropriate representation –rendering system

5 Motivation (2)

6 Viewpoint Model - Basics (1)

7 Visual Hull - Basics (2) Geometric shape obtained using silhouettes of object seen from number of views: –extruded silhouette = cone-like volume limiting the extent of object –intersection of volumes results in a visual hull –more views  better approximation of object –limitation: concavities can’t be captured (e.g. an open box looks like a solid cube)

8 Visual Hull - Basics (3)

9 Epipolar Geometry - Basics (4) The tree points [COP 1,COP 2,P] form an epipolar plane Intersection of this plane with image planes results in epipolar lines The line connecting the two centers of projection [COP 1,COP 2 ] intersects the image planes at the conjugate points e 1 and e 2 which are called epipoles

10 Epipolar Geometry - Basics (5)

11 Creating Image-Based Visual Hulls (1) Algorithm input: –set of k silhouettes (binary images) with associated viewpoints –desired viewpoint (in this case, constructed visual hull is viewpoint-dependent) Algorithm output: –sampled image of the visual hull, each pixel containing a list of occupied intervals of space

12 Creating Image-Based Visual Hulls (2) The Basic Algorithm: –cast ray into space for each pixel in the desired view of the visual hull –intersect ray with the k silhouette cones  k lists of intervals; intersect together  single list of intersections of the viewing ray with the visual hull

13 Creating Image-Based Visual Hulls (3)

14 Creating Image-Based Visual Hulls (4) Trick: due to Epipolar Geometry interval calculation can be done in image space of reference images: –3D: intersecting silhouette cone with viewing ray –2D: intersecting projected viewing ray with silhouette

15 Creating Image-Based Visual Hulls (5)

16 Rendering IBVH (1) Reference images are used as textures For each pixel: –rank reference-image texture from “best” to “worst” according to angle, take reference with lowest –avoid texturing surface points with an image whose line-of-sight is blocked by some other point of the visual hull –consider visibility during shading based on visual hull (not actual geometry)

17 Rendering IBVH (2)

18 Rendering IBVH (3)

19 System Implementation (1) Four calibrated and triggered digital cameras One desktop PC per camera for capturing and pre-processing video frames (image segmentation) Silhouette and texture information sent to central server for IBVH processing

20 System Implementation (2) Server runs IBVH intersection and shading algorithms IBVH objects can be combined with OpenGL background System runs in ‘real time’ with heavy optimization (like caching strategies for silhouette intersection)

21 System Implementation (3)

22 Summary & Results Use visual hull as object shape approximation Using silhouette information from reference views to generate view dependent visual hull Reference images are used as ‘textures’ Results: Videoclips

23 Future Work Find Techniques for blending between textures to produce smoother transitions Scale up system by using larger number of cameras Split workload on multiple servers, as algorithm parallelizes fairly much Speed up viewing ray silhouette intersections (most expensive part of the computation)

24 Personal Opinion (1) Pros: –simple technique / low-cost hardware –image-based representation partially compensates simplification problems –epipolar geometry reduces 3D-intersection problems to 2D-intersections

25 Personal Opinion (2) Cons: –texture flipping during viewpoint transitions produces ugly results –shadows are considered as part of the object –preprocessing is really expensive (85 ms for image foreground segmentation)

26 The End “If there are no questions, there won’t be any answers.” ? ? ? ? ? ?