Presentation is loading. Please wait.

Presentation is loading. Please wait.

Multi-view real-time depth estimation based on combination of visual-hull and hybrid recursive matching HHI Wolfgang Waizenegger.

Similar presentations

Presentation on theme: "Multi-view real-time depth estimation based on combination of visual-hull and hybrid recursive matching HHI Wolfgang Waizenegger."— Presentation transcript:

1 Multi-view real-time depth estimation based on combination of visual-hull and hybrid recursive matching HHI Wolfgang Waizenegger

2 Overview Field of application: 3D Presence 2D Videoconferencing
3D Presence concept and 3D displays The camera system 3D Analysis 3D algorithmic chain Hybrid recursive matching (HRM) Visual Vull (VH) HRM and VH combination Results Hardware Conclusion and Outlook

3 3D Presence Consortium Role of each project partner: Radvision: coding
Philips: 3D displays Telefonica/HHI: 3D analysis TU/e: Usability

4 SoA of Telepresence Systems
Telepresence System by CISCO HP Halo Telepresence System Polycom TPX System 4

5 Drawbacks of conventional telepresence systems
No eye contact, e.g. it is hard to recognize who is talking to whom Misleading gestures and body language Ideal situation: Every local participant has its own view for each remote conferee Solution: Immersive 3D videoconferencing Missing eye contact (CISCO system)

6 SoA of 3D Videoconferencing
MultiView by Univ. of California,Berkeley, 2004 Virtue/im.point by Fraunhofer HHI, 2003/2004 - Low quality, less cameras -> potential to improve Real Meet Room, France Telecom R&D, 2001 6

7 The concept of 3D Presence
Multi-party 3D videoconferencing 3D multi-user auto-stereoscopic display technology Multi-party eye contact and gesture-based interaction Replace remote conferees by 3D displays Three parties Two conferees per party

8 Multi-View 3D Displays Multiple 3D views from different perspectives
Advantages: Own view for each local conferee Adapted viewing perspective 3D impression Multiple views allow conferees to switch perspective by moving the head multiple viewing cones

9 Multi-View 3D Display

10 The Multi-View Camera System
Freedom to setup a huge variaty of camera configurations Narrow and wide baseline stereo matching Combination of two narrow baseline stereo pairs -> trifocal system Improve results by combining trifocal with wide baseline or with visual hull Narrow baseline system Robust disparity estimation Consistency check by trifocal matching Wide baseline system Increased depth resolution Option to combine with Visual Hull

11 The Mock-up for Camera Configuration Testing

12 3D Analysis Chain

13 Hybrid-Recursive Matching (HRM)
update vector pixel recursion choice of best disparity disparity vector start vector left image block recursion disparity memory Die wesentliche Intention des hybrid-rekursiven Matchings ist die Kombination einer robusten Block-Rekursion mit einer adaptiven Komponente, der Pixel-Rekursion. Die Block-Rekursion liefert für unbewegte Bildinhalte sehr zuverlässige Werte, da bereits berechnete Disparitäten genutzt werden. Auftretende Bildänderungen werden durch die Pixel-Rekursion erfasst. right image 3 candidates

14 after consistency check
Trifocal system vertical narrow baseline HRM is a decent algorithm within a trifocal setup Depth resolution is to low Incorporation of wide baseline results improve depth resolution Not enough, better results needed There is another technique for which wide basline is beneficial: visual hull horizontal narrow baseline after consistency check

15 Multi-View Video Analysis Chain
Colored Visual Hull reconstruction

16 Visual Hull Techniques
Polygonal Volume based space carving (VH) Image based (IBVH) 3D Presence demands real-time processing!! Parallelization of the last two approaches on graphics hardware is straightforward!

17 IBVH Algorithm Our implementation is based on the initial work of Matusik et al. (2000) Advantages of our algorithm Improved caching strategy that allows pixel pre-selection which significantly speeds up the computation GPU only implementation using CUDA Establishes an interconnection to voxel based implementation by applying cameras at infinity.

18 IBVH interconnection to voxel based methods

19 VH vs. IBVH Timings for two GPU based implementations with different resolutions. The image upload time is included. Volume based approach from Ladikos et al (VH_Lad) Our image based approach (PPSIBVH, without pixel pre-selection IBVH) Input: Middlebury dinoRig dataset ( 48 images, 640 x 480 ) Hardware 1283 2563 5123 VH_Lad 4 x 8800GTX 99.89 ms ms - IBVH 1 x GTX280 47.9 ms 82.5 ms 280.6 ms PPSIBVH 41.6 ms 60.9 ms 150.6 ms

20 IBVH result for the dinoRig dataset
left) Voxel representation of the IBVH result (5123), right) image based depth map

21 IBVH result for a 3D Presence conferee
Timing for a typical 3D Presence setup with depth maps of 192x256 and 8 Visual Hull cameras: 10–20 msec on a single GTX280. Soares et al. use an eight CPU dual Opteron 2.2GHz machine to achieve almost the same results with 5 cameras and an octree based Visual Hull algorithm

22 Result for the combination of HRM and VH
Combination HRM and VH Result for the combination of HRM and VH

23 Combination HRM and VH (cont.)
Hrm + Vh: A: HRM recovers concavities, vh smoothes the depth map B: VH artifacts are removed by HRM C: Fingers are recovered by VH D: VH artifacts are removed by HRM

24 Realization: Hardware Overview for the 3D Presence setup
5 x PCs with dual Nehalem Xeon CPUs 2 x Geforce GTX295 per cluster node Infiniband 40GB/s interconnection

25 3D Presence System Architecture
Node_N Capture (4 cameras) Segmentation Lens un-distortion Rectification HRM (trifocal) Bilateral filtering Virtual view generation Encoding (video+depth) Networking Node_0 Node_1 Node_3 Node_VH Configuration allows an easy modification of the camera setup Encoding result: 1Mbit/sec per view Node_2

26 Inalienability of GPUs
Hardware: CPU: Intel 3.0GHz (single core computation) GPU: Geforce GTX280 Input: Images: 1024 x 768, RGB24 Depth Maps: 1024 x 768, float GPU results include up- and download times GPU CPU Lens un-distortion + rectification 2 msec 68 msec Bilateral filtering of depth map Virtual view synthesis (RGB) 11 msec 1000 msec 1 msec 150 msec

27 Virtual view generation based on estimated depth maps
Demo - Still segmentation problems ( we have no influence since it is an external module ) Virtual view generation based on estimated depth maps

28 Conclusion and Outlook
Three party immersive 3D Videoconferencing system Real-time 3D analysis for a 16 camera setup Fast IBVH algorithm which runs entirely on a single GPU Combination of trifocal HRM and VH significantly improves the results All processing runs in real-time on only 5 PCs System allows to rapidly test various camera configuration First real-time demonstrator prototype available by October 2009 Future: Full HD real-time 3D processing chain

29 Thank you! Contact:

30 References Atzpadin, N., Kauff, P. and Schreer, O.: Stereo Analysis by Hybrid Recursive Matching for Real-Time Immersive Video Conferencing, IEEE Transactions on Circuits and Systems for Video Technology, special Issue on Immersive Telecommunications, vol. 14, no. 3, pp , January 2004. Matusik, W., Buehler, C., Raskar, R., Gortler, S. J., and McMillan, L Image-based visual hulls. In Proceedings of the 27th Annual Conference on Computer Graphics and interactive Techniques International Conference on Computer Graphics and Interactive Techniques. Lakikos, A., Benhimane, S., Navab, N., Efficient Visual Hull Computation for Real-Time 3D Reconstruction using CUDA, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Anchorage, Alaska (USA), June Workshop on Visual Computer Vision on GPUs (CVGPU). Soares, L., Menier, C., Raffin, B., and Roch, J.L. Parallel adaptive octree carving for real-time 3d modeling. Poster at IEEE VR' Virtual Reality Charlotte, Northe Carolina, USA, March 2007.

Download ppt "Multi-view real-time depth estimation based on combination of visual-hull and hybrid recursive matching HHI Wolfgang Waizenegger."

Similar presentations

Ads by Google