Presentation on theme: "Multi-view real-time depth estimation based on combination of visual-hull and hybrid recursive matching HHI Wolfgang Waizenegger."— Presentation transcript:
1 Multi-view real-time depth estimation based on combination of visual-hull and hybrid recursive matching HHI Wolfgang Waizenegger
2 Overview Field of application: 3D Presence 2D Videoconferencing 3D Presence concept and 3D displaysThe camera system3D Analysis3D algorithmic chainHybrid recursive matching (HRM)Visual Vull (VH)HRM and VH combinationResultsHardwareConclusion and Outlook
3 3D Presence Consortium Role of each project partner: Radvision: coding Philips: 3D displaysTelefonica/HHI: 3D analysisTU/e: Usability
4 SoA of Telepresence Systems Telepresence System by CISCOHP Halo Telepresence SystemPolycom TPX System4
5 Drawbacks of conventional telepresence systems No eye contact, e.g. it is hard to recognize who is talking to whomMisleading gestures and body languageIdeal situation:Every local participant has its own view for each remote confereeSolution:Immersive 3D videoconferencingMissing eye contact (CISCO system)
6 SoA of 3D Videoconferencing MultiView by Univ. of California,Berkeley, 2004Virtue/im.point by Fraunhofer HHI, 2003/2004- Low quality, less cameras -> potential to improveReal Meet Room, France Telecom R&D, 20016
7 The concept of 3D Presence Multi-party 3D videoconferencing3D multi-user auto-stereoscopic display technologyMulti-party eye contact and gesture-based interactionReplace remote confereesby 3D displaysThree partiesTwo conferees per party
8 Multi-View 3D Displays Multiple 3D views from different perspectives Advantages:Own view for each local confereeAdapted viewing perspective3D impressionMultiple views allow conferees toswitch perspective by moving theheadmultipleviewing cones
10 The Multi-View Camera System Freedom to setup a huge variaty of camera configurationsNarrow and wide baseline stereo matchingCombination of two narrow baseline stereo pairs -> trifocal systemImprove results by combining trifocal with wide baseline or with visual hullNarrow baseline systemRobust disparity estimationConsistency check by trifocal matchingWide baseline systemIncreased depth resolutionOption to combine with Visual Hull
13 Hybrid-Recursive Matching (HRM) update vectorpixel recursionchoice of best disparitydisparity vectorstart vectorleft imageblock recursiondisparity memoryDie wesentliche Intention des hybrid-rekursiven Matchings ist die Kombination einer robusten Block-Rekursion mit einer adaptiven Komponente, der Pixel-Rekursion. Die Block-Rekursion liefert für unbewegte Bildinhalte sehr zuverlässige Werte, da bereits berechnete Disparitäten genutzt werden. Auftretende Bildänderungen werden durch die Pixel-Rekursion erfasst.right image3 candidates
14 after consistency check Trifocal systemverticalnarrow baselineHRM is a decent algorithm within a trifocal setupDepth resolution is to lowIncorporation of wide baseline results improve depth resolutionNot enough, better results neededThere is another technique for which wide basline is beneficial: visual hullhorizontalnarrow baselineafter consistency check
15 Multi-View Video Analysis Chain Colored Visual Hull reconstruction
16 Visual Hull Techniques PolygonalVolume based space carving (VH)Image based (IBVH)3D Presence demands real-time processing!!Parallelization of the last two approaches ongraphics hardware is straightforward!
17 IBVH AlgorithmOur implementation is based on the initial work of Matusik et al. (2000)Advantages of our algorithmImproved caching strategy that allows pixel pre-selection which significantly speeds up the computationGPU only implementation using CUDAEstablishes an interconnection to voxel based implementation by applying cameras at infinity.
19 VH vs. IBVHTimings for two GPU based implementations with different resolutions. The imageupload time is included.Volume based approach from Ladikos et al (VH_Lad)Our image based approach (PPSIBVH, without pixel pre-selection IBVH)Input: Middlebury dinoRig dataset ( 48 images, 640 x 480 )Hardware128325635123VH_Lad4 x 8800GTX99.89 msms-IBVH1 x GTX28047.9 ms82.5 ms280.6 msPPSIBVH41.6 ms60.9 ms150.6 ms
20 IBVH result for the dinoRig dataset left) Voxel representation of the IBVH result (5123), right) image based depth map
21 IBVH result for a 3D Presence conferee Timing for a typical 3D Presence setup with depth maps of 192x256 and 8 Visual Hull cameras: 10–20 msec on a single GTX280.Soares et al. use an eight CPU dual Opteron 2.2GHz machine to achieve almost the same results with 5 cameras and an octree based Visual Hull algorithm
22 Result for the combination of HRM and VH Combination HRM and VHResult for the combination of HRM and VH
23 Combination HRM and VH (cont.) Hrm + Vh:A: HRM recovers concavities, vh smoothes the depth mapB: VH artifacts are removed by HRMC: Fingers are recovered by VHD: VH artifacts are removed by HRM
24 Realization: Hardware Overview for the 3D Presence setup 5 x PCs with dual Nehalem Xeon CPUs2 x Geforce GTX295 per cluster nodeInfiniband 40GB/s interconnection
25 3D Presence System Architecture Node_NCapture (4 cameras)SegmentationLens un-distortionRectificationHRM (trifocal)Bilateral filteringVirtual view generationEncoding (video+depth)NetworkingNode_0Node_1Node_3Node_VHConfiguration allows an easy modification of the camera setupEncoding result: 1Mbit/sec per viewNode_2
26 Inalienability of GPUs Hardware:CPU: Intel 3.0GHz (single core computation)GPU: Geforce GTX280Input:Images: 1024 x 768, RGB24Depth Maps: 1024 x 768, floatGPU results include up- and download timesGPUCPULens un-distortion + rectification2 msec68 msecBilateral filtering of depth mapVirtual view synthesis (RGB)11 msec1000 msec1 msec150 msec
27 Virtual view generation based on estimated depth maps Demo- Still segmentation problems ( we have no influence since it is an external module )Virtual view generation based on estimated depth maps
28 Conclusion and Outlook Three party immersive 3D Videoconferencing systemReal-time 3D analysis for a 16 camera setupFast IBVH algorithm which runs entirely on a single GPUCombination of trifocal HRM and VH significantly improves the resultsAll processing runs in real-time on only 5 PCsSystem allows to rapidly test various camera configurationFirst real-time demonstrator prototype available by October 2009Future: Full HD real-time 3D processing chain
30 ReferencesAtzpadin, N., Kauff, P. and Schreer, O.: Stereo Analysis by Hybrid Recursive Matching for Real-Time Immersive Video Conferencing, IEEE Transactions on Circuits and Systems for Video Technology, special Issue on Immersive Telecommunications, vol. 14, no. 3, pp , January 2004.Matusik, W., Buehler, C., Raskar, R., Gortler, S. J., and McMillan, L Image-based visual hulls. In Proceedings of the 27th Annual Conference on Computer Graphics and interactive Techniques International Conference on Computer Graphics and Interactive Techniques.Lakikos, A., Benhimane, S., Navab, N., Efficient Visual Hull Computation for Real-Time 3D Reconstruction using CUDA, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Anchorage, Alaska (USA), June Workshop on Visual Computer Vision on GPUs (CVGPU).Soares, L., Menier, C., Raffin, B., and Roch, J.L. Parallel adaptive octree carving for real-time 3d modeling. Poster at IEEE VR' Virtual Reality Charlotte, Northe Carolina, USA, March 2007.
Your consent to our cookies if you continue to use this website.