Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays SAIT – INRIA collaboration Period: 15 July 2012 / 15 January 2013 Date: 3-4 December 2012

INRIA team Georgios Evangelidis, postdoc, 100% Michel Amat, development engineer, 100% Soraya Arias, senior development engineer, 20% Jan Cech, posdoc, 20% Radu Horaud, 10%

Past achievements A method and software for aligning TOF data with a stereoscopic camera pair Extension to the calibration of several TOF- stereo units 3D texture-based rendering of the TOF data using the color-image information

Publications One CVPR 2011 paper A tutorial at ICIP 2011 One Springer Briefs book just published The two teams published several other papers

Current achievements Finalization of the calibration & rectification methods/software TOF to stereo-pair mapping with filtering TOF + texture in live mode Disparity map initialization Stereo correspondence based on seed-growing Final high-resolution depth map with gap filling A paper submitted to CVPR’13

Improved Calibration New calibration board – mat sticker glued to a rigid plane – plane attached to a tripod Refined Calibration algorithm – TOF-Stereo Calibration error: <1.5 pixel Improved Rectification – Rectification error: <0.25 pixel

Given a calibrated TOF-Stereo system Each TOF point P T defines a correspondence between P L and P R

Correspondences (samples) obtained by using the calibration parameters each correspondence comes from a TOF point different color -> different depth

Correspondences (samples) obtained by using the calibration parameters each correspondence comes from a TOF point different color -> different depth

TOF-to-Left Mapping We use the left image as reference

TOF-to-Left Mapping is not perfect Resolution mismatch

Left-to-Tof Occlusions TOF-to-Left Mapping is not perfect Left-to-Tof Occlusions: the depth decreases from left to right

Tof-to-Left Occlusions TOF-to-Left Mapping is not perfect Tof-to-Left Occlusions: the depth increases from left to right

Point Cloud filtering We reject points in left-to-tof occluded area We keep the minimum-depth points in case of overlap (due to Tof-to-left occlusions)

Disparity Map: Initialization Run Delauney-Triangulation on low-resolution point cloud

Disparity Map: Initialization Run Delauney-Triangulation on low-resolution point cloud… …and initialize the stereo disparity map It looks good, but it’s noisy and non-accurate!

Seed-Growing Idea Start from points with known disparities (seeds) and propagate the disparity to neighboring points (video?) Main issues: – What are our seeds? – What is the visiting order of seeds? – How do I propagate the message? – How the stereo and depth data are fused within this framework?

Depth-Color Fusion Built on the seed-growing idea – A :Depth data, S : Stereo data, d N : neighbor of d – For each pixel (node), find its disparity value that maximizes the posterior probability (MAP) S S A A d N d Input data Pixel with unknown disparity Range-search constraint Penalize the choice wrt to depth information Penalize the choice wrt to color information Pixel with known disparity A represents the initial estimation of d (obtained by the previous interpolation) S represents the color matching cost that corresponds to d

Depth-Color Fusion Bayes rule translates each posterior into a likelihood If likelihood terms are chosen from the exponential family, the “-log”-ness translates MAP into an energy minimization scheme S S A A d N d Input data Pixel with unknown disparity Pixel with known disparity We are currently working on these terms! Because of the uniform distribution Because of the Bayes rule

Seed-Growing Idea (revisited) For each pixel, an energy function is defined and we look for its minimizer (disparity) Main issues: – What are our seeds? the points from Tof-to-Left mapping after refinement – What is the visiting order of seeds? First visit reliable seeds (points with low energy value) – How do we propagate the message? Given the disparity of a seed, bound the disparity-range for its neighbor – How the stereo and depth data are fused within this framework? Described above

Examples White areas: unreliable matches Black areas: Occlusions

Examples (with gap filling)

Paper Submission Stereo-Depth Fusion for High-Resolution Disparity Maps. G. Evangelidis, R. Horaud, M. Amat, and S. Lee – submitted to CVPR An extended version of the CVPR submission is under preparation and it will be submitted to IEEE TPAMI in January/February 2013.

Work during the remaining month Improve the accuracy of the matching by better exploiting the color/texture information Currently the software implementation runs in offline-mode: We will provide a live-mode version at approximatively 1-2 frames/second An updated version will be available at the end of the period (~15 January 2013)

Prospects for the next collaboration (1 February 2013 – 31 January 2014) Finalize the TOF-stereo seed-growing algorithm, in particular improve the performance in non-textured areas Depth disambiguation using TOF-TOF and TOF-stereo Combine depth disambiguation with the seed-growing algorithm Perform full 3D realistic rendering with four TOF-stereo units Perform continuous 3D reconstruction with a moving TOF-stereo unit