Ray Divergence-Based Bundle Adjustment Conditioning for Multi-View Stereo Mauricio Hess-Flores 1, Daniel Knoblauch 2, Mark A. Duchaineau 3, Kenneth I. Joy 4, Falko Kuester 5 Abstract - An algorithm that shows how ray divergence in multi-view stereo scene reconstruction can be used towards improving bundle adjustment weighting and conditioning is presented. Starting with a set of feature tracks, ray divergence when attempting to compute scene structure for each track is first obtained. Assuming accurate feature matching, ray divergence reveals mainly camera parameter estimation inaccuracies. Due to its smooth variation across neighboring feature tracks, from its histogram a set of weights can be computed that can be used in bundle adjustment to improve its convergence properties. It is proven that this novel weighting scheme results in lower reprojection errors and faster processing times than others such as image feature covariances, making it very suitable in general for applications involving multi-view pose and structure estimation. 1,4 Institute for Data Analysis and Visualization, University of California - Davis, Davis, CA, USA Introduction AlgorithmResults (continued) Multi-view scene reconstruction is currently a prominent research area, with many applications. Despite accurate feature matching, reconstruction accuracy relies on accurate camera calibration. In the absence of ground-truth, algorithms usually resort to bundle adjustment to reduce reprojection error, which is very expensive and needs a good starting point. Main contribution: show how simple reconstruction ray divergence is an inexpensive yet powerful tool that can aid in bundle adjustment convergence. Ray divergence: shortest distance between rays from each respective camera center C i and through each pixel position of a given feature match This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA Lawrence Livermore National Laboratory, Livermore, CA, USA For SIFT feature matches between two images, eliminate epipolar geometry outliers with RANSAC. Compute ray divergence per inlier feature match. ‘Good’ baselines are necessary, such as those obtained through frame decimation. Our weighting scheme provides the best combination of final reprojection error, processing time and computational complexity. Beder’s algorithm (UWBA and UIBA) involves matrix inversion and SVD, while the feature covariance method CBA requires multiple exponential evaluations for each covariance matrix. Input images Palmdale dataset Reconstructions with our method: original point clouds (left) and adding Furukawa’s PMVS (right): Medusa dataset (a) (b) LLNL-POST Ray divergence computation DatasetCBAUWBAUIBARDBA Stockton 43, 0.621, , 0.171, , 0.072, , 0.015, 0.78 Palmdale 23, 4.687, , 1.692, , 0.831, , 0.113, 0.37 Castle-P19 150, , , 4150, , , , , 0.62 Dinosaur 26, 2.631, , 0.286, , 0.09, , 0.162, 0.05 Megascene1 49, 12.14, , 0.179, , 0.074, , 0.124, 0.04 Coneland 150, 28052, , , , , , 81.86, ,5 University of California - San Diego, San Diego, USA Features Ray divergence ‘d’ per feature match Assuming accurate feature matching, ray divergence varies smoothly, and is a function of all camera intrinsic, extrinsic and radial distortion inaccuracies. Dense, smooth ray divergence surfaces Ray divergence correlates well with Beder et al.’s confidence measure for scene structure 1, but is cheaper to compute and includes radial distortion. Ray divergences – Palmdale image pair. Reconstruction confidence measure 1 – Palmdale image pair. Encodes errors at all stages. Ray divergence histogram Ray divergences vary smoothly – their histogram can be modeled as a Gaussian probability density function. Weighted bundle adjustment Ray divergence histogram Bundle adjustment (a) minimizes reprojection error by solving the normal equations (b) at each iteration. Gaussian pdf values for each feature match are input to weighted bundle adjustment (c) as 2x2 covariance matrices into the diagonal matrix ∑ x. High divergences correspond to lower pdf values and are ‘penalized’ as lower weights, thus improving bundle adjustment convergence. Multi-view extension In sequential multi-view reconstruction, for each feature in a new image we assign the Gaussian- based weight corresponding to the ray divergence statistics for the feature matches to the prior image. Megascene1 dataset Stockton dataset Iterations I, final total reprojection error R (pixels) and processing time t (seconds) in (I,R,t) format for the four different weighting schemes: Ray directions: Closest-distance points on each ray: (x 1,y 1 ) ( x 2,y 2 ) (x 1,y 1 ) (x 2,y 2 ) K i = intrinsics R i = absolute rotation t i = parametric distance (c) Results Processing times and reprojection errors obtained for weighted bundle adjustment using ray divergences (RDBA) was compared versus other weighting schemes: image feature covariances (CBA) and reconstruction confidence 1 with (UWBA) and without (UIBA) including image feature covariances. 1 Beder, C. and Steffen, R. Determining an initial image pair for fixing the scale of a 3d reconstruction from an image sequence. In: DAGM Symposium Berlin, Germany 2006, S Lighter colors = higher values