Geometrically Stable Sampling for the ICP Algorithm Natasha Gelfand Stanford University Leslie Ikemoto UC Berkeley Szymon Rusinkiewicz Princeton University Marc Levoy 9/17/2018

Two Flavors of ICP Point-point error Point-plane error faster to converge pi qi ni pi qi ICP picks a set of corresponding points and computes the transformation that aligns two meshes by minimizing the error metric derived from the distance between them Point-point minimizes the distance between the points Point-plane minimizes the distance between points on one mesh and tangent planes at the corresponding points at the other Point-plane is a better approximation of true point-surface error when meshes are close to each other Converges better because input surfaces can slide against each other 9/17/2018

Source of Failures Complex error landscape Many local minima Usually have to start from a good guess Point selection and error metric in minimization are key to convergence Some shapes are particularly “difficult” Noise ICP is a local search algorithm, so it’s guaranteed to find the closest local minimum from its starting position. The error landscape is often complex Especially in the presence of noise in the input, there can be many local minima Landscape depends on the choice of error metric Point-point metric can have artificial local minima even if the true distance between the surfaces does not have a local minimum Point-plane error does not create such artificial local minima Point-plane error may have difficulties when features are small And when there is noise in the input data 9/17/2018

Requirements for Error Metric Planar and spherical areas with no features should not contribute to distance metric “Lock and key” features should pull the surfaces to correct relative pose Robust even as features get small and featureless regions noisy Point-plane error metric tends to perform well in alignment since it allows two meshes to “slide” against each other For example, two planes moving against each other should not incur a distance penalty, but they would with a point-point error metric. In the point-plane error metric this would be a plateau in the error landscape This is where the point-point error would lock the meshes, but point-plane has a plateau in the error landscape If there are features on the meshes, they should pull them into alignment But, since this in effect creates a shallow error landscape, it can suffer from slow convergence in the presence of noise Small contributions from noisy regions, but since they are large compared to the features, the contributions add up and prevent the features from pulling the meshes together 9/17/2018

Solution Detect when parts of the input patch are prone to sliding Use covariance analysis Concentrate samples in “lock and key” areas to improve convergence Use geometrically stable point-selection strategy Similar to normal-space sampling [Rusinkiewicz & Levoy] Detect when input has no “lock and key” features Do not align that scan pair at all Improves convergence of global registration [Ikemoto et.al] Evenly distributing samples creates a shallow error landscape around the global minimum in the presence of geometry that can slide Shallow landscape means lock and key features may be overwhelmed Find which parts of the input are the lock and key features and sample there Would also be nice if this method told us that two meshes cannot be aligned uniquely since their overlap doesn’t have any locks Useful in global registration b/c we may get the alignment wrong due to sliding and it will then propagate through the entire set of scans 9/17/2018

Minimization Equation Find (R, t) that minimize Linearizing rotations: find r=[rx ry rz]T, t=[tx ty tz]T that minimize ni qi pi 9/17/2018

Analyzing the Metric (1) Change in error from each point-pair: For some transformations e=0: “torque” “force” translation perpendicular to “force” P Q P Q 9/17/2018

Analyzing the Metric (2) Change in error from each point-pair: For some transformations e=0: “torque” “force” rotation perpendicular to “torque” P Q Q P 9/17/2018

Covariance Matrix Aligning transform is given by Cx=b, where C encodes the change in error when surfaces are moved from optimal alignment Now that we looked at the contribution from a single point-pair, what happens when we put them all together? Stack all torque-force vectors into 6x1 matrix F Covariance of F gives how much the error changes when a transform is applied to the point set Shows the error landscape around the current set of point-pairs Directions in which the error landscape is shallow corresponds to sliding transformations 9/17/2018

Sliding Directions Eigenvectors of C with small eigenvalues correspond to sliding transformations 3 small eigenvalues 2 translation 1 rotation 3 small eigenvalues 3 rotation 2 small eigenvalues 1 translation 1 rotation Some examples of common shapes that exhibit sliding in one or more directions 1 small eigenvalue 1 rotation 1 small eigenvalue 1 translation 9/17/2018

Sample Selection Goal of sampling is to produce covariance matrix C with a good condition number Equally constrain all eigenvectors Find points that well sample the small features on P Since P and Q are similar in the overlap region, use both points and normals from P to compute C Okay, we showed that we can detect if the given input meshes have large correlated featureless areas Now we want to sample the features that will pull the meshes together These are the same features that make the landscape deep around the global minimum All transformations (eigenvectors) equally constrained Analyze which features prevent P from sliding against itself Since P and Q are similar where they overlap Do not compute correspondences, use both points and normals from P 9/17/2018

Implementation (1) Estimate C from sparse uniform sampling of input mesh C = XLXT 6x6 matrix 6 eigenvectors form basis for the space of transforms 9/17/2018

Implementation (2) For each point form For each eigenvector x j compute Magnitude of the constraint pi exerts on x j Sort points in decreasing “order of influence” for each eigenvector p2 p3 p4 p1 p7 p5 p9 p8 p6 … x1 x2 x3 x4 x5 x6 Sorting is speeded up by binning 9/17/2018

Implementation (3) Let be an estimate of how well eigenvector x j is constrained by already chosen points Choose next pi from from list with the smallest corresponding t j Most unconstrained eigenvector Compute closest point qi in Q Once enough points are chosen, solve for alignment and iterate Notice that this algorithm is only an approximation: we never re-compute the matrix C Although we do not look at Q when computing the covariance matrix and sorting the points we need to perform an overlap test before updating the totals Test by computing for each point p, its closest point q \in Q and checking that it’s not on the boundary 9/17/2018

Sample Selection Small eigenvalues 9/17/2018 This patch was aligned to a copy of itself Three unstable transformations: translation in XY plane and rotation around Z It’s not guaranteed that two translations and rotation are going to be the small eigenvectors, any basis spanning that subspace may be picked, here we re-projected into translations are rotation to make the illustration simpler 9/17/2018

Sample Selection Large eigenvalues 9/17/2018 Only a few of the points in the corners need to be picked to ensure that the mesh does not rotate out of the plane 9/17/2018

Results (Planar Patches) Without noise, any sampling works fine Similar to normal-space sampling for planar regions For the planes example Condition number under uniform sampling = 66.1 Condition number after stable sampling = 3.7 (30% of the points were selected) 9/17/2018

Results (Planar Patches) 25 iterations to converge Naïve implementation about 3 times slower per iteration than uniform sampling uniform sampling stable sampling FUR: 25 iterations to converge while sampling 10% of the points Each iteration is 5 second for stable sampling and 1.5 for uniform But our implementation can be improved, most obviously by caching closest points from the stability computation stage into the alignment stage 9/17/2018

Results (Spherical Patches) Normal-space sampling cannot align this Because the sliding is rotational, normal space sampling will not align this In the sphere, all normals are well distributed so normal-space and uniform are going to behave similarly Initial condition number = 26.1 Final condition number = 4.1 9/17/2018

Conclusions Point-plane error metric can have shallow error landscape when registering shapes with small features Noise adds many small local minima These conditions can be detected by analyzing the covariance matrix of the minimization equation Eigenvectors with small eigenvalues give sliding directions Stable sampling improves these landscapes by equally constraining all eigenvectors Selects point samples inside the features Allows ICP to align “difficult” input 9/17/2018

Limitations: Noise The algorithm can fail if input data is particularly noisy Smoothing can help One of the failure modes of the algorithm is in the presence of noise The algorithm can favor noisy areas because they will look like they have large influence on the covariance matrix Noise looks like good features to sample Smoothing can help, but only is the size of the noise is smaller than the size of the features The groove in this picture is 10mm wide and the mean height of the noise is 0.2 mm 9/17/2018

Limitations: Noise Success depends on relative size of features If the size of the noise is size of the features are similar, we’ll smooth both away and the sampling will look random The groove in this picture is 1 mm wide and the mean height of the noise is 0.2 mm The groove disappears after 6 iterations of simple smoothing by averaging with the neighbors 9/17/2018

Extensions and Future Work Use stability analysis to weigh mesh pairs in global registration A hierarchical method for aligning warped meshes [Ikemoto et al] Stable sampling for global registration Simultaneous covariance analysis of the entire set of scans Weighing eigenvectors based on the area outside the overlap Large vs. small leverage on the entire scan 9/17/2018

Extensions and Future Work Use of stability analysis as shape descriptor Segmentation into areas with similar sliding transformations 9/17/2018