Reverse Engineering Niloy J. Mitra

Capturing Geometry: 3D Scanners laser scanner 3D geometry

Shape Acquisition

Shape Acquisition

Shape Acquisition

Shape Acquisition

Shape Acquisition

Partial Shape Similarity partial similarity

Total vs Partial Matching Total matching is easy PCA (Principal Component Analysis)

Total vs Partial Matching Partial matching is difficult Which region matches which other region(s)? Space of rigid transforms  rotation + translation Brute force approach  not feasible Instead of exhaustive searching, use local geometry to guide where to search Easy to verify a transform

Descriptor Based Alignment Feature based alignment Combinatorial search, need multiple objects together Descriptor based alignment Fails for partial similarity [Osada et al. `02] [Kazhan et al. `03]

Registration What is registration? Finding a one-to-one mapping between two or more coordinate systems such that corresponding features of models in the different systems are mapped to each other Using the mapping to align a model(s) Pair-wise model alignment Transformation to a canonical pose/coordinate system A one-to-one mapping between the coordinates in one space and those in another such that the points in the two spaces that correspond to the same feature are mapped to each other. (from Maurer). Audette 2000 M. Kazhdan

Registration OR What is the resulting alignment/pose used for? Object recognition in scenes Stitching together parts of a model captured from different views Alignment for pose-dependent shape descriptors OR Chang and Krumm, 1999 Funkhouser, COS 597D Class Notes

Registration Choice of Transformation Rigid: mutual distances of points within a model are conserved during transformation R is a rotation matrix and t is a translation vector Non-rigid Account for surface deformations in the transformation Affine transformation Audette 2000

Normalization Use PCA to place models into a canonical coordinate frame Covariance Matrix Computation Principal Axis Alignment In the past, this problem has been addressed by trying to obtain the optimal alignment without explicitly trying all possible rotations. In particular, the models are normalized for rotation by placing each model into its own canonical coordinate frame. Two models are then compared by comparing them at their normalized rotation, and assuming that this provides the optimal pair-wise alignment. Traditionally, Principal Component Analysis has been used to provide the canonical coordinate frame. In particular, given a model, the covariance ellipsoid is computed, and the model is then rotated so that the principal axes of the ellipsoid align with x-, y-, and z-axes. M. Kazhdan

Steps for finding principal axes Translate point set {pi} to origin by center of mass: Result is new point set {qi}

Steps for finding principal axes Calculate second-order covariance matrix:

Steps for finding principal axes Decompose symmetric covariance matrix: Matrix U contains 3 principal axes (eigenvectors) as rows: A, B, C Matrix S contains eigenvalues

Aligning Scans Start with manual initial alignment [Pulli] The first thing to do for a new scan is to align or register it with the previous scans. We did an interactive rough alignment by moving the latest scan, shown in red, to approximately match the other scans. If we had tracked the gantry location, this manual alignment would have been unnecessary. Start with manual initial alignment [Pulli] S. Rusinkiewicz

Aligning Scans Improve alignment using ICP algorithm [Pulli] This is followed by automated accurate alignment with one of the neighboring scans using a variant of ICP, that is, Iterative Closest Points method. Improve alignment using ICP algorithm [Pulli] S. Rusinkiewicz

Pairwise Rigid Registration Goal Align two partially- overlapping meshes given initial guess for relative transform So, let’s formalize the rigid registration problem as solving for a rigid-body transform (i.e., translation and rotation, or 6 total degrees of freedom in 3D) that minimizes the distance between two partially-overlapping meshes. We’ll focus on iterative algorithms that converge to a local minimum, so we’ll assume that we have an initial guess already. S. Rusinkiewicz

Aligning 3D Data If correct correspondences are known, can find correct relative rotation/translation The basic idea behind ICP is that, if we somehow knew correspondences, we could solve for the translation that minimizes pairwise distance. S. Rusinkiewicz

Aligning 3D Data How to find correspondences: User input? Feature detection? Signatures? Alternative: assume closest points correspond What ICP does is make a seemingly-radical guess: that closest points are the correspondences. You take these, run the point pairs through the minimization algorithms, and get… the wrong answer. However, if you started out reasonably close, this process got you closer… S. Rusinkiewicz

Aligning 3D Data … and iterate to find alignment Iterative Closest Points (ICP) [Besl & McKay 92] Converges if starting position “close enough” … which means that you can repeat the process and converge to the right answer. S. Rusinkiewicz

Basic ICP Select e.g. 1000 random points Match each to closest point on other scan, using data structure such as k-d tree Reject pairs with distance > k times median Construct error function: Minimize (closed form solution in [Horn 87]) So, here’s a slightly formalized version of that algorithm. S. Rusinkiewicz

ICP Variants Variants on the following stages of ICP have been proposed: Selecting source points (from one or both meshes) Matching to points in the other mesh Weighting the correspondences Rejecting certain (outlier) point pairs Assigning an error metric to the current transform Minimizing the error metric w.r.t. transformation If we look at the preceeding algorithm, we can see that it consists of a bunch of different stages, and there are different choices you can make at each stage. It turns out that there’s been a lot of research on ICP and, to a good approximation, you can classify all the different tweaks to ICP that people have proposed into variants on 6 different stages. S. Rusinkiewicz

Point-to-Plane Error Metric Using point-to-plane distance instead of point-to-point lets flat regions slide along each other [Chen & Medioni 91] The idea is that instead of minimizing the distance between pairs of points, we’ll minimize the distance between one of the points and the plane passing through the other and perpendicular to its normal. (This, of course, assumes that you’ve estimated normals, which is typically done just by fitting planes to small neighborhoods of points, etc.) The big advantage of this is that it better captures the notion that sliding two planes along each other doesn’t increase the distance between them. Consider, for example, the situation shown at right. You have two scans that are mostly planar, with only the little bump constraining their left-and-right motion. Using the original point-to-point distance, however, the pairs of points in the flat region will prevent the scans from “sliding along each other” to reach the correct transformation: think of them as little springs that want to minimize their length. Using the point-to-plane distance in this case will lead to significantly faster convergence. S. Rusinkiewicz

Point-to-Plane Error Metric Error function: where R is a rotation matrix, t is translation vector Linearize (i.e. assume that sin   , cos   1): Result: overconstrained linear system So, how do we actually implement this? The first thing is to write down the error function: we’re taking the transformed version of point pi, finding the vector to qi, then taking just the component of that vector along the normal ni. This is a little nasty, since the matrix R involves some trig functions, etc., and minimizing this error function directly (i.e., solving for the components of R and t) can’t be done in closed form. So, we’re going to make the problem linear by assuming that the angles we need to solve for are small, so we can use the usual first-order approximations to sine and cosine. This lets us rewrite the energy function in a way that’s linear in the unknowns, and solve it using standard linear least squares. In practice, making the assumption that angles are small is not too bad: the angles will be small in later iterations, while during early iterations the error of this approximation tends to be dominated by the fact that the correspondences aren’t really correct… In short, it works really well in practice. S. Rusinkiewicz

Point-to-Plane Error Metric Overconstrained linear system Solve using least squares So, how do we actually implement this? The first thing is to write down the error function: we’re taking the transformed version of point pi, finding the vector to qi, then taking just the component of that vector along the normal ni. This is a little nasty, since the matrix R involves some trig functions, etc., and minimizing this error function directly (i.e., solving for the components of R and t) can’t be done in closed form. So, we’re going to make the problem linear by assuming that the angles we need to solve for are small, so we can use the usual first-order approximations to sine and cosine. This lets us rewrite the energy function in a way that’s linear in the unknowns, and solve it using standard linear least squares. In practice, making the assumption that angles are small is not too bad: the angles will be small in later iterations, while during early iterations the error of this approximation tends to be dominated by the fact that the correspondences aren’t really correct… In short, it works really well in practice. S. Rusinkiewicz

Closest Compatible Point Closest point often a bad approximation to corresponding point Can improve matching effectiveness by restricting match to compatible points Compatibility of colors [Godin et al. 94] Compatibility of normals [Pulli 99] Other possibilities: curvatures, higher-order derivatives, and other local features The idea is that instead of just blindly choosing the closest point as your correspondence, you instead try to find better ones. One way of phrasing this is as selecting the closest point that is compatible with the source point, by some metric(s). This can be done based on color, normals, or any of the features covered earlier. Note that there’s a different way of incorporating more information into the matching process, which is to directly do the matching in a higher-dimensional space, where 3 of the dimensions are just Euclidean coordinates, while the remaining dimensions correspond to your feature space. This certainly can be made to work, but you sometimes encounter difficulties relating to how the different dimensions are scaled. That is, there’s no natural way of saying e.g. how a distance of 3mm in space should be weighted relative to a distance of 10 in a color coordinate. In practice, you need to use something like a Mahalanobis distance (i.e., give the dimensions a weight computed using variances (and covariances) of the data itself). S. Rusinkiewicz

Global Registration Goal Given: n scans around an object Goal: align them all First attempt: ICP each scan to one other Now let’s turn to global registration. Of course, the problem we’re trying to solve is aligning more than just 2 scans. As hinted earlier, just aligning scans pairwise can lead to disappointing results. S. Rusinkiewicz

Global Registration Goal Want method for distributing accumulated error among all scans We really want to optimize the pairwise distances of all pairs of overlapping scans. S. Rusinkiewicz

Correspondence Space Search the whole model for correspondences Range query for descriptor values Cluster and pick representatives Q P

Evaluating Correspondences Coordinate root mean squared distance Requires best aligning transform Looks at correspondences individually

Rigidity Constraint Pair-wise distances between features and correspondences should be the same Q P

Rigidity Constraint Pair-wise distances between features and correspondences should be the same Q P

Rigidity Constraint Pair-wise distances between features and correspondences should be the same Q P

Alignment Results Input: 2 scans Our alignment Refined by ICP

Alignment Results Our alignment Refined by ICP Input: 10 scans

Non-rigid Alignment Algorithm Input Algorithm Goal [Allen et al. 2003] Algorithm Input Set of human range images All in similar poses Set of colored feature markers Algorithm Goal Develop correspondence from template to target Compute affine transform for each vertex Minimize error function T. Funkhouser

Optimization Variables Algorithm viewed as optimization problem Given an initial template surface with vertices vi Corresponding affine transformation matrices Ti Current state is Ti vi for all i (see diagram) Find values of Ti to minimize objective function Attempts to find a “good fit” (blue) of template (cyan) to target (red) T. Funkhouser

Objective Function Objective Function has three weighted terms Data error Smoothness error Marker error Will use different weights in each phase of process Multistep / Multi-resolution fitting process T. Funkhouser

Objective Function – Data Error Data Error term requires current match to be close to target Uses distance from each transformed vertex to the target surface Weighted by confidence measure (from scanning) Hole regions have weight = 0 Sums total error Distance function Uses transformed template vertex Takes minimum distance to “compatible” vertices in target Compatible defined as those with normal w/in 90 deg. T. Funkhouser

Objective Function – Smoothness Error Measures smoothness of deformation applied to template Problem is under-constrained using data error E_s measures change in T_I between adjacent vertices Encourages similarly-shaped features to be mapped to each other Uses Frobenius norm (vector L2 norm) T. Funkhouser

Objective Function – Marker Error Data and Smoothness Error can hit local minima Example: left arm transformed to right arm Solution: Use pre-labeled markers on the test subjects Viewed as white dots in the range image Correspondences set up beforehand (as in Consistent Mesh Parameterization) 74 markers per subject (not all are used, however) Measure distance from template marker to target marker K_I are the indices of the markers in template, m_I are target markers T. Funkhouser

Algorithm Procedure Minimize error function using L-BFGS-B algorithm Quasi-Newton method with limited memory usage T. Funkhouser

Motivation – Correct Hole Filling Also, in some places, such as between the legs, holes should be added T. Funkhouser

More Information http://www.cs.princeton.edu/~bjbrown/iccv05_course/ http://www.cs.princeton.edu/courses/archive/fall03/cs597D/index.html http://www.cs.princeton.edu/courses/archive/fall03/cs597D/resources.html#Registration http://graphics.stanford.edu/papers/fasticp/ http://www.geometrie.tuwien.ac.at/ig/ (Surface Matching and Registration)