1. Robust 3D Shape Correspondence in the Spectral Domain
Varun Jain and Hao (Richard) Zhang
Graphics, Usability, and Visualization (GrUVi) Lab
School of Computing Science
Simon Fraser University
Burnaby, BC Canada
June 15, 2006

2. The correspondence problem
Given two shapes represented by triangle meshes, find a meaningful correspondence between their vertices
Not a (continuous) parameterization problem, e.g., [Kraevoy & Sheffer 04] ― min. distortion, mapped features
Applications: mesh parameterization, morphing, shape registration, tracking, recognition, and retrieval, etc.

3. Background A classical problem studied in computer vision mostly
We are interested in fully automatic and purely shape-based approaches, i.e., without using prior knowledge
Goals:
Invariance to common rigid and non-rigid transformation
Robustness against noise, different object size, etc.
Ultimately, return meaningful correspondences
Despite intense studies, all proposed methods can fail on seemingly easy cases for humans

4. Two basic types of techniques
Extrinsic methods
Point coordinates defined in some global frame
Optimization-based and mostly iterative, e.g., iterative closest point (ICP)
Initial alignment is crucial
Intrinsic methods
Point coordinates based on relative information
A descriptor defined from the perspective of that point
Descriptors can be absolute coordinates, e.g., spectral, or statistical, e.g., shape contexts [Belongie et al. 02]
Non-rigid ICP [Chui et al. 2004]
rotation

5. Related works With the aid of initial manual feature correspondence
Cross parameterization [Praun et al. 01, Kraevoy & Sheffer 04]
Feature-guided ICP [Sumner & Popovic 04]
Barycentric interpolation between features [Zayer et al. 05]
Other deformation based approaches
Automatic extrinsic methods: ICP and its variants
Original ICP [Besl & McKay 92]
Many variants of rigid ICP [Rusinkiewicz & Levoy 01]
Robust ICP based on refinement [Zinber et al. 03]
Non-rigid ICP based on thin-plate splines [Chui et al. 03]

6. Related works Local shape descriptors Spectral methods
Shape context [Belongie et al. 02, Körtgen et al. 03]
Spin images [Johnson & Hebert 99]
Other approaches that handle rigid transformations [Gelfand et al. 05, Li & Guskov 05]
Curvature map [Gatzke et al. SMI 05]
Spectral methods
Original work on correspondence [Shapiro & Brady 92]
MDS for retrieval of isometric shapes [Elad & Kimmel 03]
Others: compression [Karni & Gotsman 00], spherical parameterization [Gotsman et al. 03], mesh sequencing [Isenberg & Lindstrom 05, Liu et al. 06], segmentation [Liu & Zhang 04], surface reconstruction [Kolluri et al. 04]

7. Spectral correspondence
[Shapiro & Brady, 92]: Given two point sets P and Q
Construct symmetric “affinity” matrices AP and AQ using pair-wise L2 distances and a Gaussian kernel
Construct spectral embedding by k leading eigenvectors of AP and AQ, sorted by descending eigenvalues
Compute best matching using embedded coordinates via L2 distance in the k-dimensional spectral domain
Why spectral correspondence?
Affinities are intrinsic measure (but high-dimensional)
Eigenvectors have good approximation properties
Spectral embeddings normalize shapes with respect to all rigid body transformations and uniform scaling

8. Key observations Flexibility of affinity measures
Whichever transformation one needs the correspondence to be invariant of, build that invariance into affinities
Eigenvectors need to be scaled properly, e.g., at least to handle data with difference sizes
Eigenvectors can “switch” (never reported before)
Signs of eigenvectors need to be consistent
Non-rigid shape transformation can cause non-rigid transformation in the spectral domain

9. Summary of our approach
Use geodesic affinities for invariance to shape bending
Eigenvector scaling using squared root of eigenvalues
Proper handling of objects with difference sizes
Eigenvalue decay leaves approach less sensitive to k
Variance-normalization + interpretation from kernel PCA
Heuristics to resolve eigenvector switch and sign flip
Non-rigid ICP via thin-plate splines in spectral domain
Net result:
Proper correspondence of articulated shapes
Consistently more robust correspondence results

10. Evaluation paradigm Visual examination via color plots
Manually color one model based on parts
Color second model using computed correspondence
Plot of percentage of correct matches
Manually provided ground truth (small feature sets)
Ground truth automatically identified via “in-place” mesh decimation
Plot of correspondence error
Sum of correspondence errors at the vertices
Error at a vertex: geodesic distance between ground truth and computed corresponding point

11. Basic steps of our algorithm
Eigenvector scaling
Non-rigid ICP via thin-plate splines
Geodesic-based
spectral embedding
Best
matching

12. Geodesic affinities Given two meshes M1 and M2 of sizes n1 and n2
Affinity matrices A1 (n1n1) and A2 (n2n2) given by
where d1 and d2 are geodesic distances on M1 and M2
Gaussian: importance of far-away vertices attenuated
Gaussian width set as maximum geodesic distance
Other kernel, e.g., exponential, may be applied

13. Spectral embeddings Eigen-decompose each affinity matrix A = UUT
Obtain k-D spectral embedding of mesh vertices using the k leading (scaled) eigenvectors of A
First eigenvector ignored as it is almost a constant
k-dimensional spectral embedding coordinates of ith the point of mesh

14. Examples: 3D spectral embeddings
Use of 2nd, 3rd, and 4th scaled eigenvectors

15. Eigenvector scaling
EkEkT gives the best rank-k approximation of the affinity matrix A (namely, dot product  affinity)
Scaling using the square root of the eigenvalues is shown to normalize the variance of data
The scaling is also a natural one when seen from the perspective of kernel PCA [Jain 2006]

16. Eigenvector switching and sign flips
Signs of the eigenvectors are decided by eigensolver and are difficult to correspond automatically
Discrepancy between shapes can also cause certain eigenvectors to switch places
An eigenvector switch or a sign flip corresponds to a reflection in the spectral domain

17. Exhaustive search and greedy heuristic
Reflection-invariant shape descriptors possible, e.g., high-D shape context or symmetric polynomials, but more invariance  less descriptivity [Frome et al. 04]
Choose among 2kk! possible eigenvector ordering and sign flips to minimize a correspondence cost:
Besides exhaustive search (for very small k), can use greedy scheme to order one eigenvector at a time

18. Non-rigid transformations
Perform non-rigid ICP using thin-plate splines in the spectral domain
Experimentally, very fast convergence (5-10 iterations)

19. Recap of algorithm Eigenvector scaling Geodesic-based
Non-rigid ICP via thin-plate splines
Geodesic-based
spectral embedding
Best
matching

20. Result: % of correct correspondences
Manual initial alignment used for first three!
% is out of ground-truth matches
( vertices; k = 5 eigenvectors used throughout)

21. Visual results for correspondence
Models with articulation and moderate stretching
Many more results in color plate (page 300).

22. Limitations Intrinsic geodesic affinities
Symmetry issue
Topological issue: hybrid approach [Jain & Zhang 06]
Rather primitive heuristic for resolving eigenvector switching and sign flips
Effectiveness attributed to spectral normalization
Euclidean metric as correspondence cost
No particular reason, except for a computational one
Challenge: what is right?
Computational complexity: O(n2logn)

23. Follow-up and future works
Sampling via Nyström approximation [Liu et al. 06]
Spectral embedding: O(n2logn)  O(pnlogn + p3)
Little loss of quality at low sampling (10 out of 4000)
Farthest point sampling used
More sophisticated sampling schemes [Liu & Zhang 06]
Retrieval of articulated shapes [Jain & Zhang 06]
Outperforms light-field descriptor [Chen et al. 03] and spherical Harmonics descriptor [Kazhdan et al. 03] (even when these are applied to spectral embeddings)
But not so on Princeton Benchmark database (yet) due to various artifacts in the models
How about eigenspaces?

24. Acknowledgement Thank you! NSERC Grant 611370
MITACS Grant on project: Mathematical Surface Representations for Conceptual Design
MATLAB code of non-rigid ICP [Chui et al. 03]
Greg Mori for helpful discussions
Reviewers’ comments and for pointing out a couple of missing references
Thank you!