1. Non-Rigid Registration of 3D Surface by Deformable 2D Triangular Meshes Speaker:James Chang

2. Introduction Non-rigid surface registration, particularly registration of human face, finds a wide variety of application in computer vision and graphics. This study present a new automatic surface registration method utilized both attraction forces originating from geometrical and textural similarities, and stresses due to non- linear elasticity of the surface.

3. Project flow chart 3D Reference Surface 3D Target Surface 2D Target Plane 2D Reference Plane Registration via Triangular Mesh Deformation Image Matching with Triangular Meshes Nonlinear Elastic Deformation Multiresolution Registration Algorithm Deformed Surface Mapping Surfaces onto Image Planes result

4. Mapping Surfaces onto Image Planes

5. Blanz and Vetter: A morphable model for the synthesis of 3-D faces. Proc. SIGGRAPH’99, Los Angeles, USA, 187–194, 1999. -Cylindrical projection: Project their vertices onto planes. -Large distortions in the regions where the angle between the surface normal and the projection direction is large. -Worse still, a bijective mapping is not guaranteed.

6. Bijection

7. Mapping Surfaces onto Image Planes Briefly the algorithms are categorized according to mapping distortion criterion that they minimize. Length-preserving, angle-preserving, are- preserving parameterization approaches. Least squares conformal mapping technique.(LSCM),angle-preserving. B. Levy, S. Petitjean, N. Ray and J. Maillot. Least squares conformal maps for automatic texture atlas generation. ACM Transactions on Graphics, 21(3):362– 371, 2002. Proceedings of SIGGRAPH 2002.

8. Mapping Surfaces onto Image Planes

9. Registration via Triangular Mesh Deformation in 2D Domain

10. Lets denote the reference surface as A which is to be mapped to a target surface B. Their planar parameterizations yield 2D domains If we can find a mapping from to,this leads to the mapping between the surfaces in 3D space.

11. Registration via Triangular Mesh Deformation in 2D Domain Image Matching with Triangular Meshes. Nonlinear Elastic Deformation. Multiresolution Registration Algorithm.

12. Image Matching with Triangular Meshes A mapping from to can be expressed via a vector field. Where denotes the 2D image coordinates and is a constant displacement A B

13. Image Matching with Triangular Meshes For each triangle of the reference mesh, we have a mapping function that maps a point in a triangle of the reference to the point in the target. This function interpolates the mapped coordinates of the triangle vertices by

14. Image Matching with Triangular Meshes Where barycentric coordinates are obtained by P1 P2P3 P is the area of triangle

15. Image Matching with Triangular Meshes An image matching energy,,which accounts for the total square matching error over domain becomes Where is the set of triangles,, is the domain of triangle, and represent multi-modal image values, and is a weighting matrix.

16. Image Matching with Triangular Meshes To exploit different surface attributes during matching, weighted sums of image attributes are used by means of diagonal weighting. The weights are determined according to both variance and importance of the channel.

17. Image Matching with Triangular Meshes The driving forces that deform a reference mesh to a target come from minimization of the image matching energy. This energy can be minimized by using the gradients at each mesh node with respect to its mapped coordinate s

18. Image Matching with Triangular Meshes The gradient at vertex is obtained through the chain rule as is the set of triangles connected to the node, is the.vertex of triangle that corresponds to node, and is thus the.barycenteric coordinate for the point.

19. Image Matching with Triangular Meshes The gradients are evaluated at each node of the mesh to update the mapping vector,, in a gradient decent scheme. The image gradients are evaluated using 3 X 3 Scharr mask. J. Weickert and H. Scharr. A scheme for coherenceenhancing diffusion filtering with optimized rotation invariance.Jornal of Vis. Comm. Im. Rep., 13:103–118, 2002. We approximate the integrals in (4) and (5) by sampling at the recursively subdivided triangle centers.

20. Nonlinear Elastic Deformation

21. St. Venant Kirchoff material model. M. Bro-Nielsen. Medical image registration and surgery simulation.IMM-DTU Ph.D. Dissertation, 1996. The potential energy for St. Venant Kirchoof material is given by Where and are the Lame material constants.

22. Nonlinear Elastic Deformation Green-Lagrange strain tensor

23. Nonlinear Elastic Deformation We use minimization of the Green-Lagrange strain tensor to attain elasticity. The deformation energy is defined as the Froebenius norm of this strain,

24. Nonlinear Elastic Deformation Analytic expressions for gradients of with respect to vertex coordinated mappings must be obtained for the energy minimization task. With triangular mesh elements nonlinearities are not too complex and can be solved directly with a gradient descent scheme. During this minimization, stresses at each node of the mesh deform its neighboring triangles to regularize the displacements due to forces generated by image mating errors.

25. Multiresolution Registration Algorithm.

26. The total energy minimized over the reference triangulation T is We minimize the total energy via gradient descent methods in a coarse-to-fine approach. In this equation controls the amount of elasticity and thus the regularization during the registration

27. Multiresolution Registration Algorithm. To solve the problem in different scales: First, Gaussian image pyramids are constructed one at the reference, the other at the target. Second, at each resolution, meshes are generated adapted to the scale of the pyramid level.

28. Multiresolution Registration Algorithm. Registration starts from the coarsest level, and the estimated deformations are subsequently transferred to a finer level. The deformation transfer from one mesh to another is realized by barycentric mapping. For each node of a finer level, its barycentric coordinates at the coarser mesh are calculated, and thus mapping of that node is obtained by a weighted sum of the previous mesh node value.

29. Multiresolution Registration Algorithm. To produce meshes: First, constrained Delaunay triangulation is performed. The boundary vertices, are used as constraints in this triangulation. Feature points can also be used as additional constraints. J. D. Boissonnat, O. Devillers, S. Pion, M. Teillaud, and M. Yvinec. Triangulations in CGAL. Comput. Geom. Theory Appl., 22:5–19, 2002.

30. Multiresolution Registration Algorithm. Second, a mesh generation algorithm which limits the maximum allowed triangle edge length is applied to the triangulation. J. R. Shewchuk. Mesh generation for domains with small angles. Proc. 16th Annu., AACM Sympos. Comput. Geom.,1-10, 2000.

31. Multiresolution Registration Algorithm. Energy minimization restarts in one higher resolution level initialized with the values of the previous step. This procedure continues until the finest and final deformation estimates are found.

32. Multiresolution Registration Algorithm. Referencetarget Coarsest level Finest level Mesh deformation Barycentric mapping

33. Multiresolution Registration Algorithm.

34. Result

35. Identities

36. Expressions

38. Result This algorithm is quite efficient. The number of iterations till convergence were in between 50 and 200, and total registration duration changed from 3 to 7 seconds using 3.0GHz CoreDuo processor.

39. Conclusion A new automatic algorithm to solve the non- rigid registration problem of two surfaces is presented. Registration is realized by deforming one surface onto the other one after mapping the onto a plane.

40. Conclusion This is achieved by first using a bijective mapping, and then utilizing finite element method to obtain the matching deformation. The differences of geometric and texture related surface attributes generates deformation forces, while elastic stresses regularize the resulting deformations.

41. Conclusion Nunlinear elasticity is used to cope with the large deformations by minimizing Green- Lagrange strain tensor. Also, following a multiresolution approach, good results can be obtained in a short time.

42. Future work Currently the algorithm is not designed to handle expression with open mouth. This team also plan to experiment using different surface attributes and investigate better strategies to combine these modalities.