1. Deformation Modeling for Robust 3D Face Matching Xioguang Lu and Anil K. Jain Dept. of Computer Science & Engineering Michigan State University

2. Problem Although 3D facial scans do not vary with lighting or pose changes, nonrigid facial deformations can hurt recognition Although 3D facial scans do not vary with lighting or pose changes, nonrigid facial deformations can hurt recognition Collecting and storing multiple expression template scans for each subject is not practical Collecting and storing multiple expression template scans for each subject is not practical Expressions can have differing intensities Expressions can have differing intensities

3. Proposed Scheme A (hierarchical) geodesic sampling is used to quantify facial expression A (hierarchical) geodesic sampling is used to quantify facial expression Expression variations are learned from a small control group Expression variations are learned from a small control group These variations are used to create a deformable model from gallery templates These variations are used to create a deformable model from gallery templates This deformable model is fit to the target scan and matching distance computed This deformable model is fit to the target scan and matching distance computed

5. Sampling Landmarks are manually selected (nose tip, eye corners, mouth corners, and mouth contour) Landmarks are manually selected (nose tip, eye corners, mouth corners, and mouth contour) Geodesic distance between certain features is computed (hierarchically in latest work) Geodesic distance between certain features is computed (hierarchically in latest work) Geodesics are split into L segments of equal length to generate L-1 new feature points Geodesics are split into L segments of equal length to generate L-1 new feature points

6. Deformation Transfer Register non-neutral scan with neutral scan of same face to estimate landmark displacement Register non-neutral scan with neutral scan of same face to estimate landmark displacement Establish a mapping Φ from the neutral gallery to the neutral target face Establish a mapping Φ from the neutral gallery to the neutral target face Use Φ to transfer landmarks in the non-neutral gallery scan to the (synthesized) non-neutral target Use Φ to transfer landmarks in the non-neutral gallery scan to the (synthesized) non-neutral target Establish a mapping ψ from the neutral to non-neutral target Establish a mapping ψ from the neutral to non-neutral target Interpolate ψ using thin-plate-spline mapping Interpolate ψ using thin-plate-spline mapping Boundary constraints are included in thin-plate-spline calculation as additional landmark points Boundary constraints are included in thin-plate-spline calculation as additional landmark points

7. Registration Neutral and non-neutral target are aligned using features which don ’ t move much with expression changes, such as eye corners and nose tip Neutral and non-neutral target are aligned using features which don ’ t move much with expression changes, such as eye corners and nose tip This separates rigid transformations from nonrigid transformations This separates rigid transformations from nonrigid transformations

8. Thin-Plate Splines Goal: find a mapping from landmark set U to V with known correspondences Goal: find a mapping from landmark set U to V with known correspondences Method: imagine V as a thin metal sheet and find a function which minimizes bending energy Method: imagine V as a thin metal sheet and find a function which minimizes bending energy Solution: F(u) = c + A*u + W T *s(u) Solution: F(u) = c + A*u + W T *s(u) s(u) = (|u – u 1 |, |u – u 2 |, …) T s(u) = (|u – u 1 |, |u – u 2 |, …) T An analytical solution can be obtained for 3D points An analytical solution can be obtained for 3D points

9. Deformable Model Construction To generate a deformable model, each learned expression is simulated on a neutral gallery face To generate a deformable model, each learned expression is simulated on a neutral gallery face Face is represented as a combination of shape vectors: Face is represented as a combination of shape vectors: M is the number of synthesized templates, α i is the weight of each template M is the number of synthesized templates, α i is the weight of each template By adjusting the weights α i, various combinations of expressions can be generated By adjusting the weights α i, various combinations of expressions can be generated To reduce computational complexity, one deformable model per expression is generated To reduce computational complexity, one deformable model per expression is generated

10. Matching Coarse alignment performed as during deformation transfer Coarse alignment performed as during deformation transfer Alignment refined with iterative closest point algorithm Alignment refined with iterative closest point algorithm Associate each point with nearest neighbor, calculate transform to minimize distance, repeat Associate each point with nearest neighbor, calculate transform to minimize distance, repeat Minimize a cost function by solving for α i s Minimize a cost function by solving for α i s R and T are rotation and translation matrices, S is the deformable model, and S t is the test scan R and T are rotation and translation matrices, S is the deformable model, and S t is the test scan Use these α i s to compute a new iterative closest point distance, and return to step 2 until convergence Use these α i s to compute a new iterative closest point distance, and return to step 2 until convergence

11. Experiment I Self-collected database of 10 subjects at 3 different poses, with 7 different expressions, for 210 total scans and 10 gallery models Self-collected database of 10 subjects at 3 different poses, with 7 different expressions, for 210 total scans and 10 gallery models 5 subjects at random chosen as control group, leaving 105 scans for recognition 5 subjects at random chosen as control group, leaving 105 scans for recognition Results: Results:

12. Experiment II Control group: 10 subjects from Experiment I Control group: 10 subjects from Experiment I Test group: 90 additional subjects, with 6 scans each at different viewpoints (in most cases) Test group: 90 additional subjects, with 6 scans each at different viewpoints (in most cases) 533 total test scans 533 total test scans Results: Results:

13. Experiment III A subset of FRGC v2.0 dataset A subset of FRGC v2.0 dataset Scans with the earliest timestamp and neutral expression are used as templates Scans with the earliest timestamp and neutral expression are used as templates 50 gallery scans, 150 test scans 50 gallery scans, 150 test scans 10 subjects in Experiment I used as control group 10 subjects in Experiment I used as control group Latest results (after publication): Latest results (after publication):

14. Conclusions One area for improvement (noted in the paper) was the dependence on manual landmark labeling One area for improvement (noted in the paper) was the dependence on manual landmark labeling Also, I thought that there might be some application of geometric invariants to replace their registration step (which is subject to local minima) Also, I thought that there might be some application of geometric invariants to replace their registration step (which is subject to local minima)

15. Questions?