Expression-invariant Face Recognition using Geodesic Distance Isometries Kerry Widder A Review of ‘Robust expression-invariant face recognition from partially missing data’ and other works by A. Bronstein, M. Bronstein and R. Kimmel

© 2006 by Kerry R. Widder 2 Outline Problem statement Earlier work (Canonical Form) –Model –Novel idea –Implementation –Issues Recent work (Partial Embedding) –Motivation –New idea –Implementation –Issues –Results –Future work

© 2006 by Kerry R. Widder 3 Problem Statement Goal: Expression-invariant face recognition using range data images.

© 2006 by Kerry R. Widder 4 Earlier Work – “Canonical Forms” (CF) Model the face as a Riemannian manifold, S Use minimal Geodesic distances between points on the face as a representation of the manifold –Intuition: skin on a face moves with expression, but the geodesic distances between points on the face remain almost constant, thus changes in expression are isometric transformations. –Geodesic distances between points show the intrinsic geometry of the face (i.e., the identity of the face) –Euclidean distances between points show the extrinsic geometry of the face (i.e., the expression of the face) –Experimental evidence: study done on one subject - data shows geodesic distances are isometric across expressions (is one subject sufficient to make the claim???)  Model

© 2006 by Kerry R. Widder 5 Earlier Work – “Canonical Forms” (CF) Need a distance function for comparison purposes – want d( S, f( S )) = 0 for all functions f which are isometries of S. Direct use of Geodesic distances has issues due to data being a sampled version of S : –Sample locations may be different between S & f( S ) –Number of samples may be different between S & f( S ) –Order of samples may be different between S & f( S ) Solution: Represent S as a subset of R m with the intrinsic geometry approximately preserved – ‘isometric embedding’ –The image produced is called a Canonical Form (CF)  Novel idea – Isometric Embedding

© 2006 by Kerry R. Widder 6 Earlier Work – “Canonical Forms” (CF) Compute the Geodesic distances using the Fast Marching Method (FMM) – requires O (N 2 ) or greater to compute. Error criteria: Raw Stress (won’t be exact, resulting in embedding error) –Equation: –Minimize using Multidimensional Scaling (MDS) - O (N 2 ) to compute Perform alignment of the CF image Matching – compare probe CF image with gallery CF images –Higher order moments  Implementation

© 2006 by Kerry R. Widder 7 3D recognition via geometric invariants Range camera acquires facial surface (I). The surface is smoothed (II), subsampled and cropped (III). Fast marching computes geodesic distances on the surface. Facial surface is flattened via MDS (IV). Rigid surface matching using the canonical surfaces (V). A. Bronstein, M. Bronstein and R. Kimmel, “3D face recognition using geometric invariants“, 2003 III III IV V

© 2006 by Kerry R. Widder 8 Earlier Work – “Canonical Forms” (CF) Strength – can pre-compute the signatures Limitations –Sensitivity to the definition of the boundaries. –No partial matching (including occlusions). –Computational complexity O (N 2 ) –requires ≥ 2500 samples for face recognition  Issues

© 2006 by Kerry R. Widder 9 Recent Work – “Partial Embedding” (PE) Reduce the error/distortion due to the embedding of a face into the canonical form Expand the domain of faces that can be handled to include: –Partial faces. –Faces with occlusions.  Motivation

© 2006 by Kerry R. Widder 10 Recent Work – “Partial Embedding” (PE) Embed the probe face directly into the target face from the gallery –If the two faces are isometric, the error will be zero. –If the two faces are not isometric, the error will give a measure of their similarity. Definition: Partial Embedding (PE) –A mapping, φ, of probe face manifold Q onto gallery target face manifold S.  New Idea

© 2006 by Kerry R. Widder 11 Recent Work – “Partial Embedding” (PE) Computation of the PE uses a new procedure called ‘Generalized Multidimensional Scaling (GMDS) Error criteria – generalized stress: – –Where u i, i=1,…,M denote the vectors of parametric coordinates of s i, and W = (w ij ) is a symmetric matrix of non-negative weights. –The weights are set to 0 or 1 – if the whole surface is being matched, all are set to 1; if only part of the surface is being matched, the appropriate weights are set to 0. –Minimization of the error is done iteratively Distance function – –Allows partial matching of faces –Issues with points on the boundary of an occlusion  Implementation

© 2006 by Kerry R. Widder 12 GMDS Bronstein Bronstein Kimmel 2006

© 2006 by Kerry R. Widder 13 Partial matching problem. Shown in blue dotted is a geodesic between the points q 1, q 2 Q; the corresponding inconsistent geodesic on Q’ is shown in black. Bronstein Bronstein Kimmel 2006 d Q (q 1,q 2 ) d Q’ (q 1,q 2 ) Q’ Q q2q2 q1q1

© 2006 by Kerry R. Widder 14 Recent Work – “Partial Embedding” (PE) Strengths –Reduced error/distortion. –Can handle partial data. –Not sensitive to preprocessing steps. –No alignment step. Limitations –Boundary issues for partial matching. –Can’t pre-compute distance function (can minimize this impact using a hierarchical search scheme).  Issues

© 2006 by Kerry R. Widder 15 Recent Work – “Partial Embedding” (PE) Database –FRGC data – used 30 subjects – 1 neutral expression (gallery), 5 moderate expression (probes). –Gallery images were sub-sampled at 2500 points –Probe images were sub-sampled at 53 points –Two sets of probe images created – 1. face cropped with narrow geodesic mask, 2. Severe occlusions Results –Mild occlusion – EER of 3.1% –Severe occlusion – EER of 5.5% –100% rank one recognition rate –Computation time: 1/5 sec. per comparison  Results

© 2006 by Kerry R. Widder 16 Bronstein Bronstein Kimmel 2006

© 2006 by Kerry R. Widder 17 Bronstein Bronstein Kimmel 2006

© 2006 by Kerry R. Widder 18 Future Work – “Partial Embedding” (PE) Larger data set More severe expressions Extend to texture

© 2006 by Kerry R. Widder 19 Selected References A. M. Bronstein, M. M. Bronstein, and R. Kimmel. Robust expression- invariant face recognition from partially missing data. Proc. 9 th European Conf. on Computer Vision, May A. M. Bronstein, M. M. Bronstein, and R. Kimmel. Three-dimensional face recognition. Intl. J. Computer Vision, 64(1):5-30, August A. Elad and R. Kimmel. On bending invariant signatures for surfaces. IEEE Trans. PAMI, 25(10): , A. M. Bronstein, M. M. Bronstein, and R. Kimmel. Generalized multidimensional scaling: a framework for isometry-invariant partial surface matching. Proc. Nat. Acad. Sci., 103(5): , January A. M. Bronstein, M. M. Bronstein, and R. Kimmel. Expression-invariant representations for human faces. Technical Report CIS , Dept. of Computer Science, Technion, Israel, June 2005.