Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1, 20091Chris Fallin - thesis defense

Outline Face recognition – Methods of evaluation Elastic Bunch Graph Matching – Gabor Jets – Bunch Graphs and Feature Localization My Contributions: Automatic Fiducial Points – Information Content model – Fiducial point placement – results

Face Recognition May 1, 20093Chris Fallin - thesis defense Subfield of biometrics: – life (bio) – measure (metric) – Extract identifying information from measures of human traits Face recognition: digital images of face 2D, 3D, infrared, multimodal, …

May 1, 2009Chris Fallin - thesis defense4 Image Set Face Recognition Evaluation

May 1, 2009Chris Fallin - thesis defense5 – used under terms of GNU FDL System Decision ROC curves YNYN Actual Y N

May 1, 2009Chris Fallin - thesis defense6 EER = 11.1%

Rank-one Score May 1, 2009Chris Fallin - thesis defense7 Gallery ABCDEFG A B ProbeC D E F G Rank-one:5/7 = 71.4%

Elastic Bunch Graph Matching (EBGM) Wiskott et al., USC/Bochum, mid-90s Basis of ZN-Face, successful commercial system We use Face Identity Evaluation System, from Colorado State Face features represented by Gabor filter responses Features are localized – Fit an elastic graph onto the features by localization: local optimization problems May 1, 20098Chris Fallin - thesis defense

A Face Graph May 1, 2009Chris Fallin - thesis defense9 [Wiskott99]

Gabor Jets Vector of filter responses to 40 Gabor kernels – 5 wavelengths – 8 orientations – Each is complex-valued Gabor jets capture information well: Gokberk et al. get 91% rank-one with fixed grid – On FERET: 78.5% max, with 12 grid points May 1, 2009Chris Fallin - thesis defense10

Bunch Graphs Each feature has a bunch of canonical jets Represents typical features Best-match at each feature point for novel images May 1, 2009 Chris Fallin - thesis defense 11 [Wiskott99]

Feature Localization Initial alignment: eye locations known a-priori Overlay bunch graph with average edge lengths Take Gabor jets; pick best match in each bunch Localize based on displacement estimation (local optimization problem) May 1, Chris Fallin - thesis defense

The Idea: Automatic Fiducial Point Placement Bunch graph training requires manual fiducial point placement – 70 images, 25 points Why not statistically determine optimal features to match on? We can align/normalize all faces and take some statistical measure at each point in face space to determine goodness Replaces training step; back-end algorithm is identical May 1, Chris Fallin - thesis defense

Related Work Gokberk et al.: choosing fiducial points with genetic algorithms – But their chosen points are global – Same goal as our system, excluding prelocalization Salient Points – Wavelet-based approach to image retrieval – Choras et al., 2006: similar approach with goodness function, but no EBGM May 1, Chris Fallin - thesis defense

Information Content: Variance Model Compute goodness function over face- space Inter-subject variance over intra-subject variance Self-normalizing unitless measure Requires multiple images per subject May 1, 2009Chris Fallin - thesis defense15

Computing the goodness function FRGC: 5404 images – 700 MB, 128x128 grayscale (7 GB before normalization) Each pixel: 12 seconds, on fast Athlon 64 Split into 128 Condor jobs – Each pixel is independent: easy Pre-normalize image set, dump to fast-loading binary format (single file) Run Condor jobs: three hours Post-processing to reassemble results May 1, 2009Chris Fallin - thesis defense16

Fiducial Point Placement Random placement with probability density Compute gradient of goodness function Probability is product of gradient and goodness Place points sequentially, decay probability around points exponentially Mirror-point constraint: mirror placements across centerline, or snap to center May 1, 2009Chris Fallin - thesis defense17

Prelocalization: Pseudo-Bunches Displacement estimation requires canonical feature jet from bunch We cant provide this if we have no knowledge of feature Solution: fake a jet bunch – Make educated guess with K-means clustering on jets from all images at given point Then, run displacement estimation to prelocalize points on each image May 1, Chris Fallin - thesis defense

Results Competitive with original, manual points – In both cases, automatic training points yield only ~1% performance drop – With no human training! Prelocalization did not work as intended Success without this suggested by Gokberks results May 1, 2009Chris Fallin - thesis defense19 FERET FRGC

ROC curves FERETFRGC May 1, 2009Chris Fallin - thesis defense20 (EER = 11.4%) (orig: 11.1%) (EER = 31.4%) (orig: 34.8%)

Prelocalization: causes for failure Poor pseudo-bunch clustering: K-means often found optimal clustering at self-imposed cap of N/10 clusters – Likely because initial jets are too far off Naïve localization: single-step – Bolme thesis compares several optimization algorithms Average displacement of pixels: larger than pixels in manual points May 1, Chris Fallin - thesis defense

Future Work More sophisticated prelocalization Look at pseudo-bunch statistics to determine failure mode in more detail Look at per-fiducial point statistics to determine where performance is weak Investigate: are manual pts a theoretical limit, or can we exceed them? Try new image classes – test claim of genericism May 1, Chris Fallin - thesis defense

Questions? Full thesis and source code will be posted online: May 1, Chris Fallin - thesis defense

Distance Metrics on Jets Phase-insensitive: magnitude only – Selects best jet in bunch Phase-sensitive – Can solve for displacement vector: basis of localization Displacement estimation May 1, 2009Chris Fallin - thesis defense24