1. MITRE Corporation Pose Correction for Automatic Facial Recognition Team: Elliot Godzich, Dylan Marriner, Emily Myers-Stanhope, Emma Taborsky (PM), Heather Williams Liaisons: Josh Klontz ’10 and Mark Burge Advisor: Zachary Dodds

2. Fraud detection Aid distribution Law enforcement National security Algorithmic identification of faces from images Commercial systems exist; MITRE is building a U.S. system for flexibility and security Unobtrusive relative to other biometric techniques, but with similar applications: Automated Facial Recognition

3. Privacy Concerns

4. Off-pose images are a significant challenge for automated facial recognition Many current algorithms, including MITRE's, do not include pose correction Pose Correction

5. Our approach to pose-correction involves finding and matching facial features in different images Feature-finding and shape transformation, are also useful for other image-processing tasks

6. research: use and extend existing approaches implement: within MITRE's existing codebase test: using MITRE's test scaffolding and databases Problem Statement Our goal is to research, implement, and test a pose correction library that improves MITRE's existing facial recognition system.

7. Average of Synthetic Exact Filters Active Shape Model Pose-correction pipeline PixelsFeaturesShape ASEFASM

8. Facial features, or landmarks, can support both recognition and pose-correction Features are based on spatial geometry and/or appearance Features

9. ASEF filter creation training image (with known right- eye location) human-designed synthetic output For each training image we create a synthetic output with the correct position of the feature, e.g., the right eye.

10. ASEF filter creation training image (with known right- eye location) human-designed synthetic output filter transforming the image at left into the image at right We want to create a filter that exactly transforms a training image into the desired synthetic output * =

11. ASEF filter creation In the Fourier domain, we want where Synthetic, Image, and Filter are the 2D Fourier transforms of the synthetic output, image, and filter. Complex division thus provides the filter:

12. ASEF filter creation We take the average of all of the synthetic exact filters to define, here, a final right-eye filter We average 517 filters like this…

13. ASEF filter creation We take the average of all of the synthetic exact filters to define, here, a final right-eye filter We average 517 filters like this… …to obtain the final filter?

14. ASEF filter creation We take the average of all of the synthetic exact filters to define, here, a final right-eye filter We average 517 filters like this… …to obtain the final filter.

15. ASEF filter application The filter’s strongest response is most right-eye-ey location in the image Unfiltered imageFiltered image We apply the filter in the Fourier domain; the peak in the spatial domain is a first estimate of the feature location

16. Final output

17. Gallery

18. ErrorImages within that error <.0126.3 % <.0263.9 % <.0586.1 % <.187.7 % ASEF results Many images' eyes are found quite accurately, but there are also some dramatic outliers: Units are fraction of interocular distance Percentage of pictures

19. Influence of ASEF’s Gaussian  Radius,  = 2pxRadius,  = 25pxRadius,  = 15px synthetic outputs ASEF filters

20. Radius,  = 20px ASEF tradeoffs Testing changes in Gaussian radii (  ) the opposite tradeoff more accurate localization – and more outliers left eye error (units of interocular distance) Radius,  = 5px left eye error (units of interocular distance)

21. ASEF improvements Using spatial heuristics as weights Unweighted filtered image Spatially weighted filtered image 1.0  original 0.5  original original

22. Without weighting With weighting ASEF improvements Using spatial heuristics as weights right eye error (units of interocular distance) right eye error (units of interocular distance) left eye error these clusters show mis-identifying the left or right eye

23. Average of Synthetic Exact Filters Active Shape Model Pose-correction pipeline PixelsFeaturesShape ASEFASM

24. Active Shape Models (ASM) Describe classes of objects with varying shapes geometric arrangement of facial features: eyes, nose, … Each shape is a set of points ASM trains on a training set of shapes, creating a statistical model of the variation within that shape-family.

25. ASM, step 1: Procrustes fitting Procrustes analysis determines a scaling, rotation, and translation that best align a family of shapes. training data (hundreds of faces) mean face (not necessarily angry) We use this approach to align all of the training faces and extract the mean face.

26. ASM, step 2: Estimating face space We use the most descriptive eigenvectors to describe the allowable shape domain. ASM uses principal components analysis to build a model of representative transformations of a face

27.  = 0 (mean face) -3  +3  Independent face-shape axes ASM, step 3: Transforming faces We can apply realistic transformations to the mean face along face space’s eigenvectors.

28. Second semester plans 1) Multi-resolution and weighted ASEF feature finding 2) Adding pixel appearance to the ASM shape models 3) Implementing pose-correction techniques (for pixels) shape space: yaw First approach: apply ASM's transformations to generate poses at desired values of pitch and yaw.

29. Project WorkClinic DeliverablesDue Date JanuaryWinter break Spring Semester Begins: 1/17 Phase III Presentation1/17/2012 Implement AAM, continue improving ASEF, research and select pose correction methods Final Report & Poster FebruaryBegin implementing selected pose correction methods, combine ASEF and ASM March Spring Break: 3/9-18 Spring Break Continue work on pose correction April Final ReportDraft of Poster Design4/2/2012 Revise FR, Final PresDraft 1 of Final Report4/10/2012 Final TouchesFinal Report Review4/12/2012 Feature Freeze4/13/2012 Draft of Final Report4/18/2012 Draft of Final Presentation 4/23/2012 May Finals: 5/3-4 Projects Day5/1/2012 Final Report5/4/2012 Spring Schedule MITRE clinic, spring 2012 schedule

30. Questions? Average of Synthetic Exact Filters Active Shape Model PixelsFeaturesShape ASEFASM

32. Gallery

34. Second semester plans The spring term will focus on researching and implementing landmark-based pose correction techniques. First approach: apply transformations given by ASM to generate poses at varying degrees of pitch and yaw. yaw pitch

35. ErrorWithout log transform <.0126.3 % <.0263.9 % <.0586.1 % <.187.7 % ASEF results Comparing image pre-processing techniques ErrorWith log transform <.0125.4 % <.0261.3 % <.0583.7 % <.185.6 % Fraction of interocular distance Percentage of pictures

36. AAM adds color or grayscale information to ASM’s model. AAM can generate photorealistic faces, not just geometrically realistic ones. Active Appearance Models (AAM) Shown here are faces generated by varying the central face’s appearance parameters by ±3  along two appearance axes. from T.F. Cootes, G.J. Edwards, and C.J. Taylor, Active Appearance Models

37. old pipeline new pipeline Face-recognition pipeline Face detection Recognition Landmarking Pose correction Input image Output ID Fall term’s focus Spring term’s focus

38. Next Steps Improving ASEF: We will experiment with image processing techniques and weighting based on expected pose and image complexity Extending ASM: We will implement Active Appearance Models to extend face pose-generation to face image-generation. Implementing Pose Correction: ASEF and ASM provide a baseline approach: namely, transforming a query image to a standard face pose Pixels Features Shape

39. Automated Facial Recognition Use of computers to identify faces from images Commercial systems exist, but MITRE is developing a system specifically for the US for flexibility and security Unobtrusive relative to other biometric techniques, but with similar applications:

40. Motivation: Uses for Biometrics Law enforcement and national security Fraud detection Aid distribution Social networking

41. ErrorPercent of Identifications <.010.263610315186246 <.020.638968481375358 <.050.861031518624642 <.10.876790830945559 ErrorPercent of Identifications <.010.253581661891117 <.020.613180515759312 <.050.836676217765043 <.10.859598853868195 Without cosine window With cosine window ASEF improvements Mapping

42. Last semester This semester Face-recognition pipeline

43. old pipeline new pipeline Face-recognition pipeline Face detection Recognition Landmarking Pose correction Input image Output ID

44. Training DataAverage Face ASM, step 2: Mean-face finding We use this approach to align all of the training faces and thus find the mean face.

45. We got this… ?

46. Centered! ASEF’s right-eye filter in the spatial domain

47. Face-recognition pipeline

49. Pixels Landmarks Shape Model Average of Synthetic Exact Filters (ASEF) Active Appearance Model (AAM) Landmarking algorithms

50. ASEF filter creation For each training image we create a synthetic output with the correct position of the feature, e.g., the right eye. training image (with known right-eye location) human-designed synthetic output

51. Our approach to pose-correction involves finding and matching facial features in different images Pose Correction With dots.

52. Average of Synthetic Exact Filters (ASEF) Active Shape Model (ASM) Landmarking algorithms Pixels Features Shape

53. old pipeline Face-recognition pipeline Face detection Recognition Input image Output ID Maybe we don't use this slide at all?

54. old pipeline new pipeline Face-recognition pipeline Face detection Recognition Landmarking Pose correction Input image Output ID Maybe we don't use this slide at all?

55. MITRE Corporation Pose Correction for Automatic Facial Recognition Team: Elliot Godzich, Dylan Marriner, Emily Myers-Stanhope, Emma Taborsky (PM), Heather Williams Liaisons: Josh Klontz ’10 and Mark Burge Advisor: Zachary Dodds No dots at all?