1. Face Alignment Using Cascaded Boosted Regression Active Shape Models
Michael Dixon

2. Faces in computer vision
What problems do people work on?
Detection
Alignment
High-level analysis
Face recognition
Facial expression recognition
Face tracking
Viola Jones, 2001, 1600 citations

3. Face alignment
Given an image of a face and an initial guess, localize key facial features
Approaches
Active Shape Model, 1992
Boosted Regression ASM, 2007
Active Shape Models, 1992, 1800 citations

4. 1500 hand-labeled face images
Training data
Given many examples, learn a model
1500 hand-labeled face images

5. The Active Shape Model framework
Input image
Shape
Features

6. The Active Shape Model framework
Input image
Shape
Features

7. Shape model Given many examples of a shape
Learn a set of constraints on allowable shapes
There are 20 points in this face model, and each point has an x and y coordinate, so there are a total of 40 dimensions, with which you can represent any possible face. However, those 40 dimensions can represent any possible shape you want. We want a model that just focuses on faces. So given a lot of examples of known face shapes, we build a shape model that can represent all possible faces with many fewer parameters.

8. Principal variations from the mean
Learning a shape model
Represent as a linear subspace
Mean face shape
Principal variations from the mean

9. The Active Shape Model framework
Input image
Shape
Features

10. Feature model
Given a patch near a facial feature, predict the correct position of that feature
Given
Predict

11. Learning a feature model
Generate training examples with known feature positions
Train a regression model to predict the correct displacement
20 features, and each will need to predict an x and y displacement, so a total of 40 regression models

12. Boosted regression
Goal: Learn a function to predict a set of target values
Boosting builds a strong regression model from many weak models
Evaluate a large pool of possible weak regression functions
Select the function with the lowest error and add it to the strong regression model
Update the target values and repeat

13. Weak regression function
Weak regression model
Haar wavelet response
hm =
The sum of all pixel values under the white box minus the sum of all pixel values under the black box
Haar wavelet features
Weak regression function

14. Weak regression example
b = 0.012
t = 21.7
displacement hm
fit weak regression function to data
displacement hm

15. Strong regression model
Predicted displacement
Ground-truth displacement
25 weak regression functions combined into a strong regression function

16. The Active Shape Model framework
Combining the shape and feature models
Shape
Features
Alignment

17. Fitting using Boosted Regression ASM
Initialize the feature positions
Iteratively
Predict feature positions using regression model
Constrain to fit the shape model
Update feature positions

18. Limitations of the previous work
How often does the boosted regression feature model improve on the initial estimate?
Some improvement
Significant
improvement
Displacement (in pixels)
Percent that improved
Improved by at least 50%
Any improvement
So, we’re going to be given an initial guess of where a particular feature is, and we’re going to use our feature model to estimate how far away from the correct location that guess is, and based on this we’re going to update our position.
If our estimation is good, we’ll get closer to the correct position. But if the estimate isn’t good, we might end up farther away than we started.
In these plots, we took 10,000 examples of a particular feature with random displacements, estimated the displacement using our feature model, and looked at what percentage of the time that estimate was actually better than the initial guess.
Predicted position vs. actual position

19. Accuracy trade-off
Regression model can’t accurately predict both large and small displacements
Model trained on large displacements
Model trained on small displacements
Displacement (in pixels)
Some improvement
Significant
improvement
Percent that improved
Displacement (in pixels)
Some
improvement
Significant
Percent that improved
There’s a trade-off between being able to accurately predict large displacements and being able to accurately predict small displacements.
Here we show two different feature models. One is trained on a

20. Displacement (in pixels)
Proposed solution
Train multiple models (coarse to fine) and apply them in sequence
Coarse regression model
Fine regression model
Percent that improved
Displacement (in pixels)

21. Cascaded Boosted Regression ASM
Face
Detector
Boosted Regression ASM
15 iterations
Alignment
Image
Cascaded Boosted Regression ASM
Face
Detector
Stage 1
5 iterations
Stage 2
5 iterations
Stage 3
5 iterations
Alignment
Image

22. Learning an alignment cascade
Train a new stage of the cascade using the output of the previous stage
Use a face detector as the initial stage
For each stage
Measure error distribution of each feature
Generate training examples from the error distribution
Train new feature models
Align all images using the updated model to get a new error distribution

23. Qualitative comparison
Boosted Regression ASM
Talk through some specific examples. Give the audience time to take this in.
Cascaded Boosted Regression ASM

24. Quantitative evaluation
Error metric:
where:
di is the distance between the estimated position and the ground truth position of the ith point
s is the inter-ocular distance
An alignment is only as good as its worst point
Inter-ocular distance, s
Alignment vs.
Ground-truth

25. Cumulative error distribution
Results
Evaluated on 500 unseen test images
73%
19% 3%
Cumulative error distribution
Cascaded
Standard
Average face
Alignment error

26. Results Alignment accuracy after each stage Stage 1 Stage 2 Stage 3
Cumulative error distribution
Alignment error
Stage
Median alignment error

27. Conclusions
Boosted Regression ASMs are a newly proposed method for performing face alignment
Training a cascade of Boosted Regression ASMs can significantly improve alignment accuracy

28. Questions?