1. Understanding Faces Computational Photography
12/2/14
Computational Photography
Derek Hoiem, University of Illinois
Lecture by Amin Sadeghi
Some slides from Lana Lazebnik, Silvio Savarese, Fei-Fei Li

2. Face detection and recognition
“Sally”

3. Applications of Face Recognition
Album organization
Digital photography

4. Face Detection

5. How to find faces anywhere in an image?
Filter Image with a face?

6. Train a Filter
Normalize mean and standard deviation
SVM

7. Face detection: sliding windows…
Filter/Template
Multiple scales

8. What features? Intensity Patterns (with NNs)
(Rowely Baluja Kanade1996)
Exemplars
(Sung Poggio 1994)
Edge (Wavelet) Pyramids
(Schneiderman Kanade 1998)
Haar Filters
(Viola Jones 2000)

9. How to classify? Many ways Neural networks Adaboost SVMs
Nearest neighbor

10. What makes face detection hard?
Expression

11. What makes face detection hard?
Viewpoint

12. What makes face detection hard?
Occlusion

13. What makes face detection hard?
Distracting background

14. Consumer application: iPhoto 2009

15. Consumer application: iPhoto 2009
Things iPhoto thinks are faces

16. Face Recognition

17. Face recognition
1. Detect
2. Align
3. Represent
4. Classify

18. Simple technique Treat pixels as a vector
Recognize face by nearest neighbor

19. DeepFace
3D Alignment
Deep Learning

20. Morphing and Alignment

21. Figure-centric averages
Another usage is to show a significant trend.
----- Meeting Notes (10/4/11 17:12) -----
Aligned through translation and scale
average 21
Figure-centric averages
Need to Align
Position
Scale
Orientation
Antonio Torralba & Aude Oliva (2002)
Averages: Hundreds of images containing a person are averaged to reveal regularities
in the intensity patterns across all the images.

22. How do we average faces?

23. Morphing
image #1
image #2
warp
warp
morphing

24. Cross-Dissolve vs. Morphing
Average of
Appearance Vectors
----- Meeting Notes (10/4/11 17:12) -----
the demo
Average of
Shape Vectors
Images from James Hays

25. Aligning Faces Need to Align Position Scale Orientation Key-points
The more key-points the finer alignment
Images from Alyosha Efros

26. Average of two Faces Input face key-points
Pairwise Average key-point co-ordinates
Triangulate the faces
Warp: Transform every face triangle
Average The pixels

27. Average of multiple Face
1. Warp to mean shape
2. Average pixels

28. Appearance Vectors vs. Shape Vectors
Vector of
200*150*3
Dimensions
Requires Annotation
Provides alignment!
Complements
200*150 pixels (RGB)
You can think about faces in different ways. You can represent this way or that way. These are complements.
Shape
Vector
Vector of
43*2
Dimensions
43 coordinates (x,y)

29. Average Men of the world

30. Average Women of the world
You will notice that women from every place is very attractive because the …..

31. Subpopulation means Other Examples: Average female Average kid
Average Kids
Happy Males
Etc.
Average female
We might be curious what different populations of faces look like.
Average kid
Average happy male
Average male

32. Eigen-face

33. Eigenfaces example
Training images
x1,…,xN

34. PCA General dimensionality reduction technique
Preserves most of variance with a much more compact representation
Lower storage requirements (eigenvectors + a few numbers per face)
Faster matching
What are the problems for face recognition?

35. Principal Component Analysis
Given a point set , in an M-dim space, PCA finds a basis such that
coefficients of the point set in that basis are uncorrelated
The most variation is in the first basis vector, then second, …
The goal of PCA is to find orthogonal vectors that the first r vectors have the highest variance of all.
----- Meeting Notes (10/4/11 17:12) -----
Write it more mathematically
say what PCA is solving x1 x0
1st principal component
2nd principal component

36. PCA in MATLAB x=rand(3,10);%10 3D examples M=mean(x,2);
x2=x-repmat(M,[1 n]);
covariance=x2*x2';
[U E] = eig(covariance)
U =
E =

37. Principal Component Analysis
Continued…
first r < M basis vectors provide an approximate basis that minimizes the mean-squared-error (MSE) in the approximation (over all bases with dimension r)
Choosing subspace dimension r:
look at decay of the eigenvalues as a function of r
Larger r means lower expected error in the subspace data approximation r M 1
eigenvalues

38. Top eigenvectors: u1,…uk
Eigenfaces example
Top eigenvectors: u1,…uk

39. Face Space How to find a set of directions to cover all space?
We call these directions Basis
If number of basis faces is large enough to span the face subspace:
Any new face can be represented as a linear combination of basis vectors.

40. Limitations
Global appearance method: not robust to misalignment, background variation

41. Use Shape information 200*150 pixels (RGB) 43 coordinates (x,y)
Appearance
Vector
200*150 pixels (RGB)
You can think about faces in different ways. You can represent this way or that way. These are complements.
Shape
Vector
43 coordinates (x,y)

42. First 3 Shape Basis Mean appearance
----- Meeting Notes (10/4/11 17:12) -----
show both extremes

43. Manipulating faces
How can we make a face look more female/male, young/old, happy/sad, etc.?
Current face
Prototype 2
Prototype 1
----- Meeting Notes (10/4/11 17:24) -----
Add a matlab thing to show the differences. show how to use PCA in matlab.

44. Manipulating faces We can imagine various meaningful directions. Sad
Masculine
Current face
Feminine
Happy

45. Psychological Attributes

46. Human Perception

47. Prince Charles, Woody Allen, Bill Clinton, Saddam Hussein, Richard Nixon and Princess Diana

48. Jim Carrey (left) and Kevin Costner

49. Nixon and Winona Ryder

52. Which face is more attractive?
----- Meeting Notes (10/4/11 17:12) -----
The goal is to keep the same person but more attractice.
beautified
original

53. System Overview

54. Things to remember Face Detection Face Recognition Appearance Vector
Shape Vector
Face Transformation
Principal Component Analysis
A good way to represent faces is not just the intesities of images, we can also use shape and ….
To show the variation, we can use exemplars, or we can use various directions to be able to represent faces with a few vectors