Redundant Equations for Matrices The eigenvectors are going to give us a new representation of each original face, plus we will know how much redundancy there is in the system, so that the true independent dimensions are much fewer than 256x256. In the brute force approach, each database pic is represented by its 256x256 number of pixels, hence the matching had to go thru each of these 256x256 numbers. If we can represent each pic now by fewer numbers, say 50, we will have a speedier approach. We next look at how this lowering is done.

How to Represent a face by a few #’s Given a Database of M Named Photos (assume each pic is 256x256) 1) Convert each Picture into a vector, by taking the first row of pixels and making them the first 256 entries in the vector, then taking the next row of pixels and making them the next 256 entries in the vector, and so on. This makes the vector be 256x256, which is 65536. Each vector is called τ (later these are τᵉ for each face, e) 2) Next, collect all of the τ vectors into a rectangular matrix, A. So, Column 1 would be the first τ, Column 2 would be the second τ, and so on. The matrix A has the dimension 65536xM. 3) Compute a new matrix C, from C ← A×Aᶧ where Aᶧ is the matrix obtained by turning every row of A into a column, (and every column would have to consequently become a row). Aᶧ is called the transpose of A. 4) Compute Eigenanalysis on C to get a set of {uᵢ}, and a set of {λᵢ}

How to Represent a face by a few #’s Compute Eigenanalysis on C to get a set of {uᵢ}, and a set of {λᵢ}. The set of {uᵢ} is the new eigenvectors, while the {λᵢ} are indicating how non-redundantly (strongly) each dimension (in the new set of eigenvectors) exists. Sort the {λᵢ} in descending order, and decide to keep the best β of them, typically β could be 50. Each λᵢ is tied to a uᵢ, so this means keeping the associated uᵢ; so these uᵢ’s now form our new Eigenface Set. Detour One for theory: The {uᵢ} are a basis set for the space. •This means that any member of the space, say, τ, is representable by the basis set, so τ = ᵢ₌₁ ∑⁶⁵⁵³⁶ (cᵢ uᵢ )

How to Represent a face by a few #’s τ = ᵢ₌₁ ∑⁶⁵⁵³⁶ (cᵢ uᵢ ) • Each τ as you know is a facepic (it came from a facepic). So, any uᵢ has the same form (same vector length, i.e., 65536) as the τ. So, that means that we can ask, “what does each uᵢ look like when viewed as a picture?” • Here are some samples of what the vectors look like:

How to Represent a face by a few #’s τ = ᵢ₌₁ ∑⁶⁵⁵³⁶ (cᵢ uᵢ ) Here are some samples of what the vectors look like: So, τ = ᵢ₌₁ ∑⁶⁵⁵³⁶ (cᵢ uᵢ ) = c₁u₁ + c₂u₂ + c₃u₃ + . . . + c₆₅₅₃₆u₆₅₅₃₆ . = c₁ + c₂ + c₃ + . . .

How to Represent a face by a few #’s So, saying that τ = c₁ + c₂ + c₃ + . . . Means that τ can be composed of some c₁ amount of the first eigenface PLUS some c₂ amount of the second eigenface PLUS some c₃ amount of . . . and so on. Hence, the question becomes “How well (perfectly) do we want to re-construct τ ?” If we want perfection, i.e., we want our moms to like our picture (τ, of our face), then we need to use all 65536 elements. That means we would need all 65536 of the cᵢ. If this is the case, then we have made no gains in our quest to keep our cost down -- this is because previously we needed 65536 numbers to represent us (previously they were actual pixels, while now they are these cᵢ numbers.) The bottom line is that there is no savings.

How to Represent a face by a few #’s If we are willing to give up perfection, for the gain of low cost, i.e., if we are willing to allow each of our re-composed faces to be lacking something, then we could use less (fewer) of the cᵢ. Let us consider this. We see that: τ = ᵢ₌₁ ∑⁶⁵⁵³⁶ (cᵢ uᵢ ) = ᵢ₌₁ ∑ᵝ (cᵢ uᵢ ) PLUS ᵢ₌ᵦ₊₁∑⁶⁵⁵³⁶ (cᵢ uᵢ ) i.e. β is cut-off If these uᵢ’s have been already ranked in order of importance (by having sorted the λᵢ’s earlier) , then this means that the lesser important uᵢ’s are in the second part of the summation.

How to Represent a face by a few #’s τ = ᵢ₌₁ ∑ᵝ (cᵢ uᵢ ) PLUS ᵢ₌ᵦ₊₁∑⁶⁵⁵³⁶ (cᵢ uᵢ ) If these uᵢ’s have been already ranked in order of importance (by having sorted the λᵢ’s earlier) , then this means that the lesser important uᵢ’s are in the second part of the summation. i.e., ignoring the second part of the summation, will degrade the re-composition of the τ, but we can choose the β to keep a sufficient amount of the elements, so that our re-composition is still good enough for distinguishing one face from any other in the database. In practice, it seems that for a database of about 1000 unique faces, a β value of about 50 seems to be sufficient to keep individuals separately recognizable.

How to Represent a face by a few #’s ● Hence, we have τ ≈ ᵢ₌₁ ∑ᵝ (cᵢ uᵢ ) where ≈ means “approximately equals” ● It is time to return to our steps. We were at step 5, where we have limited our β to be 50. ● This means that in this new world, each original face can now be represented by merely 50 numbers (and not the original 65536), so we will have a huge cost savings when we compute our match score. ● The Next Step, then, is, For each face e we need to get the 50 numbers that stand for that face. So, For each e, from 1 to M, For each v, from 1 to β (possibly 50) compute wᵉᵥ ← τᵉ . uᵥ

How to Represent a face by a few #’s 6) For each face e we must get the 50 numbers that stand for that face. For each e, from 1 to M, For each v, from 1 to β (possibly 50) compute wᵉᵥ ← τᵉ . uᵥ It is time for Detour Two for Theory: We can ask for the meaning of τᵉ . uᵥ We start by: τᵉ . uᵥ = ??? We know that for any τ, we have that τ = ᵢ₌₁ ∑⁶⁵⁵³⁶ (cᵢ uᵢ ) Hence, for our τᵉ, it must be that τᵉ = ᵢ₌₁ ∑⁶⁵⁵³⁶ (cᵉᵢ uᵢ )

How to Represent a face by a few #’s We start by: τᵉ . uᵥ = ??? We know that for any τ, we have that τ = ᵢ₌₁ ∑⁶⁵⁵³⁶ (cᵢ uᵢ ) Hence, for our τᵉ, it must be that τᵉ = ᵢ₌₁ ∑⁶⁵⁵³⁶ (cᵉᵢ uᵢ ) So, τᵉ . uᵥ = ( ᵢ₌₁ ∑⁶⁵⁵³⁶ (cᵉᵢ uᵢ )). uᵥ = (cᵉ₁u₁ + cᵉ₂u₂ + … + cᵉᵥuᵥ + … + cᵉ₆₅₅₃₆u₆₅₅₃₆). uᵥ

How to Represent a face by a few #’s τᵉ . uᵥ = ( ᵢ₌₁ ∑⁶⁵⁵³⁶ (cᵉᵢ uᵢ )). uᵥ = (cᵉ₁u₁ + cᵉ₂u₂ + … + cᵉᵥuᵥ + … + cᵉ₆₅₅₃₆u₆₅₅₃₆). uᵥ Theory gives us that the uᵢ are OrthoNormal. Ortho means each uᵢ is perpendicular to any other and Normal means the length is 1. Hence, we have that uᵢuᵣ = 0 when I ≠ r (Ortho) and uᵢuᵣ = 1 when I = r (Normal) Thus, τᵉ . uᵥ = (cᵉ₁u₁ + cᵉ₂u₂ + … + cᵉᵥuᵥ + … + cᵉ₆₅₅₃₆u₆₅₅₃₆). uᵥ = cᵉᵥ which is the v-th coefficient of the uᵢ when faceᵉ is expanded. Hence, we are getting the correct wᵉᵥ

How to Represent a face by a few #’s 7) Now Build the new table (database) | wᵉ₁ | wᵉ₂ | wᵉ₃ … | wᵉ₅₀ ----------|------|-------|-------------|------ Face1|w¹₁ | w¹₂ | w¹₃ … Face2|w²₁ | w²₂ | w²₃ … Face3| ⁞ ⁞ ⁞ ⁞ This concludes our steps to convert from the original face database to the new table of Weights (few #’s) for efficient face recognition.

Notes about getting the new few numbers ● 1) Each face e will have different numbers for the cᵢ for any particular uᵢ i.e., when you consider a particular eigenface uᵢ, you will see that each one of us will have different amounts of that eigenface in us. So, far example, if the eigenface seems to emphasize a beard, then those of us who have no beard will have a low (small) cᵢ while those of us with a similar beard (to the eigenface) will have a higher cᵢ .

Notes about getting the new few numbers ● 2) Where is the system squeezing out the redundancy from in our faces? Think about our cheek areas. Maybe the left cheek falls around vector entries 58, 59, 60, 320, 321, 322, etc. For my face, my cheek pixels in those areas will have similar values. So in those regions of the vector, I will have similar values. That same behavior will be true of each one of you, as well; i.e., those pixels 58, 59, 60, 320, 321, etc., will have similar values. Hence, we find that those Row Numbers have entries that are almost identical. Remember that going across a row in our Matrix A, refers to a specific region of the face, but for each one of us. So, if our cheek regions are homogenous, i.e., texture-less, then, our Matrix A will have several rows that are similar. The same principle will hold for other homogenous areas of the face, and that is why there is so much redundancy in Matrix A, which will be found.

How to Process a Test Face for Recognition Given a Database of M Named people (assume each pic now has 50 #’s) Convert the Test Picture into a vector, by taking the first row of pixels and making them the first 256 entries in the vector, then taking the next row of pixels and making them the next 256 entries in the vector, and so on. This makes the vector be 256x256, which is 65536. This vector is called τᵀᵉᶳᵗ 2) For each v, from 1 to β (β could possibly be 50) compute w ᵀᵉᶳᵗᵥ ← τ ᵀᵉᶳᵗ . uᵥ (so, compute 50 #’s) 3) For each face e in database, errorᵉ = squareRoot ( ᵥ₌₁∑⁵⁰ ( (w ᵀᵉᶳᵗᵥ --- wᵉᵥ )² ) ) 4) Find the e that produces the minimum error, or the e’s that have error less than some Threshold, or the five e’s with lowest error

Uses of Face Recognition A System that does Face Recognition (or Face Identification) could be used in at least two broad ways: To allow someone access to things (enter a secure building, your computer account, your ATM account, your home, etc.) In this case the Test picture is that of someone who is co-operating with the system because they wish the recognition to succeed. To pick out trouble-makers, so that they can be excluded from an activity: a known shoplifter walks into a store, a pedophile near an elementary school, terrorist at an airport, etc. In this case, the test picture is of someone who is not going to co-operate with the system, because indeed they are trying to foil it. ● Most working systems today do the first task above. The second is the world of fraudulent government contractors, who claim they can solve the second problem, without much success.

When do Face Recognition systems fail? When, between the Database picture and the new Test picture, there are changes in: 1) Face expressions 2) Face tattoos/facepaint/makeup 3) Head hair style 4) Facial hair 5) Face is sweaty 6) Facial surgery/scars 7) Aging 8) Weight gain/loss 9) Facial jewelry/hats 10) New pic may have motion blur 11) Light intensity 12) Light color 13) Light direction (causes different shadows) 14) Distance to camera 15--17) Head Roll/Pitch/Yaw 18) How the images are compressed, or resolution