Computer vision: models, learning and inference Chapter 18 Models for style and identity

Identity and Style 22Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Identity differs, but images similar Identity same, but images quite different

Structure 33Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Factor analysis review Subspace identity model Linear discriminant analysis Non-linear models Asymmetric bilinear model Symmetric bilinear model Applications

Factor analysis review 44Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Generative equation: Probabilistic form: Marginal density:

Factor analysis 55Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Factor analysis review 66Computer vision: models, learning and inference. ©2011 Simon J.D. Prince E-Step: M-Step:

Disadvantages This description of the data does not account for identity For images which have the same style (e.g., pose, lighting), we expect faces which have the same identity to lie in a similar part of the space 7

Factor analysis vs. Identity model 88Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Each color is a different identity multiple images lie in similar part of subspace

Subspace identity model 99 Generative equation: Probabilistic form: Marginal density: J-th training example of i-th individual same linear combination different noise term between-individual variation within-individual variation

Subspace identity model 10 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Factor analysis vs. subspace identity 11 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Factor analysisSubspace identity model

Learning subspace identity model 12 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince E-Step: M-Step:

Subspace identity model 13 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Different identities

Subspace identity model 14 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince The within-individual noise comprises whatever cannot be explained by the identity

Face verification! 15 Inference by comparing models, 0 : different identities 1: same identity, Posterior probability:

Inference by comparing models 16 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Model 1 – Faces match (identity shared): Model 2 – Faces don’t match (identities differ): Both models have standard form of factor analyzer face verication

Inference by comparing models 17 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Compute likelihood (e.g. for model zero) where Compute posterior probability using Bayes rule

Face Recognition Tasks PROBE … GALLERY ? 1. CLOSED SET FACE IDENTIFICATION … GALLERY PROBE ? NO MATCH 2. OPEN SET FACE IDENTIFICATION PROBE ? NO MATCH 3. FACE VERIFICATION 4. FACE CLUSTERING ? 18Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Inference by comparing models 19 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Limitations of identity subspace model 20 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Structure 21 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Factor analysis review Subspace identity model Linear discriminant analysis Non-linear models Asymmetric bilinear model Symmetric bilinear model Applications

Probabilistic linear discriminant analysis 22 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Generative equation: Probabilistic form: the style of this face, which differs for each instance

Probabilistic linear discriminant analysis 23 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Learning 24 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince E-Step – write out all images of same person as system of equations – Has standard form of factor analyzer joint posterior probability distribution over all of the hidden variables M-Step – write equation for each individual data point – Has standard form of factor analyzer – Use standard EM equation

Probabilistic linear discriminant analyis 25 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Inference 26 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Model 1 – Faces match (identity shared): Model 2 – Faces don’t match (identities differ): Both models have standard form of factor analyzer Compute likelihood in standard way

Example results (XM2VTS database) 27 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Building more complex models is worth the time and effort

Structure 28 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Factor analysis review Subspace identity model Linear discriminant analysis Non-linear models Asymmetric bilinear model Symmetric bilinear model Applications

Non-linear models (mixture) 29 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Mixture model can describe non- linear manifold. Introduce variable c i which represents which cluster belongs to To be the same identity, must also belong to the same cluster

Non-linear models (kernel) 30 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Pass hidden variable through non-linear function f[ ]. Leads to kernelized algorithm !!!!!! it is no longer possible to marginalize over the hidden variables !!!!!! it is no longer possible exactly to compare model likelihoods directly in the inference Approximate

Structure 31 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Factor analysis review Subspace identity model Linear discriminant analysis Non-linear models Asymmetric bilinear model Symmetric bilinear model Applications

How to solve the situation when the style of the data may change considerably 32 any given frontal face has more in common visually with other non- matching frontal faces than it does with a matching probe face. Model Styles differently

Asymmetric bilinear model 33 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Introduce style variable s ij indicates conditions in which data was observed Example: lighting, pose, expression face recognition Asymmetric bilinear model Introduce style variable s indicates conditions in which data was observed Example: lighting, pose, expression face recognition

Asymmetric bilinear model 34 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Asymmetric bilinear model 35 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Generative equation: Probabilistic form: Marginal density:

Learning 36 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince E-Step: M-Step:

Asymmetric bilinear model 37 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Inference – inferring style 38 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Likelihood of style Prior over style Compute posterior over style using Bayes’ rule

Inference – inferring identity 39 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Likelihood of identity Prior over identity Compute posterior over identity using Bayes’ rule

Inference – comparing identities 40 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Model 1 – Faces match (identity shared): Model 2 – Faces dont match (identities differ): Both models have standard form of factor analyzer Compute likelihood in standard way, combine with prior in Bayes rule

Inference – Style translation 41 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Compute distribution over identity Generate in new style

Structure 42 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Factor analysis review Subspace identity model Linear discriminant analysis Non-linear models Asymmetric bilinear model Symmetric bilinear model Applications

Symmetric bilinear model 43 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Generative equation: Probabilistic form: Mean can also depend on style...

Symmetric bilinear model 44 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Multilinear models 45 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Extension of symmetric bilinear model to more than two factors e.g.,

Structure 46 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Factor analysis review Subspace identity model Linear discriminant analysis Non-linear models Asymmetric bilinear model Symmetric bilinear model Applications

Face recognition 47 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Synthesizing animation 48 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Discussion 49 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Generative models Explain data as combination of identity and style factors In identity recognition, we build models where identity was same or different Other forms of inference such as style translation also possible