1. Computer vision: models, learning and inference Chapter 18 Models for style and identity

2. Identity and Style 22Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Identity differs, but images similar Identity same, but images quite different

3. Structure 33Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Factor analysis review Subspace identity model Linear discriminant analysis Non-linear models Asymmetric bilinear model Symmetric bilinear model Applications

4. Factor analysis review 44Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Generative equation: Probabilistic form: Marginal density:

5. Factor analysis 55Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

6. Factor analysis review 66Computer vision: models, learning and inference. ©2011 Simon J.D. Prince E-Step: M-Step:

7. Disadvantages This description of the data does not account for identity For images which have the same style (e.g., pose, lighting), we expect faces which have the same identity to lie in a similar part of the space 7

8. Factor analysis vs. Identity model 88Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Each color is a different identity multiple images lie in similar part of subspace

9. Subspace identity model 99 Generative equation: Probabilistic form: Marginal density: J-th training example of i-th individual same linear combination different noise term between-individual variation within-individual variation

10. Subspace identity model 10 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

11. Factor analysis vs. subspace identity 11 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Factor analysisSubspace identity model

12. Learning subspace identity model 12 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince E-Step: M-Step:

13. Subspace identity model 13 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Different identities

14. Subspace identity model 14 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince The within-individual noise comprises whatever cannot be explained by the identity

15. Face verification! 15 Inference by comparing models, 0 : different identities 1: same identity, Posterior probability:

16. Inference by comparing models 16 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Model 1 – Faces match (identity shared): Model 2 – Faces don’t match (identities differ): Both models have standard form of factor analyzer face verication

17. Inference by comparing models 17 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Compute likelihood (e.g. for model zero) where Compute posterior probability using Bayes rule

18. Face Recognition Tasks PROBE … GALLERY ? 1. CLOSED SET FACE IDENTIFICATION … GALLERY PROBE ? NO MATCH 2. OPEN SET FACE IDENTIFICATION PROBE ? NO MATCH 3. FACE VERIFICATION 4. FACE CLUSTERING ? 18Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

19. Inference by comparing models 19 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

20. Limitations of identity subspace model 20 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

21. Structure 21 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Factor analysis review Subspace identity model Linear discriminant analysis Non-linear models Asymmetric bilinear model Symmetric bilinear model Applications

22. Probabilistic linear discriminant analysis 22 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Generative equation: Probabilistic form: the style of this face, which differs for each instance

23. Probabilistic linear discriminant analysis 23 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

24. Learning 24 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince E-Step – write out all images of same person as system of equations – Has standard form of factor analyzer joint posterior probability distribution over all of the hidden variables M-Step – write equation for each individual data point – Has standard form of factor analyzer – Use standard EM equation

25. Probabilistic linear discriminant analyis 25 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

26. Inference 26 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Model 1 – Faces match (identity shared): Model 2 – Faces don’t match (identities differ): Both models have standard form of factor analyzer Compute likelihood in standard way

27. Example results (XM2VTS database) 27 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Building more complex models is worth the time and effort

28. Structure 28 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Factor analysis review Subspace identity model Linear discriminant analysis Non-linear models Asymmetric bilinear model Symmetric bilinear model Applications

29. Non-linear models (mixture) 29 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Mixture model can describe non- linear manifold. Introduce variable c i which represents which cluster belongs to To be the same identity, must also belong to the same cluster

30. Non-linear models (kernel) 30 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Pass hidden variable through non-linear function f[ ]. Leads to kernelized algorithm !!!!!! it is no longer possible to marginalize over the hidden variables !!!!!! it is no longer possible exactly to compare model likelihoods directly in the inference Approximate

31. Structure 31 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Factor analysis review Subspace identity model Linear discriminant analysis Non-linear models Asymmetric bilinear model Symmetric bilinear model Applications

32. How to solve the situation when the style of the data may change considerably 32 any given frontal face has more in common visually with other non- matching frontal faces than it does with a matching probe face. Model Styles differently

33. Asymmetric bilinear model 33 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Introduce style variable s ij indicates conditions in which data was observed Example: lighting, pose, expression face recognition Asymmetric bilinear model Introduce style variable s indicates conditions in which data was observed Example: lighting, pose, expression face recognition

34. Asymmetric bilinear model 34 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

35. Asymmetric bilinear model 35 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Generative equation: Probabilistic form: Marginal density:

36. Learning 36 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince E-Step: M-Step:

37. Asymmetric bilinear model 37 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

38. Inference – inferring style 38 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Likelihood of style Prior over style Compute posterior over style using Bayes’ rule

39. Inference – inferring identity 39 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Likelihood of identity Prior over identity Compute posterior over identity using Bayes’ rule

40. Inference – comparing identities 40 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Model 1 – Faces match (identity shared): Model 2 – Faces dont match (identities differ): Both models have standard form of factor analyzer Compute likelihood in standard way, combine with prior in Bayes rule

41. Inference – Style translation 41 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Compute distribution over identity Generate in new style

42. Structure 42 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Factor analysis review Subspace identity model Linear discriminant analysis Non-linear models Asymmetric bilinear model Symmetric bilinear model Applications

43. Symmetric bilinear model 43 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Generative equation: Probabilistic form: Mean can also depend on style...

44. Symmetric bilinear model 44 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

45. Multilinear models 45 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Extension of symmetric bilinear model to more than two factors e.g.,

46. Structure 46 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Factor analysis review Subspace identity model Linear discriminant analysis Non-linear models Asymmetric bilinear model Symmetric bilinear model Applications

47. Face recognition 47 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

48. Synthesizing animation 48 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

49. Discussion 49 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Generative models Explain data as combination of identity and style factors In identity recognition, we build models where identity was same or different Other forms of inference such as style translation also possible