Model-based Image Interpretation with Application to Facial Expression Recognition Matthias Wimmer

2007, December 11 th 2/25 Technische Universität München Matthias Wimmer Communication Schemes Natural human-computer interaction

2007, December 11 th 3/25 Technische Universität München Matthias Wimmer Example 1: Nissan Pivo 2

2007, December 11 th 4/25 Technische Universität München Matthias Wimmer Example 2: Sony’s Smile Shutter

2007, December 11 th 5/25 Technische Universität München Matthias Wimmer Outline of this Presentation Facial Expression Recognition Model-based image interpretation Adaptive skin color extraction image facial expression

2007, December 11 th 6/25 Technische Universität München Matthias Wimmer Facial Expression Recognition

2007, December 11 th 7/25 Technische Universität München Matthias Wimmer What are Facial Expressions?  Six universal facial expressions (Ekman et al.)  Laughing, surprised, afraid, disgusted, sad, angry  Cohn-Kanade-Facial-Expression database  Performed  Exaggerated  Determined by  Shape  Muscle motion

2007, December 11 th 8/25 Technische Universität München Matthias Wimmer Why are they difficult to estimate?  Faces look differently  Hair, beard, skin-color, …  Different facial poses  Only slight muscle activity

2007, December 11 th 9/25 Technische Universität München Matthias Wimmer Our Approach motion features and structural features

2007, December 11 th 10/25 Technische Universität München Matthias Wimmer Model Fitting with Learned Objective Functions

2007, December 11 th 11/25 Technische Universität München Matthias Wimmer Model-based image interpretation  The model The model contains a parameter vector that represents the model’s configuration.  The objective function Calculates a value that indicates how accurately a parameterized model matches an image.  The fitting algorithm Searches for the model parameters that describe the image best, i.e. it minimizes the objective function.

2007, December 11 th 12/25 Technische Universität München Matthias Wimmer Local Objective Functions

2007, December 11 th 13/25 Technische Universität München Matthias Wimmer Ideal Objective Functions P1:Correctness property: Global minimum corresponds to the best fit. P2:Uni-modality property: The objective function has no local extrema. ¬ P1 P1 ¬P2 P2  Don’t exist for real-world images  Only for annotated images: f n ( I, x ) = | c n – x |

2007, December 11 th 14/25 Technische Universität München Matthias Wimmer Learning the Objective Function x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x  Ideal objective function generates training data  Machine Learning technique generates calculation rules

2007, December 11 th 15/25 Technische Universität München Matthias Wimmer Benefits of the Machine Learning Approach  Accurate and robust calculation rules  Locally customized calculation rules  Generalization from many images  Simple job for the designer  Critical decisions are automated  No domain-dependent knowledge required  No loops

2007, December 11 th 16/25 Technische Universität München Matthias Wimmer Evaluation: Fitting Accuracy

2007, December 11 th 17/25 Technische Universität München Matthias Wimmer Adaptive Skin Color Classification

2007, December 11 th 18/25 Technische Universität München Matthias Wimmer Basics about Skin Color Classification green image 1 image 2  Skin color depends on several image conditions  Skin color occupies a large cluster  Skin color varies greatly within a set of images.  Skin color varies slightly within one image. image 2 green red

2007, December 11 th 19/25 Technische Universität München Matthias Wimmer Our Approach  Learn image-specific skin color characteristics  Parameterize a skin color classifier accordingly Offline:  Learn the skin color mask  Specific for the face detector Online:  Detect the image specific skin color model  Using the face detector  Using the skin color mask  Adapt skin color classifier

2007, December 11 th 20/25 Technische Universität München Matthias Wimmer Results  Robustness:  Detection of facial parts: eyes, lips, brows,…  Exact shape outline  Ethnic groups Correctly detected pixels:  fixed classifier:90.4%74.8%40.2%  adapted classifier:97.5%87.5%97.0%  improvement: adapted classifier fixed classifier original image

2007, December 11 th 21/25 Technische Universität München Matthias Wimmer Additional Work lip classifier eye brow classifier iris classifier tooth classifier

2007, December 11 th 22/25 Technische Universität München Matthias Wimmer Conclusion and Outlook

2007, December 11 th 23/25 Technische Universität München Matthias Wimmer Conclusion  Possible to derive information from face images  Model-based image interpretation is beneficial  Learn crucial decisions within algorithms  Don’t specify parameters by trial and error  Adaptive skin color classifier  Learned objective functions  Not yet reached goal for natural HCI  Progress is clearly visible. → Goal is achievable!

2007, December 11 th 24/25 Technische Universität München Matthias Wimmer Outlook  Learn global objective function  Learn discriminative function (direct parameter update)  Rendered AAM provides training images  Many images  Exact ground truth (no manual work required)  Learn with further features  Higher number of features  SIFT, LBP, …  Learn with better classifiers  Relevance Vector Machines  Boosted regressors

2007, December 11 th 25/25 Technische Universität München Matthias Wimmer Thank you! Online-Demonstration:

2007, December 11 th 26/25 Technische Universität München Matthias Wimmer Publications 2008  Tailoring Model-based Techniques for Facial Expression Interpretation. ACHI08  Face Model Fitting with Generic, Group-specific, and Person-specific Objective Functions. VISAPP  Low-level Fusion of Audio and Video Feature for Multi-modal Emotion Recognition. VISAPP  Facial Expression Recognition for Human-robot Interaction - A Prototype. Robot Vision 2007  Audiovisual Behavior Modeling by Combined Feature Spaces. ICASSP  Emotionale Aspekte in Produktevaluationen. Multimediatechnik  Application of emotion recognition methods in automotive research. Emotion and Computing  Human Capabilities on Video-based Facial Expression Recognition. Emotion and Computing  SIPBILD - Mimik- und Gestikerkennung in der Mensch-Maschine-Schnittstelle. INFORMATIK  Learning Robust Objective Functions with Application to Face Model Fitting. DAGM  Automatically Learning the Objective Function for Model Fitting. MIRU  Initial Pose Estimation for 3D Models Using Learned Objective Functions. ACCV  Estimating Natural Activity by Fitting 3D Models via Learned Objective Functions. VMV  Learning Local Objective Functions for Robust Face Model Fitting. PAMI (journal paper)  Enabling Users to Guide the Design of Robust Model Fitting Algorithms. ICV 2006  Learning Robust Objective Functions for Model Fitting in Image Understanding Applications. BMVC  A Person and Context Specific Approach for Skin Color Classification. ICPR  Adaptive Skin Color Classificator. Journal on Graphics, Vision and Image Processing (journal paper)  Bitte recht freundlich. Zukunft im Brennpunkt (journal paper) 2005  Sensor-based Situated, Individualized, and Personalized Interaction in Smart Environments. INFORMATIK  Adaptive Skin Color Classificator. GVIP 2004  Experiences with an Emotional Sales Agent. Affective Dialogue Systems