1. Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk

2. Aim and Motivation: Generate new emotion specific video sequences. Facial expressions + head motion. Potentially infinitely long No repeated footage Provide an animation tool: User driven. Allows emotion selection. Automatic generation of novel facial expression sequences.

3. Approach: Statistical Approach  Divides into 2 stages Stage 1 Modelling: To form generative emotion models. Shape model point data, potential to drive animations Combined appearance space model texture and shape for the generation of new video frames. Stage 2 Synthesis: Method to generate new video footage / point data

4. Emotion video footage is considered as a time series of changing facial expressions. Time series can be modelled with Auto regressive processes (ARPs)  Advantages: constructed models generate new data similar to training data. Potentially infinitely long generated sequences.  Disadvantages: For successful modelling, input data must have stationary signal. Use ARP’s to form generative emotion specific models. These allow synthesis of new facial expression emotion sequences. Stage 1: Modelling Approach Time

5. Modelling Process PCA- Principle components Analysis Data Acquisition: Video footage Apply PCA. Model variation of head position + facial expression. Calculate emotion “Eigensignatures” INPUT: Emotion eigensignature to ARP. OUTPUT: Emotion ARP model: e.g. happy ARP sad ARP 1 2 3 4 5 Construct ARP’s

6. Modelling: Data Collection Acquire video footage  Emotions: Happy, sad and, angry. Split video into: 1) Shape representation  Using hand labelled control points. 2) Texture representation  Using warped image set Neutral pose  mean shape of control point set.

7. Modelling: Facial expressions Apply principle components analysis (PCA) video data to calculate eigenvectors for: Shape space (point data) Texture space (warped pixel data) Combined space (shape + texture). Calculate shape and combined space responses. Data projected from a higher-D space down to lower D-space. Each video frame has a response Together the responses for a sequence form an eigensignature. Form linear shape and combined space models.  Allow reconstruction of original data.

8. Linear shape model: Where: X i Reconstructed point data Φ s is a set of eigenvectors X is the mean shape. b s are shape responses. Technique: Shape Space 1.Apply PCA to control point shape data. Calculate eigenvectors Φs. 2.Calculate shape responses for each video frame (EIGENSIGNATURE). 3.Form Linear shape model Shape responses b s : Project point data into the lower dimensional space defined by Φs. Where: Φs Shape Eigenvectors. X point data Mean shape bs(i) is one point in the eigensignature

9. Face Expression Modelling: Shape #2 By moving along the length of each eigenvector,, varying distance b, in turn,  The different modes of variation described by the eigenvectors can be observed. Mean Image -3 S.D +3 S.D

10. PCA- Selected modes of variation of face expression shape model Mode 1 Mode 8 Mode 6 Mode 5

11. Technique: Form Texture Model Warp each video frame to neutral pose. Apply PCA to warped image set. Calculate eigenvectors Φt for warped image set. Calculate texture responses for each video frame (as with shape space model). Form linear texture model. Linear Texture Model: Where: I i - Reconstructed warped image. Φ t - Eigenvectors which describes the majority of texture variation within the dataset. I - Mean warped image b t - Image responses

12. Technique: Form Combined Space Model 1.Combine shape and texture model. 2.Apply PCA again to get combined model, to obtain eigenvectors P c. 3.Calculate combined appearance responses, C. 4.Form linear combined appearance model Combined shape and texture responses Where: W s is a weight matrix Linear Combined Appearance Model: Where : P c are the combined appearance eigenvectors C are the combined space responses

13. Modelling: Eigensignatures Visualise eigensignature:  Plot of 1 st and 2 nd principle components (responses). Each point of the eigensignature represents a single video frame. Angry Happy Sad SHAPE EIGENSIGNATURE Happy Sad Angry COMBINED EIGENSIGNATURE

14. Modelling: Construct ARP’s 3 video clips yield: Happy eigensignature Sad eigensignature Angry eigensignature Consider eigensignature as a multivariate time series. Separate application of multivariate-ARP’s  Results: separate ARP emotion models. Models used to generate new emotion specific eigensignatures-(stage 2: SYNTHESIS)

15. Stage 2: Synthesis Generate eigensignatures  Potentially infinitely long  Retains look and feel of original. Original Eigensignature: Emotion Angry Generated ARPEigensignature: Emotion Angry

16. Shape Eigenvectors Synthesis: New sequence generation Use linear PCA models.  To project generated eigensignature back to original space.  Example: Shape Sequence Generation Generated Eigensignature Mean Shape Generated Shape sequence

17. Synthesis: Generated Shape ARP sequences: Original Angry ARP Generated Angry  Using linear shape space PCA models.  Project generated shape eigensignature back to shape space.

18. Synthesis: Generated Shape ARP sequences: ARP Generated Happy ARP Generated Sad

19. Results of ARP: Synthesised sequences using auto regressive process Original Happy ARP Generated Happy  Use linear shape space PCA models.  To project generated eigensignature back to original space.

20. Synthesis: Generated Combined space ARP sequences: Original Angry ARP Generated Angry  Using combined appearance PCA models.  Project generated eigensignatures back to combined appearance space.

21. Synthesis: Generated Combined space ARP sequences: ARP Generated Happy ARP Generated Sad

22. Results of ARP: Combined Appearance Space #3 Original Happy ARP Generated Happy

23. Application: User-driven synthesis Approach:  Self organising maps (SOMs)  SOM’s project the n-D ARP emotion data into a 2D image space, defined as ‘expression space’. Tool development  Expression Space visualises emotion ARP models.  Navigate space.  Automatic generation new emotion specific videos.

24. Self-Organising Maps Clustering technique Kohonen Neural Network Visualises n-observations as g-groups. SOM splits into 2 elements 1) Data – provided by ARP models Requires sufficient training data. 2) Weight vectors components. xy position on map After training weight vectors equivalent to ARP models. x 1,y 1 W 1, W 2 W 3 W 4 Weight Vectors ARP-coefficients x 2,y 2 x 3,y 3 x 4,y 4 SOM UNITS Self-Organising Map Structure

25. Technique: Acquisition of SOM training data. COMPLETE EMOTION EIGENSIGNATURE Use N emotion ARP’s to train 50x50 unit SOM. Split complete eigensignature into frame segments with (length-1) frame overlap. 12 N-segments Model each segment with an ARP. Result: obtain N ARP models for SOM training. Results After training : Each unit of the SOM is equivalent to a single ARP model. ARP emotion models are clustered.

26. Technique: Forming Expression Space Generate Expression space by colour coding the SOM units.  Use an Euclidean distance scheme, based on comparison between the sample ARP emotion models used for training and SOM map unit ARP’s. Red units most similar to angry ARP models Green units most similar to happy Blue unit associate to sad ARP models Intensity of colour indicates the better match to an emotion group.

27. Results: SOM Shape Space SOMCombined Space SOM Emotions cluster into 3 distinct regions.  Distinct separation between angry and all other emotions.  Large transitional region between sad and happy.  Dark regions not closely related to any emotion. Angry Cluster Sad Cluster Happy Cluster Angry Cluster Sad Cluster Happy Cluster

28. Navigating Expression Space This tool provides an intuitive interface. Aimed at non-specialist users Allows easy traversal of the highly dimensional expression space. Automatically generates new facial expression sequences for selected emotions. SOM Visualisation Generated Expression

29. Navigating the Combined Appearance Space SOM Visualisation Generated Video texture Complete Eigensignature Generated Eigensignature

30. Results: Using different ARP models from the SOM to generate sad crowd.

31. Results: Using different ARP models from the SOM to generate angry crowd.

32. Mixed Crowd Scene

33. Conclusions Demonstrated approach which can reconstruct and generate novel footage. Successful use of PCA and ARP’s:  To construct emotion models that correctly co- articulate head motion and facial expression. Eigensignature representation unique for each emotion.  Emotion ARP’s cluster distinctly  Clustered space easily navigable by non- specialists.

34. Finally…… Any questions?

35. Modelling overview: 1.Collect video footage of different emotions. 2.Model the variation of facial expression and head motion of the video data for each emotion. 3.Form separate emotion models. 4.Emotion models used a basis for the synthesis of new video sequences. 5.Form a navigable expression space by clustering and visualising the generated emotion models.

36. Overview Aim Approach  Modelling  Synthesis Results Conclusion