Laboratory of Computational Engineering Michael Frydrych, Making the head smile Smile? -> part of non-verbal communication Functions of non-verbal communication To enhance linguistic information To give new information

Laboratory of Computational Engineering Michael Frydrych, Non-verbal communication Figure by Jari Kätsyri

Laboratory of Computational Engineering Michael Frydrych, Voice prosody Figure by Jari Kätsyri

Laboratory of Computational Engineering Michael Frydrych, Head Figure by Jari Kätsyri

Laboratory of Computational Engineering Michael Frydrych, Eyes Figure by Jari Kätsyri

Laboratory of Computational Engineering Michael Frydrych, Facial expressions Figure by Jari Kätsyri

Laboratory of Computational Engineering Michael Frydrych, Gestures Figure by Jari Kätsyri

Laboratory of Computational Engineering Michael Frydrych, Conversation regulation Figure by Jari Kätsyri

Laboratory of Computational Engineering Michael Frydrych, Conversational signals Figure by Jari Kätsyri

Laboratory of Computational Engineering Michael Frydrych, Maintenance Figure by Jari Kätsyri

Laboratory of Computational Engineering Michael Frydrych, Emotions Figure by Jari Kätsyri

Laboratory of Computational Engineering Michael Frydrych, Expressing emotions Figure by Jari Kätsyri

Laboratory of Computational Engineering Michael Frydrych, Conversational signals Figure by Jari Kätsyri

Laboratory of Computational Engineering Michael Frydrych, Voice Figure by Jari Kätsyri

Laboratory of Computational Engineering Michael Frydrych, Head Figure by Jari Kätsyri

Laboratory of Computational Engineering Michael Frydrych, Eyes Figure by Jari Kätsyri

Laboratory of Computational Engineering Michael Frydrych, Face Figure by Jari Kätsyri

Laboratory of Computational Engineering Michael Frydrych, Examples

Laboratory of Computational Engineering Michael Frydrych, Dimitris Metaxas, Normal Badler Modeling of hand gestures Modeling emotions, personality…

Laboratory of Computational Engineering Michael Frydrych, Facial expressions Emotional expressions in face Are expressions showing emotions? How they can be described? What kind of emotions are there?

Laboratory of Computational Engineering Michael Frydrych, Emotion in face: To communicate or to express? Communicating vs. expressing emotions Intentional communication (Chovil) vs. Involuntary behaviour (Ekman) Technology viewpoint Emotion = Emotional expression = (Non- verbal) communication (In most situations emotions inappropriate)

Laboratory of Computational Engineering Michael Frydrych, Emotions and facial expressions (Ekman) Emotion  Facial expression ? Emotional expressions can be inhibited No evidence for non-expressed emotions Facial expression  Emotion ? Emotional expressions can be faked There are facial muscles with no voluntary control Facial expression  Emotion (?)

Laboratory of Computational Engineering Michael Frydrych, Description of expression Several descriptions of expressions exist: Facial Action Coding System – FACS By Eckman and Friesen ”Language of facial expressions” MPEG-4 Synthetic-Natural hybrid coding (SNHC) “Technical” description for low-bandwidth transmission

Laboratory of Computational Engineering Michael Frydrych, FACS Noticeable changes, not exactly facial muscles Tool for identifying facial actions Not for modeling facial actions Does not describe dynamics Action units (AUs) AU1: Inner brow raiser … AU23: Lip tightener … AU66: Crosseye

Laboratory of Computational Engineering Michael Frydrych, FACS (Facial Action Coding System) Tool for identifying facial actions Action Units Not for modelling facial actions

Laboratory of Computational Engineering Michael Frydrych, AU examples: AU7 (lid tightener)

Laboratory of Computational Engineering Michael Frydrych, AU examples: AU9 (nose wrinkler)

Laboratory of Computational Engineering Michael Frydrych, AU examples: AU11 (nasolabial furrow deepener)

Laboratory of Computational Engineering Michael Frydrych, Pure and blended emotions 7 pure emotions Pleasure (pl), surprise (su), fear (f), hate (h), sorrow (so), disgust (d), (interest) Blended emotions Combinations of pure emotions Easy: pl+su, h+d, so+f Difficult: su+f, d+pl, h+so, f+pl (Some) additivity

Laboratory of Computational Engineering Michael Frydrych, Possible causes of blended expressions in real world Mixed emotions (sadness+disgust) Consequent emotions (surprise+happiness) Masking (anger+happiness) …

Laboratory of Computational Engineering Michael Frydrych, Attempts to add emotions

Laboratory of Computational Engineering Michael Frydrych, neutral sad surprise

Laboratory of Computational Engineering Michael Frydrych, basic expressions courtesy Jari Kätsyri

Laboratory of Computational Engineering Michael Frydrych, courtesy Jari Kätsyri Blended expressions

Laboratory of Computational Engineering Michael Frydrych, Textured × non-textured Identification of expressions from pictures and videos of textured and non-textured version of AP

Laboratory of Computational Engineering Michael Frydrych, Differences among databases Number of correct responses for pictures and videos of different databases

Laboratory of Computational Engineering Michael Frydrych, Neutral expression Average evaluations for AP textured and non- textured neutral expressions

Laboratory of Computational Engineering Michael Frydrych, Better implementation Collecting data

Laboratory of Computational Engineering Michael Frydrych, Collecting data for expressions happy anger

Laboratory of Computational Engineering Michael Frydrych, HUT facial expression database First Finnish database of moving facial expressions

Laboratory of Computational Engineering Michael Frydrych, Study: Interaction of blur x type x emotion New database, 6 actors 3 levels of blur Pictures, short video sequences 6 basic expressions 3 subject groups (17 each) each group – 1 level of blur each actor 6 expressions 3 actors pictures, 3 actors videos

Laboratory of Computational Engineering Michael Frydrych, Interaction of blur x type x emotion

Laboratory of Computational Engineering Michael Frydrych, The end Thank you! and … a little exercise again