Interacting with Virtual Humans Zsófia Ruttkay Associate Professor Human Media Interaction, Dept. of CS, University of Twente, NL SSIP 2009 Debrecen

2 Overview of the talk FAAnalysis of facial expressions of humans and cartoons Emotion Disc Challenges for IP Non-photorealistic facial animation - CharToon Full body Control with markup – GESTYLE Current work - conductor Intro About Virtual Humans Evaluation of personality cues

3 About Virtual Humans I More or less human-like synthetic characters to interact with usersMore or less human-like synthetic characters to interact with users In roles as: - information provider - tutor - sales assistant - ‘translator’ (sign language) - avatar in telepresence and VR - entertainment, games - also medium for experimentsIn roles as: - information provider - tutor - sales assistant - ‘translator’ (sign language) - avatar in telepresence and VR - entertainment, games - also medium for experiments Research since 15 years, recently own events: IVA, Gesture WS, AAMAS workshop seriesResearch since 15 years, recently own events: IVA, Gesture WS, AAMAS workshop series Book by J. Cassell et al. (2000, MIT Press)Book by J. Cassell et al. (2000, MIT Press)

4Examples Rea from MIT, Cassell et al. Steve from USC J. Rickel, L. Johnson et al.

5Examples OLGA – KTH

6Examples Max – Univ. of Bielefeld

7Examples Mission Exercise, USC ISI + ICT

8Examples Carmen’s Bright Ideas from USC ICT, Marsella et al.

9Examples USC – ISI, Adele, Shaw et al.

10Examples Let’s Face It! –M. Bartlett et al. Univ. San Diego, 2008.

11 Virtual conductor Virtual Conductor HMI, Univ. of Twente, The Netherlands

12 Virtual scientific institute Meta Institute for Computational Astrophysics Piet Hut body-simulations-in-opensim/

13 Robots Hiroshi Ishiguro ATR Intelligent Robotics and Communications Laboratories

14 Geminoids Geminoid by I. Ishiguro at ATR, Japan

15 decision (what to do) About VHs - general architecture perception (input) action (output) nl processing (meta) speech understanding face interpretation gesture recognition environment monitoring (object, resources, situation) multimodal output related to environment user model affect model discourse model domain knowledge goals … real-time suspension of disbelief ‘good’ for …

16 About VHs – Design parameters Embodiment: head – torzo - fullEmbodiment: head – torzo - full ‘Look’: gender, age, profession, personality‘Look’: gender, age, profession, personality Design: realistic/not, 2d/3dDesign: realistic/not, 2d/3d Communication channels & directions speech, facial expressions, gestures, body languageCommunication channels & directions speech, facial expressions, gestures, body language What is to be expressed: emotional, cognitive, phys. state references to changing surrounding (incl. user)What is to be expressed: emotional, cognitive, phys. state references to changing surrounding (incl. user) Role: monologue – dialogue, relation to user: gender, cluture, socal statusRole: monologue – dialogue, relation to user: gender, cluture, socal status

17 About VHs – Hierarchy of behavior independent behavior independent behavior goal-oriented behavior goal-oriented behavior personality, emotions personality, emotions high-level control high-level control bidirectional mm comm. bidirectional mm comm. multimodal output multimodal output single modal output single modal output rendering, TTS,... rendering, TTS,...

18 About VHs - development cycle analysis of H-H interaction computational model implementation of VA evaluation of VA non-realism: animation, art What? How? realtime adoptable micro level in application mind body

19 About VHs – Some issues Mind and body issueMind and body issue Why to use VHs? - ease of use: broader user group, less load - more effect - more funWhy to use VHs? - ease of use: broader user group, less load - more effect - more fun Moral and legal issues: clones, manipulationMoral and legal issues: clones, manipulation To mimic humans, or less is more?To mimic humans, or less is more? Interdisciplinary researchInterdisciplinary research

20 About VHs – Do people take them? Illusion – do people believe, react to as if they were real?Illusion – do people believe, react to as if they were real? YES Clifford Nass: CASA paradigm Computers Are Social ActorsYES Clifford Nass: CASA paradigm Computers Are Social Actors Series of experiments on effect of micro and macro designSeries of experiments on effect of micro and macro design User’s characteristics matterUser’s characteristics matter

21 Overview of the talk FAAnalysis of facial expressions of humans and cartoons Emotion Disc Challenges for IP Non-photorealistic facial animation - CharToon Full body Control with markup – GESTYLE Current work - conductor Intro About Virtual Humans Evaluation of personality cues

22 Motivations for FA research Need for: expressive, animated synthetic faces Need for: expressive, animated synthetic facesexpressive Functions of facial (and other) nonverbal signals: - emotions - cognitive state - speech punctuation - dialog control - additional/redundant info (size, location, …) - personality and culture of speaker Functions of facial (and other) nonverbal signals: - emotions - cognitive state - speech punctuation - dialog control - additional/redundant info (size, location, …) - personality and culture of speaker All contribute to: comprehension, memory, effectivity … motivation, joy, engagement All contribute to: comprehension, memory, effectivity … motivation, joy, engagement

23 Questions concerning FA Which expressions to consider? Which expressions to consider? What are those expressions like? What are those expressions like? - static and dynamical characteristics - generic and individual characteristics What facial model to use? - (non-)photorealism - beyond realism/simplification of motions? What facial model to use? - (non-)photorealism - beyond realism/simplification of motions? Tool to make synthetic face models and animate them Tool to make synthetic face models and animate them

24 Facial anatomy about 44 pairs of musclesabout 44 pairs of muscles –bundle –sheet –circular physiological data missingphysiological data missing coarticulationcoarticulation many parametersmany parameters nonlinear effectsnonlinear effects

25 Psychology of facial expressions “meaningful” – physically possible - impossible“meaningful” – physically possible - impossible biological need/content/emotional state/discourse regulator/…biological need/content/emotional state/discourse regulator/… muscle activity – emotion sensing in brainmuscle activity – emotion sensing in brain difference between real and ‘on demand’ expressions data collection must be done with care: - spontaneous/stimulated expression - natural setting (e.g. in news) - real people vs. actor?difference between real and ‘on demand’ expressions data collection must be done with care: - spontaneous/stimulated expression - natural setting (e.g. in news) - real people vs. actor?

26 History …. 1882: Darwin: The Expression of the Emotions in Man and Animals1882: Darwin: The Expression of the Emotions in Man and Animals 1862: Duchenne: photos of expressions induced by electric signals1862: Duchenne: photos of expressions induced by electric signals

27

28 History: coding of facial expressions : Paul Ekman (U. of California) : Paul Ekman (U. of California) 1989: 6 general emotional expressions : happiness, surprise, sadness, fear, disgust, anger1989: 6 general emotional expressions : happiness, surprise, sadness, fear, disgust, anger FACS [1978] coding system – still preferred in USFACS [1978] coding system – still preferred in US discretediscrete visual effectof (groups of) musclesvisual effectof (groups of) muscles

29 Coding of facial expressions: MPEG4 1999: ISO standard1999: ISO standard widely usedwidely used 68 normalised parameters (FAPs)68 normalised parameters (FAPs) continuouscontinuous black points x, y, (z) displblack points x, y, (z) displ min, max, neutral valuemin, max, neutral value

30 Emberi arckifejezések leirása: MPEG4 # FAP name FAP description Unit# FAP name FAP description Unit 31 raise_l_i_eyebrow Vertical displacement of left inner eyebrow ENS31 raise_l_i_eyebrow Vertical displacement of left inner eyebrow ENS 32 raise_r_i_eyebrow Vertical displacement of right inner eyebrow ENS32 raise_r_i_eyebrow Vertical displacement of right inner eyebrow ENS 33 raise_l_m_eyebrowVertical displacement of left middle eyebrow ENS33 raise_l_m_eyebrowVertical displacement of left middle eyebrow ENS 34 raise_r_m_eyebrow Vertical displacement of right middle eyebrow ENS34 raise_r_m_eyebrow Vertical displacement of right middle eyebrow ENS 35 raise_l_o_eyebrow Vertical displacement of left outer eyebrow ENS35 raise_l_o_eyebrow Vertical displacement of left outer eyebrow ENS 36 raise_r_o_eyebrow Vertical displacement of right outer eyebrow ENS36 raise_r_o_eyebrow Vertical displacement of right outer eyebrow ENS 37 squeeze_l_eyebrow Horizontal displacement of left eyebrow ES37 squeeze_l_eyebrow Horizontal displacement of left eyebrow ES

31 Analysis of facial expressions – own investigation 2002 On demand expressions of untrained subjects Track 15 FAPs marked on face Track 15 FAPs marked on face

32 The E-Space: PCA

33 The E-Space: 2 components’ graph (tracked) Hor: eyebrows Vert: mouth corners

34 The E-Space: CharToon data

35 The E-Space: CharToon data Experienced animatorExperienced animator 59 expression stills:59 expression stills: –Anger: Be_Careful_What_You_Say –Smile: Absolute_Joy, Adoration, Smile –Other: Doubt_suspicion

36 The E-Space: Graph (cartoon) Hor: eyebrowsHor: eyebrows Vert: stretch mouthVert: stretch mouth 3rd: smile/squeeze3rd: smile/squeeze (all a-symmetric)

37 The E-Space: Conclusion (stills) Tracker FAPs not always sufficient to distinguish (negative) emotionsTracker FAPs not always sufficient to distinguish (negative) emotions More symmetry in tracked expressionsMore symmetry in tracked expressions Animator uses more of the expression spaceAnimator uses more of the expression space smiling shape of the mouth and raising of the eyebrows are most significantsmiling shape of the mouth and raising of the eyebrows are most significant

38 Time curves of FAPs

39 Actuation of muscles during smile and anger (I. Essa, PhD at MIT, 1995)

40 The E-Space: Time curves

41 Co-actuation of muscles during smile (I. Essa, PhD at MIT, 1995)

42 Facial expression analysis - dynamism Time IS characteristic and importantTime IS characteristic and important Trapezoid activation function, variants (camel, …)Trapezoid activation function, variants (camel, …) Still open questions: shapes, constraints, …Still open questions: shapes, constraints, … No data(base) published (MS either)No data(base) published (MS either)

43 STAR on facial expression tracking Near real-time tracking of faps Other than basic 6 expressions Mixture, masking, cultural variants Dynamism of articulation Spontaneous expressions Bartlett, M.S., Littlewort, G.C., Frank, M.G., Lainscsek,C., Fasel, I., Movellan, J.R. (2006). Automatic Recognition of Facial Actions in Spontaneous Expressions. Journal of Multimedia 1(6) p Bartlett, M.S., Littlewort, G.C., Frank, M.G., Lainscsek,C., Fasel, I., Movellan, J.R. (2006). Automatic Recognition of Facial Actions in Spontaneous Expressions. Journal of Multimedia 1(6) p Cohn, J. F. & Ekman, P. Methods for measuring facial actions. In J.A. Harrigan, R. Rosenthal, & K. Scherer (Eds.), Handbook of nonverbal behavior research methods in the affective sciences. NY: Oxford.Cohn, J. F. & Ekman, P. Methods for measuring facial actions. In J.A. Harrigan, R. Rosenthal, & K. Scherer (Eds.), Handbook of nonverbal behavior research methods in the affective sciences. NY: Oxford. Tian, Y.L., Kanade, T., & Cohn, J.F. Facial expression analysis. In S.Z. Li & A.K. Jain (Eds.), Handbook of face recognition. NY: Springer.Tian, Y.L., Kanade, T., & Cohn, J.F. Facial expression analysis. In S.Z. Li & A.K. Jain (Eds.), Handbook of face recognition. NY: Springer. KTH, Sweeden – cheap and light hw for facial trackingKTH, Sweeden – cheap and light hw for facial tracking

Analysis of Mona Lisa smile: 83% disgust: 9% fear: 6% anger: 2% result of computer facial expression analysis UvA + Univ. of Illionis, Dec What do the numbers mean? How were they obtained? Would it mean the same in the 1500s?

45 Overview of the talk FAAnalysis of facial expressions of humans and cartoons Emotion Disc Challenges for IP Non-photorealistic facial animation - CharToon Full body Control with markup – GESTYLE Current work - conductor Intro About Virtual Humans Evaluation of personality cues

46 Facial animation – model based Basic principle: Deformation principles – 2d skeleton, 3d real/virtual muscle, …Deformation principles – 2d skeleton, 3d real/virtual muscle, … Key positions: values of controllersKey positions: values of controllers Inbetweens by interpolationInbetweens by interpolationIssues: Reusable, parametrised repertoireReusable, parametrised repertoire BlendingBlending Superposition (talking while smiling)Superposition (talking while smiling) Visual speech - coarticulationVisual speech - coarticulation

47 Facial animation how to interpolate: linear, C 1, C 2how to interpolate: linear, C 1, C 2 how to choose t i (times for keyframes)how to choose t i (times for keyframes) t1t1 t2t2 t3t3 t4t4 t5t5 toto

48 3D model

49 3d facial modelling (CWI)

50 3d facial animation

51 MPEG4 defined animation for different faces

52 Visual speech

53Lip-sync

54 Some 2d head models

55 Some movies made with CharToon by A. Lelievre and Zs. Paal Lily Frans Nine faces Magician

56 Personality and culture in facial design

57 3D realistic3D realistic vs. 2D cartoon faces 2D cartoon 3D realistic2D cartoon

58 Component repertoire Unlimited freedom design faces from CharToon components design faces from CharToon components finetune dynamism finetune dynamism Repertoire of: facial feature components facial feature components (still) expressions and animations (still) expressions and animations Support to reuse the components: technical level (design, UI) technical level (design, UI) ‘mix & match’ recipes with respect to expected functionality ‘mix & match’ recipes with respect to expected functionality

59 Animating faces in CharToon I

60 Animating faces in CharToon II

61 Animating faces in CharToon III + constraints technology

62 Emotion Disc

63 Emotion Disc

64 Emotion Disc Cyclic neighborhood of 6 basic emotionsCyclic neighborhood of 6 basic emotionsCyclic neighborhood Cyclic neighborhood Tool to: - explore the expression space - control the expressionsTool to: - explore the expression space - control the expressions Experiments with expressive 2D facesExperiments with expressive 2D faces -how expressive they are? - contradicting expressions (2-3 discs) -how expressive they are? - contradicting expressions (2-3 discs)

65 From: Harold Schlosberg: The Description of Facial Expressions in Terms of Two Dimensions, Journal of Experimental Psychology, Vol. 44. No. 4. Oct. 1952, pp

66 Overview of the talk FAAnalysis of facial expressions of humans and cartoons Emotion Disc Challenges for IP Non-photorealistic facial animation - CharToon Full body Control with markup – GESTYLE Current work - conductor Intro About Virtual Humans Evaluation of personality cues

67 Evaluation of talking heads Work with E. Krahmer at als.Work with E. Krahmer at als. Effect of facial signals to express: - emphasis --- eyebrow & pitch - personality --- eyebrow, gaze and speechEffect of facial signals to express: - emphasis --- eyebrow & pitch - personality --- eyebrow, gaze and speech See publications, experimental mat. on web pageSee publications, experimental mat. on web page

68 The evaluation research questions Do facial cues contribute to the perception of extraversion?Do facial cues contribute to the perception of extraversion? How do gaze + brows + speech cues add up?How do gaze + brows + speech cues add up? –inconsistency

69 Personality for VHs Personality is important:Personality is important: –to improve comprehension (Laurel 1993) –to like and trust and … the VA better (Nass and Lee 2000, Isbister and Nass 2000, …) –people attribute personality to agents CASA paradigm (Reeves and Nass 1996) –fun, variety

70 Personality for VHs Trait theory (e.g. Allport 1937)Trait theory (e.g. Allport 1937) “Big 5” (e.g. Wiggins 1997): Extroversion (dominance), Agreeableness, Conscientiousness, Emotional stability, Culture“Big 5” (e.g. Wiggins 1997): Extroversion (dominance), Agreeableness, Conscientiousness, Emotional stability, Culture Cues studied so far:Cues studied so far: –speech (pitch and duration) –gaze –posture (Isbister and Nass 2000) –language

71 The experiment GazeSpeechBrows Extrovert fixed on user 2 blink wide pitch range more variation higher tempo 2x accent 3x100 ms stages each Introvert “move S-W look aside move center” 2x low pitch range little variation 20% slower no movement 2 x 2 x 2 = 8 personality profiles

72 The experiment Examples of stimuli from of stimuli from is_ig_ib es_eg_ib is_ig_ib

73 Evaluation of VHs Hot topic, evaluation is a MUSTHot topic, evaluation is a MUST Zs. Ruttkay, C. Pelachaud: From Brows to Trust (Kluwer, 2004)Zs. Ruttkay, C. Pelachaud: From Brows to Trust (Kluwer, 2004)

74 Overview of the talk FAAnalysis of facial expressions of humans and cartoons Emotion Disc Challenges for IP Non-photorealistic facial animation - CharToon Full body Control with markup – GESTYLE Current work - conductor Intro About Virtual Humans Evaluation of personality cues

75 Research questions on gestures What gestures do people use?What gestures do people use? - references to environment & user (location, size, time, motion, shape,…) - to indicate actions and concepts - emotions, cognitive and physical state - dialog status What gestures should an VA use?What gestures should an VA use? - function - application domain - redundancy - expressivity - realization – non-repetitive, tuned - realization – non-repetitive, tuned Modelling and sw engineering issues: motion characteristics, performance, reusability, …Modelling and sw engineering issues: motion characteristics, performance, reusability, … As humans? YES, but …

76 Communicative functions emphasize take turn/ start talking a) raise voice b) look at listener c) raise eyebrow d) beat with hand e) 11 (?) combinations a) look at partner b) raise hand c) voice signal (hm- hm) d) … Multi-modal signals

77 Why to bother about style? Style is source of information on:Style is source of information on: –gender, age, physical state –personality, emotional state –ethnicity, profession –relationship between speaker and listener –situation (public/private, …) … variety and joyvariety and joy does matter in HCI (Nass et al. 2000, …)does matter in HCI (Nass et al. 2000, …)

78 Our objectives VA with style in nonverbal communication (and speech)VA with style in nonverbal communication (and speech) Declare style of VADeclare style of VA Generate styled conversation by VHsGenerate styled conversation by VHs Framework (can be tailored/extended)Framework (can be tailored/extended) Easy to use mark-up languageEasy to use mark-up language Experiments!Experiments!

79 GESTYLE: the big picture Style Declaration Meaning Tags Combine Style Dictionaries into a single one Map Meanings to Gesture Expressions Text with Gesture Expressions (relative time) Expand to Basic Gestures with absolute timing Text with Basic Gesture Tags (abs. time) Combined Style Dictionary Timing of speech Generate animation in rendering system Gesture Dictionary Text with Modifier Tags Modify Combined Style Dictionary Modified Combined Style Dictionary Style Dictionary 1 Meaning mapping 1 … Meaning mapping m 1 Style Dictionary 2 Meaning mapping 1 2 … Meaning mapping m 2 … Style Dictionary n Meaning mapping 1 n … Meaning mapping m n Modifier Tags input text with GESTYLE markups

80 GESTYLE: Style dictionary <GestureSpec> </GestureSpec><GestureSpec> </GestureSpec></Meaning>…

81 Definition and generation of gestures structural characteristics:structural characteristics: composed of basic gestures by par, seq and repeat operators performance (manner) characteristics:performance (manner) characteristics: intensity, duration, start_time, precision, jerkyness, symmetry, … parameters

82 GESTYLE: Combining styles Mechanism to resolve conflicting prescriptions on:Mechanism to resolve conflicting prescriptions on: –gestures –modalities –manner Preferences to be given in Style DeclarationPreferences to be given in Style Declaration Dynamical modifications of:Dynamical modifications of: –modality usage –manner –style dominance

83 GESTYLE: Gesture repertoire <GestureRepertoire>... </GestureRepertoire>

84 Two examles Hamlet: CharToon 2DHamlet: CharToon 2D –face, eyes, hands –.avi Yt: 3D H-anim humanoid for the WebYt: 3D H-anim humanoid for the Web –hand gestures only –real-time, on the fly generation of animation Speech by MindMaker’s FlexVoice TTS, styled too for Hamlet!Speech by MindMaker’s FlexVoice TTS, styled too for Hamlet!

85Hamlet HamletMarkUp1.xml HamletMarkUp2.xml StyleDictionaries.xml

86Yt hierarchical articulated body, 54 DOF of hand joints modular gesture definition written in STEP (VU)emphasisenumeratesorry_forgreeting

87 Cognition and action The Functional Behaviour Markup Languages initiative Intent Planning Behavior Planning Behavior Realization Behavior Repository (BML) FMLBML

88 What is BML? - Abstraction gesture head speech posture gaze facialexpression locomotion viseme eyes BML = “complex/multimodal” BML ≠ “simple/unimodal” (that’s at the level of body) torso legs Coordination

89 What is BML? - Example This is an example...

90 Work on BML On-line wiki, publication at IVA07 Join the small and active community IVA09 conference Sept 2009 iva09.dfki.de BML ws: 13 Sept 2009

91 Overview of the talk FAAnalysis of facial expressions of humans and cartoons Emotion Disc Challenges for IP Non-photorealistic facial animation - CharToon Full body Control with markup – GESTYLE Current work - conductor Intro About Virtual Humans Evaluation of personality cues

92 Conducting with parametrised kinematic hand motion H. van Welbergen, HMI, UT work in progress

93 Conducting – body motion with physical model H. van Welbergen, HMI, UT work in progress

94 Fat/thin conductor H. van Welbergen, HMI, UT work in progress

95 Conductors 30 in real-time H. van Welbergen, HMI, UT work in progress

96 Image morphing - automatic identification of feature points - interpolation - per region (e.g. lip sync) - db/analysis of faces - combination of morphing and 3d transformations Challenges for IP Face and gesture recognition - elicitation of characteristics for modeling (off-line) - real-time recognition of conveyed meaning - user modeling for interaction control, on-line - user analysis to learn about how they react to VHs - multimodal recognition

97 Challenges for IP Image analysis and synthesis -NPAR (e.g. line drawing) from photos - exaggeration - ageing, tiredness and other effects Texturing - LOD

98

99 From: J. Walker, L. Sproull, R. Subramani: Using a Human Face in an Interface, CHI’94, pp