APML, a Markup Language for Believable Behavior Generation Soft computing Laboratory Yonsei University October 25, 2004

1 Contents Introduction Expressing believable behaviors MagiCster architecture A markup language for behavior specification: APML –Overview of existing markup languages for expressing human-like behavior –Defining APML tags Facial description language An example Conclusion

2 Introduction Humans communicate using verbal and non-verbal signals –Body posture, gestures, facial expressions, gaze, intonation and prosody, and words and sentences Embodied conversational agents –Virtual body that interacts with another agent –Human-like manner –Believable way Express emotion Exhibit a given personality Two approaches on ‘Body’ and ‘Mind’ –Strictly and necessarily interdependent –Mainly independent from each other

3 EU project MagiCster ‘Mind’ and ‘Body’ are interfaced by a language based on XML During the conversation –Mind decides what to communicate considering different factors that trigger the goal of communicating and influence the contents to communicate –Body reads what the Mind decides to communicate and interprets and renders it at the surface level according to the available communicative channels Define a set of language for specifying the format of dialogue moves at different abstraction levels –APML (Affective Presentation Markup Language) –Express the content of the dialogue move at the meaning level

4 Chapter Overview Describe the main features of the underlying architecture Present the APML language and how it has been used in the context of the MagiCster project Show hot it has been interfaced with a 3D realistic face call Greta and a synthetic voice Use an example in the medical domain

5 Expressing Believable Behaviors Communication: A means to influence others Beliefs forming the content of a communicative act –Information about the world Deictics, adjectival –Information about the speaker’s identity –Information about the speaker’s mind Speaker’s beliefs: Degree of certainty, metacognitive information Goals: Performativity of the sentence, topic-comment or theme-rheme distinction, rhetorical relations, turn-taking and backchannel Emotions: Affective words, gestures, intonation, facial expression, gaze, and posture

6 MagiCster Architecture (1) An example of advice-giving dialogue in the medical domain –The agent (Gi): A doctor –The Interlocutor (Uj): A patient Coordination of the speech with various expressions –In move G1, she manifests her empathy with the User –In move G2, Greta indicates her chest while saying “a spasm of chest” –In move G3, she looks at the User while saying “your problem”

7 MagiCster Architecture (2) The MagiCster system –‘Mind’ component: A content planner, a dialogue manager, and an affective agent modeling module –‘Body’ component : A 3D face/avatar with a speech synthesizer –A plan enricher: An interface between the ‘Mind’ and ‘Body’ components

8 MagiCster Architecture (3) The affective agent modeling module: Decide –Whether a particular affective state should be activated –Whether the felt emotion should be displayed in a given context The content planner –Generate the discourse plan appropriate to the context –DPML DTD: Achieve the goal in that piece of conversation

9 MagiCster Architecture (4) The dialogue manager –Top of the TRINDI architecture –Compute dialogue moves and a space in which information relevant to the move selection The plan enricher –Translate the symbolic representation of a dialogue move into an Agent’s behavior specification at the meaning level –Translate the DPML-based tree structure into APML The face and body animation –Interpret the APML-tagged dialogue move –Decide how to convey every meaning

10 A Markup Language for Behavior Specification: APML High-level primitives for specifying behavior acts Express agent behavior at different levels of abstraction Control easily the behavior of ECAs independently of the body APML (the Affective Presentation Markup Language) –Specification of the agent behavior at the meaning level –Affective aspect of the communication between the agent and the user

11 Overview of Existing Markup Languages for Expressing Human-Like Behavior APML Human markup language (HML) –Provide a very abstract level language Difficult for controlling specific agent bodies Require developing complex interpreters –Enhance the fidelity of human communications –Allow the representation of physical, cultural, social, kinetic, psychological, and intentional features VHML: Provide several languages for acting on different modalities MPML (Multimodal Presentation Markup Language): Enable authors of web pages to add agents for improving human-computer interaction (MS-Agent) BEAT (Behavior Expression Animation Toolkit): Generate embodied agent’s animation from textual input Avatar markup language (AML): Represent a new high-level language to describe avatar animation

12 Defining APML Tags APML APML DTD: Communication function (A meaning-signal pair)

13 Facial Description Language (1) Describe facial expressions as (meaning, signal) pairs Define expressions to capture a slight variation –At a high level: A facial expression is a combination of other facial expressions already defined –At a low level: A facial expression is a combination of facial parameters Combine facial expressions due to distinct co-occurring communicative acts using a Bayesian network –Facial basis (FB): a basic facial movement –Facial display (FD): a set of FBs surprise = raised_eyebrow + raised_lid + open_mouth

14 Facial Description Language (2)

15 Facial Description Language (3)

16 An Example: Medical domain application (1) DPML APML

17 An Example: Medical domain application (2)

18 Conclusion Describe the architecture of the behavior generator of a believable conversational agent Focus on the importance of Mind-Body separation Define two XML-like markup languages to represent the Mints’ output and the Body’s input APML: Not available in public