eNTERFACE 08 Project 1 “MultiParty Communication with a Tour Guide ECA” Mid-term presentation August 19th, 2008

Outline Project Summary (Repeating the lesson ;) ) System configuration Work in progress Speech Regonition Nonverbal Input Handling Input Understanding Dialogue Management Character Animation Player Conclusion

Summary We aim to create an ECA based system which will interract with two users by: Capturing their behaviors (speech, nonverbal behaviors) Combining and handling input data Generating and displaying response to these behaviors in the real time

Input Speech recognition Nonverbal behavior detection (Face detection, Facial orientiation, Hand raising) Central Part Input Understanding, Dialogue Management Output ECA animation player Platform to provide communication between the components System Configuration

Speech Recognition Two SR components will be running on different machines Keywords will be dynamically changed according the conversational state No significant overlapping between microphones (distance ~ 1.5 m) Implementation in progress: Keywords detection with confidence score and speech duration by using Loquendo API

Implemented: System which uses OpenCV to calculate difference between the frames to detect the appearance of users To do: Hand raising to ask the question by using OpenCV (If time permits) Nonverbal Inputs: Appearance of the User and Hand Raising

Implemented: Facial Orientation Detection Work in progress : Face tracking: Users are starting mutual conversation or leaving the system? Nonverbal Inputs: Face Orientation

Input Understanding Component Combines two verbal channels and image processing data to recognize users’ utterances Examples: who is speaking, who is adressee, are users’ starting mutual conversation...? Component is still in early stage since input components are not being implemented

Dialogue Management Component - Functionalities Make decisions “when and what to do to whom”: Handle multi-modal input events Handle user interruptions while the agent is doing something Keep a model of each user Keep the domain knowledge, discourse model, and context memory Generate multi-modal outputs

Dialogue Management Component - Progress Dialogue management component based on information state theory [Larsson’00] in still being implemented The progress of dialogue is represented by a set of variables These variables are updated or queried by a basic unit called dialogue move like ask, answer, repeat, inform Conversation strategies (plans) can be composed to respond to different circumstance Most appropriate plan are selected and scheduled by simple inference

Dialogue Management Component - Issues What to do when the agent is interrupted by the user? What to do when speech recognition fails? What to do when the users are talking to each other? What to do when the users starts to talk to the system at the same time? How ECA’s gaze direction can be controlled? How grounding can be implemented using verbal and non- verbal information? What to do when the response of the user is not what the system expected?

Character Animation Player Character animation (ECA) player generates speech and synchronized animation by using XML based GSML language Multiple threads in animation player obtain speech and gesture synchrony Set of animations implemented Gazing, beats, facial expressions (joy, sad...), symbolic gestures To do: construct ECA behaviors according the system scenario (canned behaviors, general states)

Character Animation Player: Example

Conclusion We are in (slow :S) progress To integrate a baseline system we need to implement: SR component Component to detect facial orientation Input Understanding DM component Behavior specification script for character Player Expected Outcome: there are many...