Herwin van Welbergen Dennis Reidsma Stefan Kopp

 Beyond turn taking interaction ◦ Continuous perception and behavior generation  Interpersonal coordination ◦ Tight coordination of the (ongoing) behavior of the agent with other (virtual) humans

 Acknowledge reception, understanding, attitude while listening  Respond to turn taking signals ◦ Graceful interruption  Elicit backchannel feedback (and wait for it)  Establish joint attention  Exercise together with the interlocutor ◦ Synchronized movement  Construct utterances incrementally, on the fly ◦ No long upfront planning

 Between the retraction of one gesture and the preparation of the next ◦ Retraction of the first gesture is skipped  Co-articulation (or lack thereof) can have a communicative function (Kendon 1980) ◦ E.g. marking information boundaries  Has to be established on the fly

 Allow last minute changes to ongoing behavior ◦ Top down (through BML) ◦ Bottom up emergent behavior  But don’t break BML constraints  Real-time, incremental, fluently connect BML increments  Under-specification handling ◦ The realizer should be smart enough to figure out unconstrained timing, generate co-articulation,.. ◦ This should happen ‘on the fly’ ◦ As late as possible

 A BML block specifies some constraints in shape and timing on its behaviors  BML blocks are generally underspecified ◦ The BML Realizer has certain realization freedom ◦ Used to achieve natural looking motor behavior

 Offers specification for multimodal synchronization, but: ◦ No specification mechanisms for inter-personal synchronization ◦ Limited specification capabilities for incrementality ◦ No specification for co-articulation

 SmartBody (Thiebeax et al 2008)  EMBR (Heloir and Kipp 2010)  Greta (Mancini et al 2008)  RealActor (Čerecović and Pandžić 2010)  Special focus on incrementality/interactional coordination ◦ Elckerlyc (van Welbergen et al. 2010) ◦ ACE (Kopp and Wachsmut 2004)

 Flexible behavior plan representation (Reidsma et al. 2011) ◦ retains BML constraints  Modified in top- down fashion  Specification mechanisms for interactional coordination

 Incremental behavior planning  Specification mechanism for chunk composition  Emergent gesture co- articulation through bottom-up plan modifications

 Co-articulation can have a communicative function  The Behavior Planner should be able to specify whether or not co-articulation may occur

At 6 pm you have another appointment. At 6 pm you have another appointment.

 Start a new BML block when a previous block is ‘retracting’ ◦ All behaviors have a retraction phase ◦ A BML block is ‘retracting’  If all its behaviors are either finished or retracting ◦ If needed, fluently overtake the retraction phase of a gesture in the previous block

 Gracious interruption depends on ◦ Behavior state (e.g. not started, retracting) ◦ Resting posture/state ◦ Current position of e.g. the hand  This information is available in the AnimationEngine ◦ Smart retraction using the AnimationEngine  Replace a interrupted gesture by a new one ◦ Co-articulation

 Top down specification that a pointing gesture/gaze should remain on target until an interlocutor event ◦ Joint attention ◦ Feedback elicitation  Bottom up emergent insertion/deletion of ‘hold’ motions

 AsapRealizer combines state of the art features of ACE and Elckerlyc  Fusion provides new capabilities: ◦ Interactional coordination + automatic hold phase construction ◦ Top down interrupt with bottom-up automatic graciousness + co-articulation  Top down+bottom up adaptation capabilities are essential for fluent interaction

Questions?

 Internal timing predictions of one or more modalities might be unpredictable ◦ E.g. for robot behavior ◦ Bottom up adjustments can be used to  Adjust the behavior itself (e.g. speed up to meet a time constraint)  Move the time constraints that cannot be met, automatically adjusting the timing on other modalities linked to these constraints

Interruption in Elckerlyc (a) Interrupting all behavior in bml1. (b) Interrupt all behavior in bml1 excluding gesture1. (c) Interrupt all behavior in bml1. Insert a behavior (relaxArm) that gracefully moves the gesturing arm back to its rest position.

 Gesture is specified in a TimedMotionUnit ◦ TimedMotionUnit specification:  State  Priority  Set of controlled joints ◦ TimedMotionUnit execution:  Set a joint rotation on the skeleton  Add a physical controller  Set a RestingTimedMotionUnit

 RestingTimedMotionUnit ◦ Manages motion related to the resting state of the virtual human  Possible implementations:  Static posture (as in ACE)  Lower body physical balance controller  … ◦ Is always executed with the lowest priority ◦ Can create transition TMUs to the resting state  Slerp  Drop the arm controller  …

 The priority of a TMU drops in its SUBSIDING state  AsapAnimationEngine executes TMUs in order of priority  If a TMU needs to execute motion on joints that are already in use by a higher priority TMU, drop the TMU  Automatically create a cleanup TMU that moves other joints back to the resting state

 Gesture changes during execution: ◦ start position of the hand ◦ hand position at the start and end of the stroke ◦ posture state at the end of the gesture  Adaptive timing and shape for gesture preparation and retraction phase ◦ Using Fitts’ law ◦ Only update if no other constraints act on e.g. both start and stroke-start timing

 BML specification mechanisms for: ◦ Interpersonal coordination  Gracious interruption [but verbose and top-down only]  Flexible plan representation ◦ Allows adaptation of the ongoing plan ◦ While maintaining BML constraints ◦ So far such plan modifications were mainly managed top-down (e.g. by new BML blocks) ◦ So far no combined shape and timing adaption mechanisms

 MURML specification for multimodal coordination  Incrementality ◦ Gesture co-articulation  Bottom up adaptation of ongoing behavior

 Top-down specification of a parameter change ◦ Increase the amplitude of an ongoing gesture  Bottom up adaptations: ◦ Update retraction and/or preparation timing and shape accordingly ◦ Might adjust unconstrained shape and timing parameters to retain biological plausibility

 Real-time multimodal behavior generation for robots and virtual humans ◦ Incremental construction of behavior  Fluent connection of increments (e.g. behavior co- articulation) ◦ Allowing last minute changes of ongoing behavior ◦ Multimodal synchronization  Builds upon ACE, Elckerlyc  BML ◦ BML 1.0 compliant (tested by RealizerTester)