1. Heterogeneous Multi-Robot Dialogues for Search Tasks Thomas K Harris, Satanjeev (Bano) Banerjee Alexander Rudnicky AAAI Spring Symposium 2005: Dialogical Robots

2. AAAI Spring ’05 Symposium: Dialogical Robots2 Communication among autonomous robots and humans → Embodied (not necessarily robotic, or even anthropomorphic) agents will become ubiquitous → NL dialogue will be a useful modality → Agent’s dialogue and task models will remain tightly coupled, and independent from other agents  We will be talking to multiple agents without a (completely) shared dialog system

3. AAAI Spring ’05 Symposium: Dialogical Robots3 Gedankenexperiment The problem of many robots: sweeper, furniture mover, baby monitor. Task: clean up house; don’t wake baby up. Who do you talk to? What do you say? Who talks back?

4. AAAI Spring ’05 Symposium: Dialogical Robots4 Issues for polylogical systems What’s a reasonable architecture? –A single system controlling multiple robots –A single DM directing autonomous robots –Shared understanding/generation –Shared microphone and speakers –Robots self-contained: Nothing shared How to communicate with multiple agents? –Directions to the entire team? (“talk into the air”) –Mediation through a designated team leader? (foreman) –Instructions to each team member? (direct management) –To a proxy agent within/without the team? (personal assistant)

5. AAAI Spring ’05 Symposium: Dialogical Robots5 What do we have today? A platform that supports multiple (robot) participants –Sphinx II ASR; Festival TTS –RavenClaw dialogue manager –Galaxy-II message-passing architecture –Several “back-end” interfaces A basic corridor movement domain –Commands to move robots along in vectors and towards named locations –Mechanisms for describing robot status and location to human interlocutor

6. AAAI Spring ’05 Symposium: Dialogical Robots6 AGENT2 AGENT1 HUMAN INTERFACE Current Architecture ASR Multi-voice TTS DM1 DM2 Robot1 Robot2 Interpret1 Interpret2

7. AAAI Spring ’05 Symposium: Dialogical Robots7 AGENT2 AGENT1 HUMAN INTERFACE Implementation Sphinx IIPhoenixHelios Galaxy Ravenclaw Bashful Ravenclaw Clyde Bashful Robot Communication System RosettaSwift

8. AAAI Spring ’05 Symposium: Dialogical Robots8 Human-Robot Treasure Hunt One Human Two robots –Segway –Pioneer Treasures “hidden” in large space TASK: Retrieve all treasures

9. AAAI Spring ’05 Symposium: Dialogical Robots9 Scenario for Search We found it! We are at

10. AAAI Spring ’05 Symposium: Dialogical Robots10 Issues in H-R communication 1. How do people decompose the task into sub-tasks? 2. What language do people use to get the tasks performed by the robots? 3. Given a human command, what is the expected robot behavior?  Explore using Wizard of Oz experiments

11. AAAI Spring ’05 Symposium: Dialogical Robots11 WOZ design Communication takes place by normal speech, walkie- talkie, and through the webcam. 1 experimenter, 2 robot actors, 1 participant. Experimenter places treasure, simulates robot treasure sensors. 1st robot actor is blind, but carries a webcam for the participant’s consumptions. 2nd robot actor can only move by crawling.

12. AAAI Spring ’05 Symposium: Dialogical Robots12 Annotation and analysis Data transcribed and annotated Utterances classified into functional categories 394 utterances 20 utterance categories 8 major categories Carnegie Mellon MockBrow annotation tool

13. AAAI Spring ’05 Symposium: Dialogical Robots13 Utterance/Task Breakdown  Controlling team behaviors Controlling team behaviors  Grounding Grounding  Positive/negative feedback  Informing robot of it’s state or the world  Explanations of commands  Orientation Grounding  Navigation Navigation  Simple Navigation commands  Spatial Referential Navigation  Object Referential Navigation  Manipulation Manipulation  Manipulating the environment  Manipulating treasure  Coverage Coverage  Manipulating the webcam view  Object coverage commands  Generic coverage  Asking about the robot’s abilities Asking about the robot’s abilities  Filler Filler  Real-time command modifications Real-time command modifications

14. AAAI Spring ’05 Symposium: Dialogical Robots14 Summary / Future Work Architecture for human-robot teams WoZ study of language requirements Different WoZ scenarios Implement language for robot team “Field” testing

15. AAAI Spring ’05 Symposium: Dialogical Robots15 Controlling team behaviors  “you guys get together”  “T- you go first and B- follow”

16. AAAI Spring ’05 Symposium: Dialogical Robots16 Grounding  Positive/negative feedback  “ok that’s better”  Informing robot of state  “so that’s up”  “I don’t see anything there”  Explanations of commands  “so I can see which direction is up”  Orientation Grounding  “What you’re facing now with the camera – is that the vehicle that you just circumnavigated”  “I can tell you’re going in the wrong direction, stop”

17. AAAI Spring ’05 Symposium: Dialogical Robots17 Navigation  Simple Navigation commands  “so um T- turn to you left”  “T- I want you to turn right 90 degrees”  “can you go in that general direction”  “can you proceed in that direction”  Spatial Referential Navigation  “go to that open area”  “continue around the periphery of that open area”  “back out of that alley”  “proceed in that direction until you find an opening to turn left”  Object Referential Navigation  “go over by T-”  “can you go on the other side of that vehicle”  “go over by the posters”

18. AAAI Spring ’05 Symposium: Dialogical Robots18 Manipulation  Manipulating the environment  “T- why don’t you move the trash can”  Manipulating treasure  “T- bring the coin to me”  Manipulating the webcam view  “ok B- look to your left”  “B- can you look around with the camera a little”

19. AAAI Spring ’05 Symposium: Dialogical Robots19 Coverage  Object coverage commands  “ok so examine the shelf”  “do you see something on that shelf in front of B-”  “can you look over by that table over there”  Generic Coverage  “do you see anything that looks interesting”

20. AAAI Spring ’05 Symposium: Dialogical Robots20 Asking about the robot’s abilities  “is that possible”

21. AAAI Spring ’05 Symposium: Dialogical Robots21 Filler  “and now um”  “ok um”

22. AAAI Spring ’05 Symposium: Dialogical Robots22 Real-time command modifications  “keep going”  “stop”  “a little more”  “change of plans”  “other direction”