1. Spoken Dialog Management for an Astronaut’s Procedure Assistant Presented by: Dan Bohus (dbohus@cs.cmu.edu) Collaborators: Gregory Aist, RIALIST Group NASA Ames/RIACS Summer Student Research Program August 2002 S.D.M.A.P.A.

2. 10-10-02Dialog Management for an Intelligent Procedure Assistant2 RIALIST Group @ NASA Ames  Location: NASA Ames / Moffett Field / Mountain View, CA (CMU West)  People  Jim Hieronymous  John Dowding  Beth-Ann Hockey  Greg Aist  + 7 summer students  Projects  Personal Satellite Assistant  Mobile Robots  Intelligent Procedure Assistant

3. 10-10-02Dialog Management for an Intelligent Procedure Assistant3 Motivation  Crew members on International Space Station (ISS) have to perform a wide variety of procedural tasks:  ISS maintenance and repair  Operations (i.e. EVA Suit Checkout)  Setup and running scientific experiments  Medical Exams

4. 10-10-02Dialog Management for an Intelligent Procedure Assistant4 Typical setup  Paper documentation  2 or more astronauts involved: one reads, one executes the procedure

5. 10-10-02Dialog Management for an Intelligent Procedure Assistant5 Proposed alternative  Build a spoken dialog system for procedural task guidance and support  Create more crew-time!  Used also for ground training  Potential for increased safety

6. 10-10-02Dialog Management for an Intelligent Procedure Assistant6 Outline  Motivation  The Intelligent Procedure Assistant  A Dialog Management Component for IPA  Demo  Future work

7. 10-10-02Dialog Management for an Intelligent Procedure Assistant7 The Intelligent Procedure Assistant Multi-modal spoken dialog system providing crew support and training for the execution of procedural tasks  Guide astronauts through procedural tasks  Provide information access (diagrams, docs)  Act as a training tool  Facilitate task delegation  Provide support for voice annotations, intercom

8. 10-10-02Dialog Management for an Intelligent Procedure Assistant8 IPA Version 0: June 2002  RIALIST IPA Version 0  Load procedure  Next  Previous  Drill-down  Describe remaining steps  Increase volume  Decrease volume  Say again

9. 10-10-02Dialog Management for an Intelligent Procedure Assistant9 Outline  Motivation  The Intelligent Procedure Assistant  A Dialog Management Component for the IPA  Demo  Future work

10. 10-10-02Dialog Management for an Intelligent Procedure Assistant10 Speech Recognition Dialog Management Language Generation Synthesis U DSI SONLS Semanticizer (parsing) Resources/Agents Contributions: Dialog Management & Resources

11. 10-10-02Dialog Management for an Intelligent Procedure Assistant11 Nuance [recognition] Dialog Management Gemini [generation] Festival U DSI SONLS Gemini [parsing] Resources/Agents Nuance [recognition] Dialog Management Gemini [generation] Festival U DSI SONLS Gemini [parsing] Resources/Agents Audio Dialog Agent RavenClaw DM framework (CMU) Contributions: Dialog Management & Resources 1 2 3 CORE/ Conversational Skills Dialog Task Specification The actual domain task (IPA domain) Handles dialog flow, initiative, focus shifts, universal dialog mechanisms

12. 10-10-02Dialog Management for an Intelligent Procedure Assistant12 1. Resources  Procedures represented in XML (DTD, XSLT)  Title, Description.  Steps: Basic Steps, Conditional Steps, Messages, Images  Messages: Notes, Cautions  Images: Sub-images  Water Sampling Procedure  28 steps, 150 sub-steps, 18 conditional steps, 1 image with 5 sub-images  HTML version (using XSLT) HTML version

13. 10-10-02Dialog Management for an Intelligent Procedure Assistant13 2. Dialog Task Development  In RavenClaw, a dialog task is implemented as a hierarchy of dialog agents (C++ classes) IPA Welcome LoadProcedureDo Procedures DoProcedure1 IntroClosing DoProcedure2 Step1 Step28Image1 SubStep1 SubStep4 Help Suspend/ Resume

14. 10-10-02Dialog Management for an Intelligent Procedure Assistant14 2. Dialog Task Development (cont)  52 dialog task agents (types) were implemented:  Loading procedures by name  Guidance  Step-by-step execution  Drill-down  Branching  Presenting notes / cautions / images  Information access  Talking on/about images, objects  Navigational behavior  History / Trace through procedure  Next, Previous step/substep  Start over

15. 10-10-02Dialog Management for an Intelligent Procedure Assistant15 3. Conversational Skills Enhancements  Timeouts  System reacts after a timeout period  User Barge-in  Requests for repeat  Requests for help  generic “help!”  “what can I say ?”  Context establishment  “where are we ?”

16. 10-10-02Dialog Management for an Intelligent Procedure Assistant16 Outline  Motivation  The Intelligent Procedure Assistant  A Dialog Management Component for IPA  Demo  Future work

17. 10-10-02Dialog Management for an Intelligent Procedure Assistant17 Demo Guidance Volume control Information Access

18. 10-10-02Dialog Management for an Intelligent Procedure Assistant18 Outline  Motivation  The Intelligent Procedure Assistant  A Dialog Management Component for IPA  Demo  Future work  IPA System (RIALIST Group)  Advances in dialog management

19. 10-10-02Dialog Management for an Intelligent Procedure Assistant19 Future work for IPA system (RIALIST Group)  IPA system functionality goals  Guide astronauts through procedural tasks  Provide information access  Act as a training tool  Facilitate task delegation  Provide support for voice annotations, intercom  Iterative improvement process  Other features, including those suggested by JSC training personnel and astronauts

20. 10-10-02Dialog Management for an Intelligent Procedure Assistant20 Timeline …  Reliable prototype by the end of September  Guidance/Information Access  First fieldable system by March 2003

21. 10-10-02Dialog Management for an Intelligent Procedure Assistant21 Summer @ NASA is cool !

22. 10-10-02Dialog Management for an Intelligent Procedure Assistant22 Being back is also cool !  2 new systems to be developed based on the Communicator SDS setup / RavenClaw DM  RoomLine system  Provide conference room reservation services within SCS  Ridy Lie  BusLine system  Provide access to bus line information  Project “Let’s go!” (Maxine/Alan)?  Brian Langner, Antoine Raux  Hope you can call them sometime before Christmas !!!