MERL 1 COLLAGEN: Applying Collaborative Discourse Theory to Human-Computer Interaction Charles Rich Candace L. Sidner Neal Lesh Mitsubishi Electric Research Laboratories Cambridge, Massachusetts

MERL 2 Road Map p Philosophy and approach p Theory and implementation p Live demos p Collagen architecture p Specialized topics p Related and future work

MERL 3 Task-Oriented Human Collaboration Collaborative Interface Agent communicate interact observe * SharedPlans per Grosz, Sidner, Lochbaum, Kraus, et al. plan tree focus stack * Collagen

MERL 4 Java Middleware for COLLaborative Interface AGENts air travel planning reading and responding (w. IBM/Lotus) GUI design tool operation car navigation system operation airport landing path planning (w. MITRE) gas turbine operator training (w. USC/ISI) personal video recorder operation programmable thermostat operation (with Delft U.) multi-modal web-based form-filling COLLAGEN

MERL 5

6

7

8

9

10

MERL 11

MERL 12 Road Map p Philosophy and approach p Theory and implementation p Live demos p Collagen architecture p Specialized topics p Related and future work

MERL 13 Intentional purposes, contributes Linguistic segments, lexical items Attentional focus spaces, focus stack SharedPlan Discourse Theory (Grosz, Sidner, Kraus, Lochbaum ) Java Implementation focus stack purpose tree

MERL 14 (Grosz, 1974) E: Replace the pump and belt please. A: Ok, I found a belt in the back. A: Is that where it should be? A: [removes belt] A: It’s done. E: Now remove the pump. … E: First you have to remove the flywheel. … E: Now take the pump off the base plate. A: Already did. replace belt replace pump replace pump and belt (fixing an air compressor, E = expert, A = apprentice) Discourse Segments and Purposes

MERL 15 E: Replace the pump and belt please. A: Ok, I found a belt in the back. A: Is that where it should be? A: [removes belt] A: It’s done Focus Stack replace belt replace pump and belt Purpose Tree replace pump and belt replace pumpreplace belt Discourse State Representation current focus space (Grosz & Sidner, 1986) replace belt replace pump and belt

MERL 16 (Lochbaum, 1998) Discourse Interpretation Algorithm starts a new segment/focus space (push) ends the current segment/focus space (pop) continues (contributes to) the current segment/... (add) The current (communication or manipulation) act either: focus stack directly achieves the purpose is a step in the plan for the purpose * identifies the recipe used to achieve the purpose identifies who should perform the purpose or a step in the plan identifies a parameter of the purpose or a step in the plan An act contributes to the purpose of a segment if it: purpose tree * does not include recursive plan recognition (see later topic)

MERL 17 A Very Simple Example public top A; public manipulation B; public manipulation C; public recipe R achieves A { step B step1; step C step2; } Recipe Library (Task Model): A BC

MERL 18

MERL 19 Road Map p Philosophy and approach p Theory and implementation p Live demos p Collagen architecture p Specialized topics p Related and future work

MERL 20 Two Live Demonstrations (1) Personal Video Recorder home application intelligent assistant speech only push-to-talk using IBM Embedded ViaVoice ( illustrating the wide range of application possibilities ) (2) Gas Turbine Operator Training industrial application intelligent tutor multi-modal (speech and direct manipulation) open mic using IBM ViaVoice Pro

MERL 21 Road Map p Philosophy and approach p Theory and implementation p Live demos p Collagen architecture p Specialized topics p Related and future work

MERL 22 Spoken-Language Discourse Interpretation Recipe Library Agenda Discourse Generation segmented history utterance menu Collagen Architecture speech understanding (JSAPI) text-to-speech (JSAPI)

MERL 23 Specialized Topics Artificial discourse language Plan recognition reduces explicit communication

MERL 24 (Specialized Topic) Artificial Discourse Language (Sidner, 1994) speaker: PROPOSE(b) Believe(speaker, b) Intend(speaker, Achieve(speaker, Believe(hearer, b) hearer: ACCEPT(b) Believe(speaker, b) Believe(hearer, b) Believe(speaker, Believe(hearer, b)) Believe(hearer, Believe(speaker, b)) Believe(speaker, Believe(hearer, Believe(speaker, b)))... mutual belief (1) Formal semantics in terms of beliefs and intentions:

MERL 25 (Specialized Topic) Artificial Discourse Language (2) Translation to and from natural languages: PROPOSE(SHOULD(Do (...))) “Let’s work on .” utterance menu speech recognition natural language understanding PROPOSE(SHOULD(Do (...))) “Let's work on .” text to speech template substitution

MERL 26 X X U plan recognition (Specialized Topic) Plan Recognition Reduces Explicit Communication (Lesh, Rich & Sidner, 1999) YZ W U

MERL 27 USER:"Let's work on ."USER: "Let's work on ." AGENT:"You have 5 unread messages.”AGENT: "You have 5 unread messages." USER: "Let's read a message." USER: [select message from Chuck]USER: [select message from Chuck] USER: "Let's read message in its own window." USER: [open message]USER: [open message] USER: "Let's do something using message." USER: [visit [visit lunch......long lunch... USER: "What was I doing?"USER: "What was I doing?" AGENT: "Working on .AGENT: "Working on . Reading message from Chuck. Reading message in its own window. Reading message in its own window. Done something using message. [visit Expecting to react to message (opt)." Expecting to react to message (opt)." USER: "Let's react to message." USER: "Let's send ." USER: [start new message]USER: [start new message] USER: "Let's fill in message." USER: [address message to Dan]USER: [address message to Dan] AGENT:"Why are you sending to Dan? (a) reacting to message from Chuck (b) as a new goal " USER:"I am reacting to message from Chuck."...additional editing actions......additional editing actions... USER: "Done filling in message."USER:"Done filling in message." AGENT: "Let's send, cancel or save message." AGENT:"Let's send, cancel or save message." USER: "Please save message."USER:"Please save message." AGENT: [save message]AGENT: [save message] without plan recognition with plan recognition

MERL 28 Other Specialized Topics Clarification sub-dialogues Turn taking and mixed initiative Collaborative negotiation Near-miss plan recognition Learning task models (recipe libraries) “Engagement”

MERL 29 Related Work Using Collagen Tutoring as a kind of collaboration: Rickel, USC/ISI Ohi, Mitsubishi Electric, Japan Keyson, Delft U. of Technology Gertner, MITRE Kojima, Mitsubishi Electric, Japan Communicating with intelligent consumer products: Intelligent assistance for complex software:

MERL 30 Other Related Work multiple participant collaboration (vs. two participants) e.g., Tambe et al. other theoretical models of collaboration (vs. SharedPlan) e.g., Levesque & Cohen, Carberry application-specific collaborative dialogue systems (vs. middleware) e.g., MERIT, MIRACLE, DenK, TRIPS other interface agents (without discourse theory) e.g., Maes, and many others other agent-related middleware (without discourse management) e.g., PRS, and other BDI interpreters

MERL 31 Future Work Clarification sub-dialogues Turn taking and mixed initiative Collaborative negotiation Engagement Other Specialized Topics: