1. MERL 1 COLLAGEN: Middleware for Building Mixed-Initiative Problem Solving Assistants ( Neal Lesh, Andy Garland, Chris Lee, David McDonald, Egon Pasztor, Chris Maloof, Luke Zettlemoyer, Jim Davies, Myrosia Dzikovska, Steve Wolfman, Jacob Eisenstein, Allison Bruce ) Charles Rich Candace L. Sidner Mitsubishi Electric Research Laboratories Cambridge, MA

2. MERL 2 Outline of the Talk Introduction Demo: DiamondHelp Some Theory Some Architecture Some More Technical Details Related Work

3. MERL 3 Mixed-Initiative and Collaboration * Collaboration: A process in which two or more participants coordinate their actions toward achieving shared goals. [Grosz & Sidner] CollaborationMixed-Initiative implies i.e., all collaborative systems are mixed-initiative Mixed-Initiative strongly suggests Collaboration i.e., most interesting mixed-initiative systems are collaborative Mixed Initiative: …efficient, natural interleaving of contributions by users and automated services… [Horvitz]

4. MERL 4 Collaboration covers a wide spectrum of interactions depending, among other factors, on: - the relative knowledge of the participants - which participant predominantly has the initiative - the primary goal of the collaboration e.g., tutoring versus assistance usually involves some form of communication (discourse) between the participants, e.g., in natural language.

5. MERL 5 Demo

6. MERL 6 Outline of the Talk Introduction Demo: DiamondHelp Some Theory Some Architecture Some More Technical Details Related Work

7. MERL 7 SharedPlan Collaborative Discourse Theory (Grosz, Sidner, Kraus, Lochbaum 1974-1998) Attentional focus spaces, focus stack Intentional goals, recipes, plans Linguistic segments, lexical items

8. MERL 8 (Grosz, 1974) E: Replace the pump and belt please. A: Ok, I found a belt in the back. A: Is that where it should be? A: [removes belt] A: It’s done. E: Now remove the pump. … E: First you have to remove the flywheel. … E: Now take the pump off the base plate. A: Already did. replace belt replace pump replace pump and belt (fixing an air compressor, E = expert, A = apprentice) Discourse Segments and Purposes

9. MERL 9 E: Replace the pump and belt please. A: Ok, I found a belt in the back. A: Is that where it should be? A: [removes belt] A: It’s done Focus Stack replace belt replace pump and belt Plan Tree replace pump and belt replace pumpreplace belt SharedPlan Discourse State Model current focus space (Grosz & Sidner, 1986) replace belt replace pump and belt

10. MERL 10 A B C d [ user ]e [ user ] f [ agent ]g [ user ] Plan Tree:Focus Stack: A B SharedPlan Discourse Interpretation Algorithm 2#d [ user ] live 1. User performs e. 2. User performs d. 3. Agent performs f. 4. Agent says “Please perform g.” 1#e [ user ] live 3#f [ agent ] 4#Propose.Should [ agent, g[user] ] (Lochbaum, 1998) Updating the discourse state in response to new discourse events (communications or manipulations) g

11. MERL 11 User says "What next?" Agent says "What do you want to do?" [Choosing the fabric and stain.] User says "Choose the fabric and stain." [Done choosing the fabric.] [Done successfully navigating.] [Done user successfully popping up the fabric load selection display.] Agent says "Please press the Fabric Load picture to pop up the fabric choices." Agent points to where you press the Fabric Load picture to pop up the fabric choices. User pops up the fabric load selection display. User closes the current pop-up window (by pressing OK in the window corner). User says "What next?" [Choosing the stain.] [Done successfully navigating.] [Done user successfully popping up the stain selection display.] Agent says "Please press the Stain picture to pop up the stain choices." Agent points to where you press the Stain picture to pop up the stain choices. User pops up the stain selection display. [Next expecting optionally to select a stain.] [Next expecting to close the current pop-up window (by pressing OK in the window corner).] [Expecting optionally to adjust detailed settings.] [Expecting optionally to run the selected cycle.]

12. MERL 12 Discourse Theory vs. Problem-Solving Theory Even though it includes an intentional (plan tree) component, SharedPlan discourse theory is not a complete problem-solving theory: For example, it does not tell you how to build new recipes (for that, you might use, e..g., first-principles planning or case-based reasoning) If a problem solver does not collaborate, then it does not need a discourse model! However, a mixed-initiative problem solving assistant needs both a discourse model and a problem-solving model (e.g., BDI). problem solving theory discourse theory

13. MERL 13 Discourse Theory vs. Problem-Solving Theory The discourse model constrains the problem solving model: For example, the discourse model constrains which subproblem to work on next based on the focus of attention in the collaboration. This modularity is possible because SharedPlan discourse theory captures structure that is independent of the domain and the problem solving model, i.e., structure that is fundamentally about the collaboration process itself. The discourse model also provides structure needed for linguistic processing, such as reference resolution (via focus spaces). problem solving model discourse model beliefs intentions desires first-principles planning discourse interpretation plan recognition

14. MERL 14 Outline of the Talk Introduction Demo: DiamondHelp Some Theory Some Architecture Some More Technical Details Related Work

15. MERL 15 Theoretical Orientation: Applying SharedPlan collaborative discourse theory to improve human-computer interaction. Practical Goal: Building collaborative agents (mixed-initiative problem solving assistants) for a wide range of applications with a maximum degree of software reuse. The COLLAGEN Project MERL: Charles Rich Candace Sidner USC/ISI: Jeff Rickel MITRE: Abigail Gertner TU Delft: David Keyson, Elyon Dekoven MIT Media Lab: Justine Cassell, Tim Bickmore

16. MERL 16 Task-Oriented Human Collaboration Collaborative Agent communicate interact observe plan tree focus stack COLLAGEN

17. Software Reuse: Prototypes Built with Collagen MERL MERL/MELCOMITRE USC/ISI MERLLOTUS/IBM MERL/MELCO MERL

18. 18 Discourse State Respond ** Interpret * Choose user event agent event Collagen Architecture Generate agenda * Lesh, Rich, Sidner (1999-2001) -- plan recognition Grosz, Sidner, Kraus, Lochbaum (1974-1998) -- discourse interpretation ** Rich, Lesh, Rickel, Garland (2002) -- plugins Task Model (Recipes) Implementation of SharedPlan Discourse Theory Weak Problem-Solving Model

19. MERL 19 Outline of the Talk Introduction Demo: DiamondHelp Some Theory Some Architecture Some More Technical Details Related Work

20. MERL 20 EngageEngine StartGeneratorOpenBleedValve Purge OpenFuelValve Engage TurnOn Fragment of Gas Turbine Task Model Non-primitive act Primitive act Recipe step Ordering constraint Other constraints (not shown): * - act pre/postconditions - recipe applicability conditions - equality between parameters - other logical predicates ( Hierarchical Task Network ) *Truth maintenance system

21. MERL 21 X X U plan recognition Plan Recognition Reduces Explicit Communication (Lesh, Rich & Sidner, 1999) YZ W U

22. MERL 22 USER:"Let's work on email."USER: "Let's work on email." AGENT:"You have 5 unread messages.”AGENT: "You have 5 unread messages." USER: "Let's read a message." USER: [select message from Chuck]USER: [select message from Chuck] USER: "Let's read message in its own window." USER: [open message]USER: [open message] USER: "Let's do something using message." USER: [visit http://foo.bar.com]USER: [visit http://foo.bar.com]...long lunch......long lunch... USER: "What was I doing?"USER: "What was I doing?" AGENT: "Working on email.AGENT: "Working on email. Reading message from Chuck. Reading message in its own window. Reading message in its own window. Done something using message. [visit http://foo.bar.com] Expecting to react to message (opt)." Expecting to react to message (opt)." USER: "Let's react to message." USER: "Let's send email." USER: [start new message]USER: [start new message] USER: "Let's fill in message." USER: [address message to Dan]USER: [address message to Dan] AGENT:"Why are you sending email to Dan? (a) reacting to message from Chuck (b) as a new goal " USER:"I am reacting to message from Chuck."...additional editing actions......additional editing actions... USER: "Done filling in message."USER:"Done filling in message." AGENT: "Let's send, cancel or save message." AGENT:"Let's send, cancel or save message." USER: "Please save message."USER:"Please save message." AGENT: [save message]AGENT: [save message] without plan recognition with plan recognition

23. MERL 23 Discourse State Respond Interpret Choose user event agent event Natural Language Processing Generate agenda SLG SLU Task Model (Recipes)

24. MERL 24 Artificial Discourse Language (Sidner, 1994) speaker: PROPOSE(b) Believe(speaker, b) Intend(speaker, Achieve(speaker, Believe(hearer, b) hearer: ACCEPT(b) Believe(speaker, b) Believe(hearer, b) Believe(speaker, Believe(hearer, b)) Believe(hearer, Believe(speaker, b)) Believe(speaker, Believe(hearer, Believe(speaker, b)))... mutual belief (1) Formal semantics in terms of beliefs and intentions:

25. MERL 25 Artificial Discourse Language (2) Translation to and from natural languages: PROPOSE(SHOULD(DoEmail(...))) “Let’s work on email.” utterance menu speech recognition natural language understanding PROPOSE(SHOULD(DoEmail(...))) “Let's work on email.” text to speech template substitution * * also using SPUD (Stone, 2003) Devault, Rich, Sidner 2004

26. MERL 26 Related Work (vs. Collagen) multiple participant collaboration (vs. two participants) e.g., Tambe et al. other theoretical models of collaboration (vs. SharedPlan) e.g., Levesque & Cohen, Carberry application-specific collaborative dialogue systems (vs. middleware) e.g., MERIT, MIRACLE, DenK, TRIPS other interface agents (without discourse model) e.g., Maes, and many others other agent-related middleware (without discourse model) e.g., PRS, and other BDI interpreters * * Recently evolving into CPS middleware

27. MERL 27 Conclusions problem solving theory discourse theory COLLAGEN Questions? (Free research licenses available)