1. 1 Title A Joint Intention Based Dialog Architecture Rajah. A Subramanian University of Rochester

2. 2 Need for a Generic Dialog Engine Current spoken dialog systems  Domain based systems.  Need to explicitly code all possible rules and conditions for prompting, informing etc.  Very difficult to port from one domain to another, breaks easily and cannot be used for complex task-based scenarios. We propose an architecture that can reason in a general manner and can replace many of the rules and condition that are explicit in current dialog systems.  Handle over-answering, multi-agent interaction, and certain dynamic changes in the environment.  Build Systems that are robust and domain independent.

3. 3 Dialog Process We are concerned here in the Dialog Management Strategy. Language Understanding Dialog Management. Strategy Back-end (various) Language Generation Speech Recognition Speech Synthesis

4. 4 Outline of the talk Overview  Joint Intentions Theory  Semantics of Communicative Acts  Belief Reasoner  STAPLE interpreter Room Reservation Demo Summary, Future & Related work

5. 5 Joint Intention Theory (JIT) - I Belief, Desire, Intention (BDI) Model – Bratman/Israel/Pollack, Georgeff/Lansky … Joint Intention Theory (JIT) (Cohen, Levesque). Uses modal logic (beliefs, goals), dynamic logic of actions (sequence, concurrent actions, etc.), temporal logic (eventually, always). INTENTION – Choice with commitment. PGOAL x – Individual persistent goal of an agent. Once adopted, the agent is committed to it until achieved, impossible or irrelevant.

6. 6 Joint Intention Theory (JIT) - II PWAG x y – Joint goal between x and y.  PGOAL to achieve the goal relative to PWAG.  if already achieved, impossible or irrelevant, then PGOAL to mutually make it known to concerned agents. Teamwork: JPG x y – Two way PWAG’s/ X has a PWAG towards Y and Y towards X. Prescribes how to execute actions jointly while being in a shared mental state  Requires starting mutual belief (MB)  Requires MB about task completion, impossibility, or irrelevance  It predicts communication in the event of private beliefs

7. 7 Semantics of Communicative Acts Formal compositional semantics of commonly used speech acts using JIT constructs  REQUEST and INFORM are the basic CA’s. The goal of a request (x y a ): y eventually does the action a and also have a PWAG with respect to the requester x to do a. The goal of an inform (x y p) is to establish MB between the informee (x) and the informed (y) that x believes the informed predicate (p).  The other CA’s such as ASK, AGREE, INFORMREF etc. are built from REQUEST and INFORM. For example, ASK is an REQUEST to INFORMREF.  KNOWREF – To know the referent of the description of a required value.

8. 8 Communicative Actions (cont..) Identified speech acts that can be used to create and discharge teams Showed how to achieve mutual belief Explicated defeasible assumptions about communication STAPLE (Sanjeev Kumar) – Implements the constructs JI theory and interprets PGOAL’s, PWAG’s, Communicative Acts, etc. Can you please help me move the table? Sure REQUEST + AGREE = Joint Commitment Back

9. 9 STAPLE Architecture (Kumar 2006)

10. 10 STAPLE Optimization Original version – Java only  Not efficient for unification, that constitutes more than 80% of execution time. To optimize – combined Java and Prolog, completed as part of RPE project.  Java for threads, stacks, triggers etc.  Prolog for knowledge base, unification etc.  More than 150 times faster for single-agent and 70 times faster for multi-agents.

11. 11 Planning Relevant Questions – Snapshot of a Goal Stack in STAPLE INTEND (agent, request (agent, user, action (informref (user, agent, i (D, date (Room,D)))* PGOAL (agent, done (action (ask-ref ( agent, user, i(D, date (Room, D))* PGOAL (agent, (KNOW-REF agent, i (D,date(Room,D))) * PGOAL(agent,  Room (reserved(Room,Date,Time,Size)) PWAG(agent, user, user (done(reserved(Room, Date, Time,…)))) *The PGOAL is actually a bit more complex – the system wants to know the referent of the date that the user wants the room reservation; also the sub-goals are relative to the original goal.

12. 12 STAPLE: Multi-agent Dialog Example Room reservation domain Three Microsoft agent characters play the roles of User, System, & Database agent Peedy (SYSTEM) Robby (USER) Merlin (DATABASE AGENT)

13. 13 Multi-Agent Dialog Example - Setup Robby (USER) 1. Initial state : - Believes that it wants a room at a particular date & time 2. Actions: - Cannot perform any actions other than communicative acts Peedy (SYSTEM) 1. Initial state: - Believes that the user knows the date & time for reserving the room - Believes that the database agent can query for rooms and hold a room if necessary. 2. Actions: - Can perform action reserve room that happens to be a named action expression (i.e., a plan) Merlin (DATABASE AGENT) 1. Initial state : - No domain related beliefs. 2. Actions: - Can perform actions to query the database and to hold a room.

14. 14 Handling Over-Answering PGOAL (KNOW-REF i (D,date(Room,D))) Agent: Please say the date you want the room User: I would like it on March 20 th at Three PM Bel(user, /\ ([ March20 = i(Date, date (Room, Date))), 3PM = i(Time, time (Room, Time)))])). Bel(user, March20 = i(Date, date (Room, Date)))) Bel(user, 3PM = i(Time, time (Room, Time)))). PGOAL (KNOW-REF i (T,time(Room,T))) PGOAL (KNOW-REF i (S,size(Room,S))) PGOAL Discharged

15. 15 Multi-Agent Interaction PGOAL(system-agent, done(action (getrooms, X,E),…. System Agent USER Original JPG to reserve a room Bel(system-agent, cando(getrooms, database-agent)) … (Action Expression Decomposition) Database Agent

16. 16 Multi-Agent Interaction PGOAL(system-agent, done(action (getrooms, X,E),…. INTEND(system, action(establish_jpg([system, database], done(action(getrooms(……relative to PWAG reserveroom ….. INTEND(system, action(request(system,[system, database], action(getrooms, …… Database Agent REQUEST System Agent USER Original JPG to reserve a room

17. 17 Multi-Agent Interaction System Agent USER Original JPG to reserve a room JPG to do getrooms AGREE Database Agent REQUEST Back

18. 18 Handling Dynamic Changes System Agent USER JPG to do holdroom Database Agent Original JPG to reserve a room INTEND(database,action(holdroom,database)..Relative to original PWAG to reserve a room JPG to do holdroom becomes impossible PGOAL(database, mb([system, database], always(not(done(action(holdroom(….. INTEND(database, done(action(inform( database,system,always(not(done (action(holdroom(….. INFORM – not able to hold the room Room is TAKEN!!! Conversation between user and system to confirm the room

19. 19 Handling Dynamic Changes System Agent USER JPG to do holdroom Database Agent Original JPG to reserve a room JPG to do holdroom becomes impossibe Check if there are other rooms that are available or inform user that no rooms are available.

20. 20 Demo Back

21. 21 The Point We get automatic team and task-related communication behavior  If agent believes that somebody else knows an answer then initiates dialogue to find out that answer.  If an agent believes that somebody else can do the action that it wanted done then it establishes a team for that purpose  Dialogue about mutual belief establishment follows automatically.  No mutual observation leads to explicit communication, otherwise explicit communication is absent. The above behavior can be chained to elicit even more interesting behavior.  Example: A sub-dialogue may get started in the middle of joint action execution to find the truth value of a proposition if it is the precondition for doing a part of the jointly committed action.

22. 22 Future Work In Progress  Implicit Confirmations  Explicit Clarifications Still Mulling over…  Partial and weak beliefs  Belief update and revision

23. 23 Related Work Dialogue & Communication  Collagen: MERL – Rich, Sidner, Lesh  Artimis & Arcol: France Telecom - Sadek, Panaget, Bretier,…  TRIPS: University of Rochester – Allen, Ferguson, …  FIPA, KQML Teamwork  STEAM & Teamcore: USC – Tambe et al.  Grate: Liverpool, UK - Jennings)

24. 24 Appendix A: STAPLE Program – Multi Agent Interaction and Dynamic Changes bel(system,can_do(getrooms(Room,List), database)). bel(system,exists(D,bel(user,equals(D,i(Date,date(Room,Date)))))). plan(reserveroom,4) :- [args: [Room, [Date, Time], E1,E2], achieve_precondition: true, effects: [(roomReserved(Room,[Date,Time]),1.0)], body: { action(roominfo([date(Room, Date), time(Room, Time)]),system,E1)}]. plan(roominfo,1) :- [args: [[date(Room, Date), time(Room, Time)]], effects: [(roomsAvailable(Room, [Date, Time]),1.0)], body: { action(getrooms(Room, [Date, Time]),X, Ev1)}].

25. 25 Appendix B: Belief Reasoner (BEL self p) :- p (BEL α p) :- (BEL α □p) (BEL α  p) :- (BEL α p) (BEL α (BEL α p)) :- (BEL α p)  (BEL α p):- (BEL α  p)  (BEL α p) :- \+ (BEL α p) (BEL α  p) :- (BEL α p) (BEL α □□p):-(BEL α □p) (BEL α  p):- (BEL α  p) (BEL α p) :- (BEL α q=>p) /\ (BEL α q) (BEL α p/\q) :- (BEL α p) /\ (BEL α q) (BEL α □  (p/\q)) :- (BEL α □  p)  (BEL α □  q)

26. 26 Appendix C: Mutual Belief Reasoner (BEL self p) :- (MB self y p) (BEL self (BEL y p)) :- (MB self y p) (BMB self y p) :- (MB self y p) (BMB y self p) :- (MB self y p) (BEL self (MB self y (BEL self p))) :- (MB self y p) (BEL self (MB self y (BEL y p))) :- (MB self y p) (BEL self (MB self y p/\q)) :- (BEL self (MB self y p)) /\ (BEL self (MB self y q)) (BEL self (MB self y □  (p/\q))) :- (BEL self (MB self y □  p))  (BEL self (MB self y □  q))