1. Dialogue Modelling Milica Gašić Dialogue Systems Group

2. Why are current methods poor?

3. Dialogue as a Partially Observable Markov Decision Process (POMDP) atat stst s t+1 rtrt otot o t+1 State is unobservable and depends on the previous state and action: P(s t+1 |s t, a t ) – the transition probability State depends on a noisy observation P(s t |o t ) -- the observation probability Action selection (policy) is based on the distribution over all states at every time step t – belief state b(s t )

4. How to track belief state?

5. Belief propagation Probabilities conditional on the observations Interested in the marginal probabilities p(x|D), D={D a,D b } D a D b x

6. Belief propagation D a D b x D c D d Split D b further into D c and D d

7. Belief propagation D a D c a c D b b

8. Belief propagation D a D b a b

9. How to track belief state? atat stst s t+1 rtrt otot o t+1

10. Belief state tracking Requires summation over every dialogue state!!! atat stst s t+1 rtrt otot o t+1 Requires summation over all possible states at every dialogue turn – intractable!!!

11. Challenges in POMDP dialogue modelling How to define the state space? How to tractably maintain the belief state? How to define transition and observation probabilities?

12. How to represent dialogue state? Needs to know what happened before – the dialogue history Markov property Needs to know what user wants – the user goal Task oriented dialogue Needs to know what user says – the user act Robust to errors

13. Dialogue state factorisation Decompose the sate into conditionally independent elements: user goal user action stst gtgt utut dtdt dialogue history atat rtrt otot o t+1 g t+1 u t+1 d t+1

14. Belief update gtgt utut dtdt atat rtrt otot o t+1 g t+1 u t+1 d t+1 Requires summation over all possible goals– intractable!!! Requires summation over all possible histories and user actions– intractable!!!

15. Dialogue models for real-world dialogue system Hidden Information State (HIS) system Bayesian Update of Dialogue State (BUDS) system

16. Hidden Information State system Real world dialogue system based on POMDP Takes an N-best input of user utterances Maintains a distribution over most probable dialogue states in real time

17. Hidden Information State system – dialogue acts inform ( pricerange = cheap, area = centre) dialogue act typesemantics slots and values Is there um maybe a cheap place in the centre of town please? inform request confirm … type=restaurant food=Chinese …

18. Hidden Information State system -- ontology typerestaurantareanorthsouthfoodChineseIndianhotelstarts

19. Hidden Information State system – belief update Only the user acts from the N-best Iist Dialogue histories take a small number of values Goals are grouped into partitions All probabilities are handcrafted

20. Dialogue history in the HIS system Dialogue history ideally represent everything that happened History states: system informed, user informed, user requested, system requested for each concept in the dialogue either 1 or 0 and defined by a finite state automaton

21. HIS partitions Represent group of (most probable) goals Dynamically built during the dialogue is set to a high value if g t+1 is in line with g t and a t, otherwise a small value

22. HIS partitions --example System: How may I help you? request(task) User: I’d like a restaurant in the centre. inform(entity=venue, type=restaurant, area=centre) entity ! venue entity venue type area !restaurant entity venue type area restaurant !central entity venue type area !restaurant central entity venue type area restaurant central entity=venue type=restaurant area=central

23. Pruning 23 entity ! venue entity venue type area !restaurant entity venue type area restaurant !central entity venue type area !restaurant central entity venue type area restaurant central entity=venue 0.9 type=restaurant 0.2 area=central 0.5

24. Hidden Information State systems Any limitations?

25. Bayesian Update of Dialogue State system Further decomposes the dialogue state Tractable belief state update Learning of the shape of distribution

26. Bayesian network model for dialogue gtgt utut dtdt atat rtrt otot o t+1 g t+1 u t+1 d t+1 g t food d t food u t food g t area d t area u t area g t+1 food d t+1 food u t+1 food g t+1 area d t+1 area u t+1 food

27. Belief tracking For each node x Start on one side, and keep getting p(x|D a ) Then start on the other ends and keep getting p(D b |x) To get a marginal simply multiply these

28. Bayesian network model for dialogue atat rtrt otot o t+1 g t food d t food u t food g t area d t area u t area g t+1 food d t+1 food u t+1 food g t+1 area d t+1 area u t+1 food θ

29. Training policy using different parameters Policy trained using reinforcement learning (explained in next lecture) Examined on different errors in the user input Average reward

30. Summary Essential ingredients to include in dialogue state Belief state maintaining Dialogue modelling for real world problems Learning of the shapes of probability distributions