1. Cognitive User Interfaces: An Engineering Approach Machine Intelligence Laboratory Information Engineering Division Cambridge University Engineering Department Cambridge, UK Steve Young

2. 2 ICASSP Plenary April 2009 © Steve Young Outline of Talk  Introduction: what is a cognitive user interface?  Example: a simple gesture-driven interface.  Human decision-making and planning.  Partially Observable MDPs – an intractable solution?  Scaling up: statistical spoken dialogue systems.  Conclusions and future work.

3. 3 ICASSP Plenary April 2009 © Steve Young What is a cognitive user interface?  Capable of reasoning and inference  Able to optimize communicative goals  Able to adapt to changing environments  Able to learn from experience An interface which supports intelligent, efficient and robust interaction between a human and a machine.

4. 4 ICASSP Plenary April 2009 © Steve Young Example: A Simple Gesture-Driven User Interface Swipe Scroll Forward Scroll Backward Delete Photo Swipe A photo sorter

5. 5 ICASSP Plenary April 2009 © Steve Young Interpreting the Input Backwards Delete Forwards Backwards Delete Forwards angle P(angle) Decision Boundaries

6. 6 ICASSP Plenary April 2009 © Steve Young Pattern Classification angle P(angle) G=forwardsG=deleteG=backwards Conf(G=backwards)

7. 7 ICASSP Plenary April 2009 © Steve Young Flowchart-based Decision Making Confidence ? Gesture ? backwards Move back >= Threshold Do Nothing < Threshold

8. 8 ICASSP Plenary April 2009 © Steve Young What is missing?  No modeling of uncertainty  No tracking of belief in the user’s required goal  No quantifiable objectives hence sub-optimal decision making

9. 9 ICASSP Plenary April 2009 © Steve Young Modeling Uncertainty and Inference – Bayes’ Rule Reverend Thomas Bayes (1702-1761) new belief data old belief action Bayesian Network b(s) s move back ? b’(s) s Inference via Bayes Rule

10. 10 ICASSP Plenary April 2009 © Steve Young Optimizing Decisions – Bellman’s Equation Richard E Bellman (1920-1984) Reward= ++ + + … Policy s1s1 s2s2 s T-1 sTsT a1a1 a2a2 a T-1 aTaT o1o1 o2o2 o T-1 oToT b1b1 b2b2 b T-1 bTbT Reinforcement Learning

11. 11 ICASSP Plenary April 2009 © Steve Young Optimizing the Photo-Sorter Swipe Scroll Forward Scroll Backward Delete Photo Swipe { scroll-forward, scroll-backward, delete-photo } User’s Goal (states) { go-forward, go-back, do-delete, do-nothing } System Action +1 +5 0 Rewards -20 All other: -1 Iteratively optimize policy to maximize rewards …

12. 12 ICASSP Plenary April 2009 © Steve Young Performance on the Photo-Sorting Task 10%20%30%40%50% 0% Reward Effective Error Rate Flow-charted Policy Fixed Policy and Model Adapted Policy and Model Training Point

13. 13 ICASSP Plenary April 2009 © Steve Young Is Human Decision Making Bayesian? Humans have brains so that they can move. So how do humans plan movement? ….

14. 14 ICASSP Plenary April 2009 © Steve Young A Simple Planning Task Prior Observation Kording and Wolpert (Nature, 427, 2004)

15. 15 ICASSP Plenary April 2009 © Steve Young Models for Estimating Target Location 012Lateral shift (cm) Probability Prior Posterior Observation Kording and Wolpert (Nature, 427, 2004) 0 1 0 1 2 0 1 0 1 2 0 1 0 1 2 Deviation from Target True lateral shift Prior ignoredBayesianMin Error Mapping

16. 16 ICASSP Plenary April 2009 © Steve Young Practice makes perfect

17. 17 ICASSP Plenary April 2009 © Steve Young Bayesian Model Selection in Human Vision Inventory Test “Which is more familiar?” Train “Watch these!” Orban, Fiser, Aslin, Lengyel (Proc Nat. Academy Science, 105, 2008) Not visible to subjects

18. 18 ICASSP Plenary April 2009 © Steve Young Partially Observable Markov Decision Processes  Belief represented by distributions over states and updated from observations by Bayesian inference  Objectives defined by the accumulation of rewards  Policy which maps beliefs into actions and which can be optimized by reinforcement learning Principled approach to handling uncertainty and planning Humans appear to use similar mechanisms Principled approach to handling uncertainty and planning Humans appear to use similar mechanisms So what is the problem ?

19. 19 ICASSP Plenary April 2009 © Steve Young  The state and action sets are often very large.  Real-time belief update is intractable.  The mapping is extremely complex  Exact policy optimization is intractable.  The state and action sets are often very large.  Real-time belief update is intractable.  The mapping is extremely complex  Exact policy optimization is intractable. Scaling-up Applying the POMDP framework in real world user interfaces is not straightforward:

20. 20 ICASSP Plenary April 2009 © Steve Young Spoken Dialog Systems (SDS) Database Recognizer Semantic Decoder Dialog Control Synthesizer Message Generator User WaveformsWords Dialog Acts Is that near the tower? confirm(near=tower) negate(near=castle)No, it is near the castle.

21. 21 ICASSP Plenary April 2009 © Steve Young Architecture of the Hidden Information State System Belief Update Speech Understanding Speech Generation User Dialog Policy Williams and Young (CSL 2007) Young et al (ICASSP 2007) Two key ideas:  States are grouped into equivalence classes called partitions and belief updating is applied to partitions rather than states  Belief space is mapped into a much simpler summary space for policy implementation and optimization Summary Space Heuristic Mapping b(s) s POMDP

22. 22 ICASSP Plenary April 2009 © Steve Young The HIS Belief Space Each state is composed of three factors: User GoalUser ActDialog History Young et al (CSL 2009) find(venue(hotel,area=east)) find(venue(bar,area=east)) find(venue(hotel,area=west)) …. find(venue) User Request User Informed System Informed Grounded Denied Queried Initial × × User goals are grouped into partitions HIS Belief Space Beliefs update is limited to the most likely members of this set.

23. 23 ICASSP Plenary April 2009 © Steve Young Master Summary State Mapping Master space is mapped into a reduced summary space: find(venue(hotel,area=east,near=Museum)) find(venue(bar,area=east,near=Museum)) find(venue(hotel,area=east) find(venue(hotel,area=west) find(venue(hotel)....etc P(top) P(Nxt) T12Same TPStatus THStatus TUserAct LastSA Heuristic Mapping act type Policy Greet Bold Request Tentative Request Confirm Offer Inform.... etc Greet Bold Request Tentative Request Confirm Offer Inform.... etc VQ confirm( ) confirm(area=east)

24. 24 ICASSP Plenary April 2009 © Steve Young Learning with a simulated User Learning by interaction with real users is expensive/impractical. A solution is to use a simulated user, trained on real data. User Simulator includes ASR error model Dialog Corpus Belief Update Heuristic Mapping Summary Space Dialog Policy Q-Learning Random action Schatzmann et al (Knowledge Eng Review 2006)

25. 25 ICASSP Plenary April 2009 © Steve Young HIS System Demo

26. 26 ICASSP Plenary April 2009 © Steve Young HIS Performance in Noise Success Rate (%) Error Rate (%) 051015202530354045 95 90 85 80 75 70 65 60 55 HIS MDP Hand- crafted (HDC) Simulated User

27. 27 ICASSP Plenary April 2009 © Steve Young Representing beliefs An alternative is to model beliefs directly using dynamic Bayesian nets … Beliefs in a spoken dialog system entail a large number of so-called slot variables. Eg for tourist information: Cardinality is huge and we cannot handle the full joint distribution. P(venue, location, pricerange, foodtype, music, …) In the HIS system, we threshold the joint distribution and just record the high probability values. The partitions marginalize out all the unknowns. But this is approximate, and belief update now depends on the assumption that the underlying user goal does not change. P(venue=bar, location=central, music=jazz) = 0.32 P(venue=bar, location=central, music=blues) = 0.27 P(venue=bar, location=east, music=jazz) = 0.11 etc P(venue=bar, location=central, music=jazz) = 0.32 P(venue=bar, location=central, music=blues) = 0.27 P(venue=bar, location=east, music=jazz) = 0.11 etc

28. 28 ICASSP Plenary April 2009 © Steve Young Modeling Belief with Dynamic Bayesian Networks (DBNs) Decompose state into DBN, retaining only essential conditional dependencies g type g food u type u food h type h food a u o g’ type g’ food u’ type u’ food h’ type h’ food a’ u’ o’ g u h Time tTime t+1 Eg restaurant Eg chinese Thomson et al (ICASSP, 2008)

29. 29 ICASSP Plenary April 2009 © Steve Young Factor Graph for the Full Tourist Information System Factor graphs are very large, even with minimal dependency modeling. Hence  need very efficient belief updating  need to define policies directly on full belief networks

30. 30 ICASSP Plenary April 2009 © Steve Young Bayesian Update of Dialog State (BUDS) System Thomson et al (CSL 2009) Belief update depends on message passing x1x1 xMxM …. x f sum over all combinations of variable values P(food) food Z1Z1 Z2Z2 Z3Z3 Grouping possible values into partitions greatly simplifies these summations FrIt ….

31. 31 ICASSP Plenary April 2009 © Steve Young Belief Propagation Times Network Branching Factor Time Standard LBP LBP with Grouping LBP with Grouping & Const Prob of Change

32. 32 ICASSP Plenary April 2009 © Steve Young Policy Optimization in the BUDS System Thomson et al (ICASSP, 2008) Summary space now depends on forming a simple characterization of each individual slot. Define policy as a parametric function and optimize wrt θ using Natural Actor Critic algorithm. Each action dependent basis function is separated out into slot-based components, eg 0100001000 1.0 0.0 0.0 0.8 0.2 0.0 0.6 0.4 0.0 0.4 0.4 0.2 0.3 0.3 0.4 1 st 2 nd Rest slot belief quantization action indicator function

33. 33 ICASSP Plenary April 2009 © Steve Young BUDS Performance in Noise Error Rate (%) Simulated User BUDS MDP Average Reward

34. 34 ICASSP Plenary April 2009 © Steve Young Conclusions  Future generations of intelligent systems and agents will need robust, adaptive, cognitive human-computer interfaces  Bayesian belief tracking and automatic strategy optimization provide the mathematical foundations  Human evolution seems to have come to the same conclusion  Early results are promising but research is needed a)to develop scalable solutions which can handle very large networks in real time b)to incorporate more detailed linguistic capabilities c)to understand how to integrate different modalities: speech, gesture, emotion, etc d)to understand how to migrate these approaches into industrial systems.

35. 35 ICASSP Plenary April 2009 © Steve Young Credits EU FP7 Project: Computational Learning in Adaptive Systems for Spoken Conversation Spoken Dialogue Management using Partially Observable Markov Decision Processes Past and Present Members of the CUED Dialogue Systems Group Milica Gasic, Filip Jurcicek, Simon Keizer, Fabrice Lefevre, Francois Mairesse, Jost Schatzmann, Matt Stuttle, Blaise Thomson, Karl Weilhammer, Jason Williams, Hui Ye, Kai Yu Colleagues in the CUED Information Engineering Division Bill Byrne, Mark Gales, Zoubin Ghahramani, Mate Lengyel, Daniel Wolpert, Phil Woodland