1. Compressing Mental Model Spaces and Modeling Human Strategic Intent

2. Prashant Doshi University of Georgia, USA

3. http://thinc.cs.uga.edu

4. Yingke Chen Doctoral student Yifeng Zeng Reader, Teesside Univ. Previously: Assoc Prof., Aalborg Univ. Hua Mao Doctoral student Muthu Chandrasekaran Doctoral student

5. What is a mental behavioral model?

6. How large is the behavioral model space?

7. General definition A mapping from the agent’s history of observations to its actions

8. How large is the behavioral model space? 2H  (Aj)2H  (Aj) Uncountably infinite

9. How large is the behavioral model space? Let’s assume computable models Countable A very large portion of the model space is not computable!

10. Daniel Dennett Philosopher and Cognitive Scientist Intentional stance Ascribe beliefs, preferences and intent to explain others’ actions (analogous to theory of mind - ToM)

11. Organize the mental models  Intentional models  Subintentional models

12. Organize the mental models  Intentional models E.g., POMDP  =  b j, A j, T j,  j, O j, R j, OC j  BDI, ToM  Subintentional models Frame (may give rise to recursive modeling)

13. Organize the mental models  Intentional models E.g., POMDP  =  b j, A j, T j,  j, O j, R j, OC j  BDI, ToM  Subintentional models E.g.,  (A j ), finite state controller, plan Frame

14. Finite model space grows as the interaction progresses

15. Growth in the model space Other agent may receive any one of |  j | observations |M j |  |M j ||  j |  |M j ||  j | 2 ...  |M j ||  j | t 012t

16. Growth in the model space Exponential

17. Absolute continuity condition (ACC)

18. ACC 1.Subjective distribution over histories 2.True distribution over histories

19. ACC is a sufficient and necessary condition for Bayesian update of belief over models

20. How do we satisfy ACC? Cautious beliefs (full prior) Grain of truth assumption Prior with a grain of truth is sufficient but not necessary

21. General model space is large and grows exponentially as the interaction progresses

22. It would be great if we can compress this space!  No loss in value to the modeler  Flexible loss in value for greater compression Lossless Lossy

23. Expansive usefulness of model space compression to many areas: 1.Sequential decision making (dt-planning) in multiagent settings 2.Bayesian plan recognition 3.Games of imperfect information

24. 1. Sequential decision making in multiagent settings Interactive POMDP framework (Gmytrasiewicz&Doshi05) Include models of the other agent in the state space Update beliefs over the physical state and models

25. General and domain-independent approach for compression Establish equivalence relations that partition the model space and retain representative models from each equivalence class

26. Approach #1: Behavioral equivalence (Rathanasabapathy et al.06,Pynadath&Marsella07) Intentional models whose complete solutions are identical are considered equivalent

27. Approach #1: Behavioral equivalence Behaviorally minimal set of models

28. Lossless Works when intentional models have differing frames Approach #1: Behavioral equivalence

29. Multiagent tiger Approach #1: Behavioral equivalence Impact on dt-planning in multiagent settings Multiagent tiger Multiagent MM

30. Utilize model solutions (policy trees) for mitigating model growth Approach #1: Behavioral equivalence Model reps that are not BE may become BE next step onwards Preemptively identify such models and do not update all of them

31. Redefine BE Approach #2:  -Behavioral equivalence (Zeng et al.11,12)

32. Intentional models whose partial depth-d solutions are identical and vectors of updated beliefs at the leaves of the partial trees are identical are considered equivalent Approach #2: Revisit BE (Zeng et al.11,12) Sufficient but not necessary Lossless if frames are identical

33. Approach #2: ( ,d)-Behavioral equivalence Two models are ( ,d)-BE if their partial depth-d solutions are identical and vectors of updated beliefs at the leaves of the partial trees differ by  Models are (0.33,1)-BE Lossy

34. Approach #2:  -Behavioral equivalence Lemma (Boyen&Koller98): KL divergence between two distributions in a discrete Markov stochastic process reduces or remains the same after a transition, with the mixing rate acting as a discount factor Mixing rate represents the minimal amount by which the posterior distributions agree with each other after one transition Property of a problem and may be pre-computed

35. Given the mixing rate and a bound, , on the divergence between two belief vectors, lemma allows computing the depth, d, at which the bound is reached Approach #2:  -Behavioral equivalence Compare two solutions up to depth d for equality

36. Discount factor  F = 0.5 Multiagent Concert Approach #2:  -Behavioral equivalence Impact on dt-planning in multiagent settings Multiagent Concert On a UAV reconnaissance problem in a 5x5 grid, allows the solution to scale to a 10 step look ahead in 20 minutes

37. What is the value of d when some problems exhibit  F with a value of 0 or 1? Approach #2:  -Behavioral equivalence  F =1 implies that the KL divergence is 0 after one step: Set d = 1  F =0 implies that the KL divergence does not reduce: Arbitrarily set d to the horizon

38. Intentional or subintentional models whose predictions at time step t (action distributions) are identical are considered equivalent at t Approach #3: Action equivalence (Zeng et al.09,12)

39. Approach #3: Action equivalence

40. Lossy Works when intentional models have differing frames Approach #3: Action equivalence

41. Impact on dt-planning in multiagent settings Multiagent tiger AE bounds the model space at each time step to the number of distinct actions

42. Intentional or subintentional models whose predictions at time step t influence the subject agent’s plan identically are considered equivalent at t Regardless of whether the other agent opened the left or right door, the tiger resets thereby affecting the agent’s plan identically Approach #4: Influence equivalence (related to Witwicki&Durfee11)

43. Influence may be measured as the change in the subject agent’s belief due to the action Approach #4: Influence equivalence Group more models at time step t compared to AE Lossy

44. Compression due to approximate equivalence may violate ACC Regain ACC by appending a covering model to the compressed set of representatives

45. Open questions

46. N > 2 agents Under what conditions could equivalent models belonging to different agents be grouped together into an equivalence class?

47. Can we avoid solving models by using heuristics for identifying approximately equivalent models?

48. Modeling Strategic Human Intent

49. Yifeng Zeng Reader, Teesside Univ. Previously: Assoc Prof., Aalborg Univ. Yingke Chen Doctoral student Hua Mao Doctoral student Muthu Chandrasekaran Doctoral student Xia Qu Doctoral student Roi Ceren Doctoral student Matthew Meisel Doctoral student Adam Goodie Professor of Psychology, UGA

50. Computational modeling of human recursive thinking in sequential games Computational modeling of probability judgment in stochastic games

51. Human strategic reasoning is generally hobbled by low levels of recursive thinking (Stahl&Wilson95,Hedden&Zhang02,Camerer et al.04,Ficici&Pfeffer08) (I think what you think that I think...)

52. You are Player I and II is human. Will you move or stay? Move Stay Payoff for I: Payoff for II: 3131 1313 2424 4242 IIII Player to move:

53. Less than 40% of the sample population performed the rational action!

54. Thinking about how others think (...) is hard in general contexts

55. Move Stay Payoff for I: (Payoff for II is 1 – decimal) 0.6 0.40.20.8 IIII Player to move:

56. About 70% of the sample population performed the rational action in this simpler and strictly competitive game

57. Simplicity, competitiveness and embedding the task in intuitive representations seem to facilitate human reasoning (Flobbe et al.08, Meijering et al.11, Goodie et al.12)

58. 3-stage game Myopic opponents default to staying (level 0) while predictive opponents think about the player’s decision (level 1)

59. Can we computationally model these strategic behaviors using process models?

60. Yes! Using a parameterized Interactive POMDP framework

61. Replace I-POMDP’s normative Bayesian belief update with Bayesian learning that underweights evidence, parameterized by  Notice that the achievement score increases as more games are played indicating learning of the opponent models Learning is slow and partial

62. Replace I-POMDP’s normative expected utility maximization with quantal response model that selects actions proportional to their utilities, parameterized by Notice the presence of rationality errors in the participants’ choices (action is inconsistent with prediction) Errors appear to reduce with time

63. Underweighting evidence during learning and quantal response for choice have prior psychological support

64. Use participants’ predictions of other’s action to learn  and participants’ actions to learn

65. Use participants’ actions to learn both  and Let vary linearly

66. Insights revealed by process modeling: 1.Much evidence that participants did not make rote use of BI, instead engaged in recursive thinking 2.Rationality errors cannot be ignored when modeling human decision making and they may vary 3.Evidence that participants’ could be attributing surprising observations of others’ actions to their rationality errors

67. Open questions: 1.What is the impact on strategic thinking if action outcomes are uncertain? 2.Is there a damping effect on reasoning levels if participants need to concomitantly think ahead in time

68. Suite of general and domain-independent approaches for compressing agent model spaces based on equivalence Computational modeling of human behavioral data pertaining to strategic thinking

69. Thank you for your time

70. 2. Bayesian plan recognition under uncertainty Plan recognition literature has paid scant attention to finding general ways of reducing the set of feasible plans (Carberry, 01)

71. 3. Games of imperfect information (Bayesian games) Real-world applications often involve many player types Examples Ad hoc coordination in a spontaneous team Automated Poker player agent

72. 3. Games of imperfect information (Bayesian games) Real-world applications often involve many player types Model space compression facilitates equilibrium computation