1. Advanced Decision Architectures Collaborative Technology Alliance A Computational Model of Naturalistic Decision Making and the Science of Simulation Walter Warwick Amy Santamaria & many, many others

2. Advanced Decision Architectures Collaborative Technology Alliance Overview The M&S big picture The work –Birth You can’t model this; that’s not a model – Life Where’s the data –Quiet reflection The science of simulation

3. Advanced Decision Architectures Collaborative Technology Alliance The Big Picture An effort to improve human behavior representations for M&S but incorporating a better model of decision making –Better than: a “tactical” or probabilistic decision –Allows new kinds of behavior to play inside of task network models Not an exercise in theory validation –Though we’d like our work to illuminate theory

4. Advanced Decision Architectures Collaborative Technology Alliance The Birth of the RPD Widget From a descriptive model to a theoretical model: –A clash of traditions –A lot of thrashing –The emergence of a cottage industry and an M&S land grab A new decision type (“RPD”) in the Micro Saint Sharp family of task network modeling tools Widget intended to capture: –Experience-based decision making via a multiple trace model of memory and simple reinforcement routine –Recognitional decision making via similarity-based recall mechanism that draws on *every* past experience –Expectancy generation and feedback—several different versions implemented, rarely used and no clear indication that we can do anything interesting with it

5. Advanced Decision Architectures Collaborative Technology Alliance Using the Widget To specify an RPD decision type, the modeler supplies: Cues that prompt recognition (map MSS variables into “subjective,” discrete cues) Alternative courses of action (usually given by the structure of the task network) Reinforcement (seat of the pants) Set run-time properties and parameters (seat of the pants) This defines the structure of each “trace”— a individual decision making experience comprising the cue values at decision time, the action that was taken and the outcome (good or bad)

6. Advanced Decision Architectures Collaborative Technology Alliance What You Get Four applications (validation studies); two flavors: –Categorization: Brunswik Faces and Weather Prediction –Dynamic behavior: Prisoners’ Dilemma and Dynamic Stocks and Flows

7. Advanced Decision Architectures Collaborative Technology Alliance Brunswik Faces

8. Advanced Decision Architectures Collaborative Technology Alliance The Results

9. Advanced Decision Architectures Collaborative Technology Alliance Weather Prediction

10. Advanced Decision Architectures Collaborative Technology Alliance The Results Figure. Model performance on the last 50 trials of the Weather Prediction task, averaged across 30 runs, separated by pattern. Error bars are SEM. (Figure from Santamaria & Warwick, 2008.)

11. Advanced Decision Architectures Collaborative Technology Alliance Prisoners’ Dilemma CooperateDefect Cooperate (3,3)(4,0) Defect (0,4)(1,1) Player A Player B

12. Advanced Decision Architectures Collaborative Technology Alliance The Results

13. Advanced Decision Architectures Collaborative Technology Alliance Dynamic Stocks and Flows

14. Advanced Decision Architectures Collaborative Technology Alliance The Results

15. Advanced Decision Architectures Collaborative Technology Alliance Some Interesting Comparisons Categorization –Isomorphic internal representation for different tasks Dynamic Models –Very different internal representations for similar tasks In general, fits are satisfying, but not very illuminating –Model vs modeler vs task vs ???

16. Advanced Decision Architectures Collaborative Technology Alliance Developing a Science for Simulation Model comparison has roots in two traditions The AI tradition –Long tradition in AI of “tests” for general intelligence –Similarly, competition has emerged a means for establishing benchmarks of performance –In both cases, the proof is in the pudding Success is the metric of performance The Hypothetico-Deductive tradition –Theories generate predictions; if the predictions are confirmed by observation, the theory is confirmed –In this case, build a model and see if it predicts (retrodicts) actual human performance –Experimental science 101

17. Advanced Decision Architectures Collaborative Technology Alliance Conventional Wisdom AI competition + HD method = Model Comparison –Pick a task –Develop a bunch of models –See which ones make the best predictions (given some measure of goodness-of-fit) –Declare a winner!

18. Advanced Decision Architectures Collaborative Technology Alliance Familiar Concerns Concerns about fitting the data (does a good fit really confirm anything?) Concerns about simulating the task environment (have we made too many simplifying assumptions?) Concerns about models interacting with the task environment (is the model really performing the task?) Lots of valuable and important discussions here

19. Advanced Decision Architectures Collaborative Technology Alliance A Deeper Concern The real focus in a model comparison shouldn’t be on the “winner” but on understanding how the various approaches are implemented –Good predictions are a minimum requirement The relationship between theory and model is not easily assessed –Often the most difficult part of the comparison –But the most important part Is there anything better than a qualitative assessment of reasonableness?

20. Advanced Decision Architectures Collaborative Technology Alliance Toward a Science of Model Comparison A general problem here is that the history of computer simulation as experiment is not yet well understood –Cognitive models are just one application –Working at a strange intersection of theory and engineering (cf. “Computer Science as Empirical Inquiry”) Absent a theory of the simulation as experiment, the best we can do is look at current and, we hope, best practices