1. CogTool A tool for interface design and ACT-R research Bonnie E. John HCI Institute Carnegie Mellon University

2. CogTool is an open source tool where you can describe an environment in a storyboard…

4. …and demonstrate a task

5. CogTool automatically creates an ACT-R model of a skilled person doing this task and produces predictions of task execution time.* * Based on Card, Moran and Newell’s Keystroke-Level Model (1980)

6. And you can look under thehood to see what ACT-R is doing.

7. What is prediction of skilled task execution time good for? Some examples of similar analyses PERCs “time travel” evaluation at IBM – DARPA set requirement to show 10x productivity improvement over 2002, which we can credibly demonstrate with CogTool Saved NYNEX from making a $160 million workstation purchase that would have COST them $2 million/year in operating costs IRS procurement of new IT system turned on “value” calculated, in part, using this type of analysis (IBM lost $700 million contract to AT&T’s $1.4 billion) NextGen airspace will have economic consequences of time to execute cockpit tasks Carlsbad Police reduced injuries and loss of life in their in-vehicle information systems. – SAE Recommended Practice J2365, Calculation of the Time to Complete In-Vehicle Navigation and Route Guidance Task.

8. Why should researchers at the ACT-R workshop care? If you are not in academia – Your organization may builds systems and this could be directly useful for evaluating them If you are in academia – Consulting this may be a quicker way to evaluate new systems than directly coding ACT-R – Teaching Human Factors or User Interface Design classes and this is one technique you could teach your students Psychology class, part of which may be applied Want a gentle introduction to cognitive modeling to get students excited – Research Rapid environment construction “Rapid Theory Prototyping”

9. The variability between novice modelers has been reduced by 70% This makes modeling the least variable of all usability techniques!!! First a word about teaching with CogTool KLMs “By-Hand” v. KLMs with CogTool (BRIMS 2010) By-Hand KLM: Average CV=22% CogTool KLM: Average CV=7%

10. Rapid Environment Construction Use CogTool’s storyboarding to construct an environment for your ACT-R model Export the ACT-R code CogTool creates Put in your own ACT-R model

11. Rapid Environment Construction Use CogTool’s storyboarding to construct an environment for your ACT-R model Export the ACT-R code CogTool creates Put in your own ACT-R model The interface for doing this isn’t as easy as I would like The environment isn’t yet a true ACT-R device model Anybody who would like to help and contribute to our open source code, please contact me

12. Rapid Theory Prototyping Or how I explored 7 theories before breakfast

13. Rapid UI Prototyping UI Prototype Prototypers = UI designers Far easier, quicker to build than fully- functional UI Limited to a few specific tasks Need only be “good enough” to test user behavior with a proposed UI “cheating” (e.g., wizard-of-oz) Sufficient to suggest what’s important enough to begin implementation and what should be given more thought Tool for thought and communication

14. Rapid UI Prototyping Rapid Theory Prototyping UI Prototype Prototypers = UI designers Far easier, quicker to build than fully- functional UI Limited to a few specific tasks Need only be “good enough” to test user behavior with a proposed UI “cheating” (e.g., wizard-of-oz) Sufficient to suggest what’s important enough to begin implementation and what should be given more thought Tool for thought and communication Theory Prototype Prototypers = Theory developers Far easier, quicker to build than fully- functional theory Limited to a few specific tasks Need only be “good enough” to test theory’s behavior against human data “cheating” Sufficient to suggest what’s important enough to begin implementation and what should be given more thought Tool for thought and communication

15. General approach to rapid theory prototyping 1.Start with the simplest, most generic model of a theory and ask it to do a task 2.When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledge 3.Repair the the model through rapid theory prototyping Repeat steps 2-3 until the model is failing only by chance 4.Analyze all the failures and decide: Whether prototyped theory or mechanism matches human performance If so, prioritize effort to improve the original simple theory to validate and then move into the tool for design

16. Example of an aviation task 1.Start with the simplest, most generic model of a theory and ask it to do a task Simplest, most generic model of a theory for exploring an aviation device: Information Foraging Theory (Pirolli & Card, 1999) Augmented with the Minimal Model of Visual Search (Halverson & Hornof, 2007) Using a general knowledge corpus Embodied in CogTool-Explorer (Teo & John, 2008) A task: A three step procedure to set the approach reference speed and flap angle, using the CDU in a 777.

17. 17 Example Run of CogTool-Explorer

18. 7 theory prototypes in 4 hrs The final one approaches human behavior (HFES09) P’type-1P’type-2P’type-3P’type-4 P’type-5P’type-6P’type-7 Change knowledge (aviation vocabulary) Elaboration + How it works knowledge Baseline (simplest, most generic)

19. Different prototypes target improvement in different steps P’type-1P’type-2P’type-3P’type-4 P’type-5P’type-6P’type-7

20. Example: Step 1 P’type-1P’type-2P’type-3P’type-4 P’type-5P’type-6P’type-7

21. Step 1 Given the goal: “select landing flap and reference air speed for a approach” Hit the INIT REF button

22. Example: Step 1 P’type-1P’type-2P’type-3P’type-4 P’type-5P’type-6P’type-7 Change knowledge (aviation vocabulary) Elaboration + How it works knowledge

23. Example: Step 1, Theory Prototype 4 P’type-1P’type-2P’type-3P’type-4 P’type-5P’type-6P’type-7 Hierarchical visual regions & recovery

24. Theory Prototype 4: Hierarchical visual regions + recovery We “cheated” by prototyping this theory in the CogTool storyboard, not the ACT-R code – CogTool runs ACT-R models on a UI description called a storyboard. States, which contain widgets (e.g., buttons), and transitions between states represent human actions on widgets (e.g., pressing a button) Flat representation of widgets, so visual search and information foraging considers them equally – Cheat: Prototype visual regions = large “buttons” on the regions When the correct region is chosen, it transitions to a state with only the buttons in that region When an incorrect region is chosen, it transitions to a state with only the regions that have not been chosen – 1 hour change

25. Original storyboard for this task: Each state has the CDU’s 69 buttons How we “cheated” by prototyping this theory in the CogTool storyboard: – Prototyping visual regions = large “buttons” on the regions When the correct region is chosen, it transitions to a frame with only the buttons in that region When an incorrect region is chosen, it transitions to a frame with only the regions that have not been chosen – 1 hour change

26. Theory Prototype 4: Hierarchical visual regions

27. Theory Prototype 4: Prototyped as buttons in storyboard

28. Theory Prototype 4: Correct region buttons in region INIT REF RTE DEP ARR ALTN VNAV FIX LEGS HOLD FMC COMM PROG MENU NAV RAD

29. Theory Prototype 4: Incorrect region other regions INIT REF RTE DEP ARR ALTN VNAV FIX LEGS HOLD FMC COMM PROG MENU NAV RAD No green = no cycling

30. Theory Prototype 4: Tremendous improvement on 1st step P’type-1P’type-2P’type-3P’type-4 P’type-5P’type-6P’type-7 Tremendous improvement

31. Theory Prototype 4: But still insufficient to complete task P’type-1P’type-2P’type-3P’type-4 P’type-5P’type-6P’type-7 Tremendous improvement Not sufficient to do a multi-step task

32. General approach to rapid theory prototyping 1.Start with the simplest, most generic model of a theory and ask it to do a task Flat visual search 2.When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledge Hierarchical visual regions + recovery 3.Give the model this mechanism or knowledge through “rapid theory prototyping” Quick change to CogTool’s storyboard, not ACT-R code Repeat steps 2-3 until the model is failing only by chance 7 iterations, described in the paper 4.Analyze all the failures and decide: Whether prototyped theory matches human performance If so, prioritize effort to improve the underlying theory High priority: Hierarchical visual regions + recovery

33. General approach to rapid theory prototyping 1.Start with the simplest, most generic model of a theory and ask it to do a task Flat visual search 2.When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledge Hierarchical visual regions + recovery 3.Give the model this mechanism or knowledge through “rapid theory prototyping” Quick change to CogTool’s storyboard, not ACT-R code Repeat steps 2-3 until the model is failing only by chance 7 iterations, described in the paper 4.Analyze all the failures and decide: Whether prototyped theory matches human performance If so, prioritize effort to improve the underlying theory

34. General approach to rapid theory prototyping 1.Start with the simplest, most generic model of a theory and ask it to do a task Flat visual search 2.When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledge Hierarchical visual regions + recovery 3.Give the model this mechanism or knowledge through “rapid theory prototyping” Quick change to CogTool’s storyboard, not ACT-R code Repeat steps 2-3 until the model is failing only by chance 7 iterations, described in the paper 4.Analyze all the failures and decide: Whether prototyped theory matches human performance If so, prioritize effort to improve the underlying theory

35. General approach to rapid theory prototyping 1.Start with the simplest, most generic model of a theory and ask it to do a task Flat visual search 2.When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledge Hierarchical visual regions + recovery 3.Give the model this mechanism or knowledge through “rapid theory prototyping” Quick change to CogTool’s storyboard, not ACT-R code Repeat steps 2-3 until the model is failing only by chance 7 iterations, described in the paper 4.Analyze all the failures and decide: Whether prototyped theory matches human performance If so, prioritize effort to improve the underlying theory

36. General approach to rapid theory prototyping 1.Start with the simplest, most generic model of a theory and ask it to do a task Flat visual search 2.When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledge Hierarchical visual regions + recovery 3.Give the model this mechanism or knowledge through “rapid theory prototyping” Quick change to CogTool’s storyboard, not ACT-R code Repeat steps 2-3 until the model is failing only by chance 7 iterations, described in the HFES 2009 paper 4.Analyze all the failures and decide: Whether prototyped theory matches human performance If so, prioritize effort to improve the underlying theory

37. General approach to rapid theory prototyping 1.Start with the simplest, most generic model of a theory and ask it to do a task Flat visual search 2.When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledge Hierarchical visual regions + recovery 3.Give the model this mechanism or knowledge through “rapid theory prototyping” Quick change to CogTool’s storyboard, not ACT-R code Repeat steps 2-3 until the model is failing only by chance 7 iterations, described in the paper 4.Analyze all the failures and decide: Whether prototyped theory matches human performance If so, prioritize effort to improve the underlying theory High priority: Hierarchical visual regions + recovery See Leonghwee Teo’s talk on Sunday

38. General approach to rapid theory prototyping 1.Start with the simplest, most generic model of a theory and ask it to do a task Flat visual search 2.When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledge Hierarchical visual regions + recovery 3.Give the model this mechanism or knowledge through “rapid theory prototyping” Quick change to CogTool’s storyboard, not ACT-R code Repeat steps 2-3 until the model is failing only by chance 7 iterations, described in the paper 4.Analyze all the failures and decide: Whether prototyped theory matches human performance If so, prioritize effort to improve the underlying theory High priority: Hierarchical visual regions + recovery See Leonghwee Teo’s talk on Sunday

39. General approach to rapid theory prototyping 1.Start with the simplest, most generic model of a theory and ask it to do a task Flat visual search 2.When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledge Hierarchical visual regions + recovery 3.Give the model this mechanism or knowledge through “rapid theory prototyping” Quick change to CogTool’s storyboard, not ACT-R code Repeat steps 2-3 until the model is failing only by chance 7 iterations, described in the paper 4.Analyze all the failures and decide: Whether prototyped theory matches human performance If so, prioritize effort to improve the underlying theory High priority: Hierarchical visual regions + recovery See Leonghwee Teo’s talk on Sunday Other examples of Rapid Theory Prototyping My poster with Tiffany Jastrzembsk for modeling aging aduts (sort of) Paper at the ASSETS conference modeling blind users of screen readers