1. 1 A Learning Model of a Long, Non-iterative Spreadsheet Task Frank E. Ritter, Jong W. Kim, and Jaehyon Paik College of IST, Penn State Presented at the ACT-R Workshop, 5 aug 2010 Learning, long term, large task Non-iterated task Exploration of model strategies Retention Subtask learning Individual differences

2. 2 The Spreadsheet Task Using the Dismal (Emacs) spreadsheet  (Ritter & Wood, 2005) In AquaEmacs (Reitter et omnia alia, 2009). Recorded behavior with RUI logger  (Kukreja et al., 2006) 14 subtasks, taking about 30 to 10 min.  (Kim, 2008) 1.FILE OPEN 2.SAVE AS THE FILE WITH INITIALS 3.CALCULATE AND FILL IN FREQUENCY COLUMN (B6-B10) 4.CALCULATE TOTAL OF FREQUENCY COLUMN IN B13 5.CALCULATE AND FILL IN NORMALIZATION COLUMN (C1 TO C5) 6.CALCULATE TOTAL OF NORMALIZATION COLUMN IN C13 7.CALCULATE LENGTH COLUMN 8.CALCULATE TOTAL OF LENGTH COLUMN 9.CALCULATE TYPED CHARACTERS COLUMN 10.CALCULATE TOTAL OF TYPED CHARACTERS COLUMN 11.INSERT TWO ROWS AT A0 CELL 12.TYPE IN YOUR NAME IN A0 13.FILL IN CURRENT DATE USING THE DISMAL COMMAND 14.SAVE AS PRINTABLE FORMAT 1. FILE OPEN 1.Get ready 2.Attend to file 3.Move to file 4.Click on file 5.Attend to open file 6.Move to open file 7.Click on open file 8.Attend to dismal file 9.Move to dismal file 10.Click on dismal file 11.Click to choose 2. SAVE AS THE FILE WITH INITIALS 1.Attend to dfile 2.Move to dfile 3.Click on dfile 4.Attend to save-buffer-as ….

3. 3 The Models 12 models 1 novice model, 9 rules, 540 chunks + 2 types + goal 1 expert model, 540 rules (+ 542 chunks, not used) 10 intermediate models, 0%, 10%, …90% expert 0% had 540 rules and 542 used chunks 10% had 540 rules and 488 used chunks (last 90%) 50% had 540 rules and 271 chunks used (last 50%) 90% had 540 rules and 54 used chunks (last 10%) Could be further distributions and uses of rules and chunks With of course different transfer, etc. We add: 452 keystrokes, 126 moves, and 75 handmoves = 620.6 s added

4. 4 Novice Model 9 rules to walk DM task tree, 542 chunks Production compilation starts immediately Learns to stop and retrieve start across trials! We should reset chunker time when reloading model Learned productions are several combinations On trial 100, 37 rules fire, some retrievals still

5. 5 Expert Model 542 rules, 542 chunks, no chunks used in 100% expert 54 rules fired at trial 100 Need a clever graphic to show how this happens Matt Walsh: we need a model of the model 5,854 rules learned over 100 trials

6. 6 10 runs, all Models 6 Lots of learning (mostly proceduralization) Looks ok

7. 7 How long does learning go on? (100 runs) ~10x real-time Runs, good speed up Not linear in log/log space Learns out to trial 100 Seems to predict non-power law, due to I/O constraint Model /learned rules / fired rules Novice 5,770 505,858 05,741 605,832 105,576 705,885 205,633 805,787 305,611 905,808 40 5,727 Expert 5,854

8. 8 Discussion of Models Different types of models, fundamentally novice (finds next subtask), fundamentally expert (knows next subtask) Novice walks memory, gets faster at walking DM, learns actions to do Expert 0 has 542 retrievals to do, in order, linked Expert 100 has 0 retrievals to do, in order, linked Both learn rules and most strengthen DMs Both learn to do more at once, two actions on a single rule Learns rules they shouldn’t, to do output in same rule, because /PM not used, /PM would stop chunker [major coal, but fixable we think with /PM]

9. 9 The Environment and Data  (Kim, 2008) RUI Dismal Vertical mouse 30 subjects Did the task 4 times Long term retention data gathered, but not shown here

10. 10 Comparison of Models with Data Group data does not match slope Group data does not match intercept Except trial 4 = expert Novice best shaped fit Other problem: humans separated by day, ACT-R by 0 seconds Log/log does not help People learn faster, and model is learning too fast because of not using /PM

11. 11 Limitations Does not use Perceptual-Motor Could use PM to stop learning to do all at once Does not fit data in good detail Tasks are organized as a tree Will need learning in /PM as well

12. 12 Future Work Use /PM, will slow down chunker Use graph of tasks not tree Examine effect of chunk activation on rule learning Examine structure of task and reuse/repractice of common subtasks Match individual learning data Examine retention Examine how to explain/explore such models 5,000 initial rules and 60,000 learned rules

13. 13 Summary Modeling a 30 to 10 min. non-iterative, repeated task with learning (with later retention modeling to come) Enjoyed the chunker and architecture’s speed and trace Have 30 subjects of data, logged Grappled with 65,000 rules Compared to learning data, and lost Have shown we can model long-ish term behavior, we did 30 min., and longer (100 min.) is very possible This work was supported by ONR under contracts N00014-06-1- 0164, N00014-09-1-1124, and DTRA under contract 1-09-1-0054.

14. 14  Publications & References  Kim, J. (2008). Procedural skills: From learning to forgetting. Department of Industrial and Manufacturing Engineering, The Pennsylvania State University, University Park, PA.  Kim, J., & Ritter, F. E. (2007). Automatically recording keystrokes in public clusters with RUI: Issues and sample answers. In Proceedings of the 29th Annual Conference of the Cognitive Science Society, 1787. Cognitive Science Society: Austin, TX.  Kukreja, U., Stevenson, W. E., & Ritter, F. E. (2006). RUI—Recording User Input from interfaces under Windows and Mac OS X. Behavior Research Methods, 38(4), 656-659.  Ritter, F. E., & Wood, A. B. (2005). Dismal: A spreadsheet for sequential data analysis and HCI experimentation. Behavior Research Methods, 37(1), 71-81.