1. IMPRINT models of training: Digit Data Entry and RADAR MURI Annual Meeting September 7, 2007 Carolyn Buck-Gengler Department of Psychology and Center for Research on Training University of Colorado at Boulder

2. September 7, 2007 Overview: Modeling effort Digit Data Entry (DDE; with Bill Raymond): –Completed model –Did goodness of fit evaluations (also Bengt Fornberg) –Reported at BRIMS (Buck-Gengler, Raymond, Healy, & Bourne, March, 2007) –Bengt and Bill: Model comparisons RADAR: First start in February –Beginning structure of model created; old IMPRINT –Resumed when new IMPRINT Pro available in April/May

3. September 7, 2007 Summary of DDE model when different, last year’s model in blue italics Modeled both experiments of Healy, Kole, Buck-Gengler, & Bourne (2004) in one model (only Experiment 2) Model based on cognitive model of DDE –Cognitive and physical components assigned to different keystrokes –Chunking Difference between hands (R faster than L) and hand switching Learning (RT improvement) due to repetition of numbers and due to general practice (only unique numbers; improvements made to how learning modeled) –Differences in cognitive and physical learning, assigned to different keystrokes Cognitive Fatigue (not in previous model) Accuracy over the course of the experiment –Overall, and by output length

4. September 7, 2007 Picture of RADAR task one frame

5. September 7, 2007 Shift type 4 different shift types, created by crossing mapping type and load –Mapping: Consistent (CM) vs. Varied (VM) CM: target(s), distractors from DIFFERENT character sets (letters, digits) VM: target(s), distractors from SAME set –Load: light (1-1) or heavy (4-4) 1-1: one possible target; one blip has a character 4-4: four possible targets; all 4 blips have characters

6. September 7, 2007 Other details Particular target(s) different every shift 2 sessions (training, test) –Each session 8 blocks of 20 shifts of one type 15 shifts have a target, 5 have only distractors IF target, in only one of the frames, rest only distractors –Block order each session: CM1-1 CM4-4 VM1-1 VM4-4 VM4-4 VM1-1 CM4-4 CM1-1 Half the subjects also have secondary tone- counting task (tone/no tone crossed between sessions)

7. September 7, 2007 Picture of RADAR task CM 4-4 shift example – one frame target

8. September 7, 2007 Picture of RADAR task CM 4-4 trial example target

9. September 7, 2007 The RADAR task Trials are called “shifts” –in blocks of 20 shifts of 7* “frames” each shift –either CM (“consistent” mapping) or VM (“varied” mapping) –either 1 or 4 possible targets in a shift and blips filled in each frame 15 of 20 shifts in each block have a target, 5 have only distractors (foils) Half the subjects also hear tones (approx 1 per second) and have to keep track of how many differ from established reference tone * 9, but first and last all blank and not recorded

10. September 7, 2007 Shifts and frames 4 “blips” (character-sized inverted screen rectangles, sometimes with a character in them) start in the 4 corners of the target screen and move diagonally quickly toward the center of the screen –Time to get to the center (usually) 2.062 s unless subject responds 1 shift = 9 frames: first and last have all blank blips; remaining 7 have 1 or 4 blips filled in with a character –From now on, will ignore existence of the frames with all blanks, and refer to “the 7 frames” IF target shift, ONE frame will have a target in one of the filled-in blips; distractors will be in remaining blips (if non- target shift, no frames have target, all chars are distractors)

11. September 7, 2007 CM vs VM (within S) CM = consistent mapping –Target(s) is (are) from one set (digits, letters), distractors are from the other set VM = varied mapping –Target(s), distractors from same character set Target set varied between subjects for counterbalancing purposes, did not make a difference statistically, so ignored in model

12. September 7, 2007 1 vs 4 (within S) In other reports, called 1-1 and 4-4 or low and high load 1 (low load): –Target: for each shift, subject is shown one character from the target set; that will be the target that shift –Number of blips filled in: one of the 4 blips (diff location every frame?) will be filled in with either target or distractor in each of the 7 frames 4 (high load): –Target: for each shift, subject shown four characters from the target set; one of them will be the target that shift –Number of blips filled in: all 4 blips will be filled in. In a target frame, one will be one of the 4 target characters and the rest distractors; in the remaining frames all are distractors Note: 1st and 7th frames only ever had distractors, so target only in 2nd through 6th frames

13. September 7, 2007 Tone counting (between S) Tone counting each session varied between Ss Half from each group switch groups for second session Groups: –No tone/no tone –Tone/no tone –No tone/tone –Tone/tone Not counting TONES per se but how many differ from a particular reference tone

14. September 7, 2007 Subject response GO-NO GO (at frame level): Press space bar if see target character in the frame, otherwise ignore Scored on: –Response time to correctly hitting space bar (when target) from time target frame starts –Accuracy in correctly identifying target –Correct rejection of non-target frames (by non-response)

15. September 7, 2007 Scoring Non-target shift Every frame non-target Target shift No response Correct rejectMiss Response - any frame False alarmNA - before target frame NAFalse alarm - target frame NAHit - after target frame NAMiss

16. September 7, 2007 Scoring implications Late response: If so slow as to respond to a seen target when the next frame has started, scored as a miss Unlike typical experiment where if target can only have hit/miss, with this scoring at shift level, can have false alarm on a target shift too, so the number of shifts contributing to False Alarm rate and Hit rate changes from block to block Makes sense if looking to have one score per trial Hard to model when modeling at frame level, so I recomputed hit/miss/FA/CR at the frame level

17. September 7, 2007 Results to be modeled RT: –CM faster than VM, 1 target faster than 4 –CM 1-1 and VM 1-1 about same RT; CM 4-4 a bit slower, VM 4-4 much slower Hit rate: –CM more accurate than VM, 1 target more accurate than 4 –VM 4-4 is only shift type to show accuracy degradation –Small improvement over time False alarm rate: –sharp learning curve between first and second blocks –some penalty for VM and more for VM 4-4 –lots of learning between 1st and 2nd block of VM 4-4 (blocks 4 and 5) and 1st and 2nd blocks of CM 1-1 (blocks 1 and 8) Tone counting hurts both accuracy and RT –If done during training, hurts performance at test regardless of whether tone counting at test

18. September 7, 2007 Current modeling approach Similar approach to how things done in digit data entry model –Simplest subset of data modeled first, then after that works, add new ones CM/VM x 1-1/4-4; no tone counting Tone counting Learning Add test session to training session –RT and accuracy Statistical Ss will vary around the mean S RT and accuracy for the experiment RTs vary for each trial around the base time for the S Response results from random selection of trial as response or no response based on hit and false alarm rates

19. September 7, 2007 Model Two networks –Computer: presents frames within shifts; 15 of 20 shifts per block contain target; target randomly assigned to one of frames 2-6 “presents” tones to tone-counting “subjects” (will be half of subjects in each session) –Subject goal network: wait until a frame is presented, then respond (or not) to that frame Third network planned –Probable second subject goal network: count tones that are different from the reference tone

20. September 7, 2007 Main network Do One Shift subnetwork Subject goal network

21. September 7, 2007 Do one shift subnetwork Subject network responds here Parallel paths for presenting targets and presenting tones Parallel paths rejoin here

22. September 7, 2007 CM vs. VM CM: Targets from different character set than distractors  very easy and fast discrimination for CM VM: Targets from same character set as distractors  slower decision MODEL: Two separate paths; decision time longer on VM path than CM path; time accrued for each blip looked at

23. September 7, 2007 1 target/blip filled in vs. 4 Eye movement: Occurs for each blip looked at Decision: –1: once there, just have to make the decision –4: assumption: sequential search/decision process because blips start outside fovea Target frames: Target occurs equally in any of the 4 locations, and subjects could focus first equally on any of the 4 locations. So 1/4 of the time, should need to look at just 1, 1/4 of time look at 2, etc.  time to make decision accrues for each looked at Non-target frames: Subject should have to look at all 4; when they don’t false alarm, they don’t respond and time will be maximum frame time

24. September 7, 2007 Subject goal network Respond to frame Decide on whether to respond Respond (or not) Response time (positive response) = N*(eye movement + decision time) + time to press space bar N = number of blips looked at to make decision CM VM

25. September 7, 2007 Tone counting Making the go-no go target detection decision (primary task) incurs cognitive workload Keeping track of tones (secondary task) adds cognitive workload Assumption: when workload added, if workload threshold reached, something suffers; accuracy, RT, or both (on primary task)

26. September 7, 2007 Learning Learning is our term for the observation of improved RT or accuracy over the sessions Assumption: Learning only occurs on correct trials Data: –No consistent pattern of improvement for RTs –Small improvement in hit rates (by shift type) (silent group) –More complex pattern in false alarm rates Tone counting, to the extent it decreases (detection) accuracy, will decrease improvement due to learning

27. September 7, 2007 Model details Incurs more time for shifts that have 4 targets and 4 blips than 1 and 1 Goes through a different “decision process” (path) for VM vs. CM shifts Correctly creates the different kinds of shifts in blocks the same as in the actual experiment, and creates frames which take different amounts of time depending on if a response is made or not Randomly allocates 5 out of 20 shifts as non-target in each block Incorporates both hit rate and false alarm rate into decision making Can create subjects in all four tone/no-tone conditions, and “create” tones on the “computer” side

28. September 7, 2007 Current state of model Only silent group, only first (training) session RT, Hit rate, False Alarm rate –Incorporates both hit rate and false alarm rate into decision making Trial-level variability of RTs NOT doing tone counting or learning yet Not yet incorporating subject-level variability

29. September 7, 2007 RTs for correct shifts

30. September 7, 2007 Hit rate

31. September 7, 2007 False Alarm Rate

32. September 7, 2007 Planned Learning/training –Program it similarly to method used in DDE model –Explore possibility of using new ARL training plug-in Incorporate IMPRINT workload –Tone counting impact on performance Add second (test) session –Tone vs. non-tone manipulation –Effects of training over longer period

33. September 7, 2007 Contributions of IMPRINT modeling of DDE and RADAR Translatable to MATLAB – allows using MATLAB for model comparison and parameter optimization Allows exploration of feasibility of modeling basic low-level cognitive tasks in IMPRINT Leads to insights concerning underlying cognitive processes Leads to predictions concerning training effects and suggestions for further experiments