Toward quantifying the effect of prior training on task performance MURI Annual Review September 26-27, 2006 Bill Raymond

Overview Project goal: Quantify the effects on performance of different training methods for complex military tasks. Project goal: Quantify the effects on performance of different training methods for complex military tasks. Feature decomposition: Feature decomposition: 1.Task type 2.Training method 3.Performance assessment (context & measures) 4.Training principles Planning matrix: Planning matrix: - Capture where we know of, and can quantify in terms of performance measures, effects of training method and performance context on task components. Quantify principles: Quantify principles: - Derive performance functions for points in the feature space using empirical data from laboratory tasks. - Generalize performance functions for implementation in IMPRINT modeling tool to simulate training effects on task performance.

Decomposition issues Constraints on decompositions Constraints on decompositions Features must relate to experimental designs  Must be able to describe all experimental tasks.  Task, training, and performance context features can be no finer than experimental manipulations. Features may be different for research and IMPRINT  Can’t control training in the real world as carefully as in the laboratory  Not all experimental results will be major effects.  IMPRINT task categories are already defined.  Planning features should converge to final IMPRINT features, diverging from research features

Planning matrix issues What will the matrix construction provide? What will the matrix construction provide?  Current and planned research coverage of space  May be used by us or others for future planning  Approximation of final IMPRINT training features  Initial step in determining the generality of performance functions in the space

Training variables - during skill learning: Training variables - during skill learning:  How was the skill taught?  What kind of practice did learners get?  How did practice relate to the way the skill will be used? Performance context variables - at skill use: Performance context variables - at skill use:  How does expected performance relate to training?  How long has it been since training?  Did learners get refresher training? Starting point: Analyzing training and performance Pedagogy Practice Performance } }

Task, training, and performance matrix Task components Training features Performanc e context PedagogyPractice Visual Numerical Analysis Information processing Fine motor - discrete Fine motor - continuous Gross motor - light Gross motor - heavy Communication (reading & writing) Communication (oral) IMPRINT task taxons Data entry

Pedagogy parameters Method Method  Instruction (=default)  Demonstration  Simulation  Discovery  Modeling/mimicking  Immersion Learning location (local = default, remote/distance) Learning location (local = default, remote/distance) Discussion/Question and answer (no = default, yes) Discussion/Question and answer (no = default, yes) Individualization (no = default, yes) Individualization (no = default, yes)

Task by Pedagogy parameters Task components Pedagogy Method Learning location Discussion /Q&A? Individualized? Visual Numerical Analysis Information processing Fine motor - discrete Fine motor - continuous Gross motor - light Gross motor - heavy Communication (reading & writing) Communication (oral) IMPRINT task taxons Data entry (Instruction) Classificatio n Inst/Discovery

Practice parameters Scheduling of trials and sessions Scheduling of trials and sessions  Number  Spacing (massed = default, spaced, expanding/contracting)  Distribution (mixed = default, blocked) Scope of practiced task (partial, whole = default, whole + supplemental) Scope of practiced task (partial, whole = default, whole + supplemental) Depth of processing (no = default, yes) Depth of processing (no = default, yes) Processing mediation (no = default, yes) Processing mediation (no = default, yes) Stimulus–response compatibility (yes = default, no) Stimulus–response compatibility (yes = default, no) Time pressure (no = default, yes) Time pressure (no = default, yes) Feedback - presence (no = default, all trials, periodic) Feedback - presence (no = default, all trials, periodic) Context of practice Context of practice  Distractor (no = default, yes)  Secondary activity (no = default, yes)

Task by practice Task components Practice SchedulingScopeProcessing depth Processing mediation Stimulus- response compatibility Time pressure FeedbackContext Visual Numerical Analysis Information processing Item repetition, # Sessions, Spacing Part/ whole Yes (presentation format) Yes (prior knowledge) No (Input- output Format) Yes (response & accuracy) Distractor/2nda ry activity (vocal activity) Fine motor - discrete Item repetition, # Sessions, Spacing Part/ whole Yes (response & accuracy) Distractor/2nda ry activity (vocal activity) Fine motor - continuous Gross motor - light Gross motor - heavy Communication (reading & writing) Communication (oral) Data entry IMPRINT task taxons

Performance context parameters Transfer Transfer  New context (relative to training)  New task (relative to training) Delay interval (default = none, time period) Delay interval (default = none, time period) Refresher training (default = no, schedule) Refresher training (default = no, schedule)

Task by performance parameters Task components Performance context New context New task Delay interval Refresher training Visual Numerical Analysis Information processing Yes (typing hand, output configuration) Yes Fine motor - discrete Yes (typing hand, output configuration) Yes Fine motor - continuous Gross motor - light Gross motor - heavy Communication (reading & writing) Communication (oral) IMPRINT task taxons Data entry

Quantifying training principles Data Entry used as an example Data Entry used as an example Consider two principles Consider two principles  Practice  Learning (Power law of practice)  Skill practice - no item repetition  Specific learning - item repetition  Prolonged work  Diminished performance Quantify effects for each taxon Quantify effects for each taxon  Cognitive (“Information processing”)  Physical (“Fine motor - discrete”) …and performance context …and performance context  Transfer to new items (similarity dimension)  Retention of learned skill (refresher training)

Skill practice: Quantifying learning Skill practice improves performance.5 msec/item Skill practice improves performance.5 msec/item  Mean decreases 300 msec in 640 (unique) items Where does skill practice come from? Where does skill practice come from?  Repetition of individual digits (and pairs of digits?)  Cognitive or physical learning?  Individual differences?

Skill practice: Origin or learning Pair repetition? Subjects appear to “chunk” digits 1 & 2, digits 3 & 4 Subjects appear to “chunk” digits 1 & 2, digits 3 & 4  so they may be learning something about pairs of digits

Skill practice: Origin of learning Pair repetition? Effect of 2-digit chunk practice appears minimal Effect of 2-digit chunk practice appears minimal  Skill practice is general facility at number typing

Skill practice: Type of learning Physical or cognitive? Speed improvement occurs on digits 1 and 3 Speed improvement occurs on digits 1 and 3  Learning is cognitive

Skill practice: Individual differences “ Chunkers ” are 15% slower than “ non-chunkers ” “ Chunkers ” are 15% slower than “ non-chunkers ”  Appears to be a strategy choice  Pedagogy - advantage for instruction over “discovery”?

Specific learning: Quantifying learning Repetitious practice improves performance faster initially Repetitious practice improves performance faster initially  Power law of practice

General learning functions Performance as a function of number of repetitions Performance as a function of number of repetitions  Planned experiment...?...?

General learning functions Transfer and retention Transfer and retention  Planned experiment... New items?Old items? TransferRetentionLearning

Prolonged practice Prolonged work results in an increase in errors Prolonged work results in an increase in errors  Accuracy rate decline of about 1% over 320 items Where does the decline in accuracy originate? Where does the decline in accuracy originate?  Cognitive or physical fatigue?

Prolonged practice: Type of performance decline Two types of errors: Two types of errors:  Stimulus adjacency errors: 1234  1244  Key adjacency errors: 1234  % of errors are of these two types 90% of errors are of these two types Origin of errors Origin of errors  Stimulus adjacency = cognitive  Key adjacency =motor phase, which could be motor output planning (cognitive) or motor execution (execution)

Prolonged practice: Type of performance decline Practice results in an increase in key adjacency errors Practice results in an increase in key adjacency errors  Accuracy decline occurs during the motor phase (which may be both cognitive and physical)

Prolonged practice: Type of performance decline Feedback eliminates the speed-accuracy trade-off Feedback eliminates the speed-accuracy trade-off  If feedback is cognitive, then the relevant processes in the motoric phase must be cognitive

Summary Task components Training features Performance context PedagogyPractice Information processing (Cognitive) Method: Instruction - strategy instruction may improve speed “Discovery” - some Ss 15% slower Scheduling: no reps - speed decrease linear (.5 msec/item) item reps - power law (parameters to be determined) Feedback: no feedback - accuracy decline (1%/300 items) typing/accuracy feedback - no decline Transfer: Retention: (planned experiment) Fine motor - discrete (Physical) Transfer: Retention: (planned experiment) IMPRINT task taxons