1. Aptitude, Levels of Automation, and the Effectiveness of Training Benjamin Clegg Department of Psychology Colorado State University Eric Heggestad Department of Psychology University of North Carolina – Charlotte November 4, 2005 MURI Grant Presentation for the Kickoff Meeting

2. Overview Background Levels of automation Levels of automation Aptitude Treatment interaction Aptitude Treatment interaction Proposed experiments Integration within MURI group Outcomes & implications Relevant example scenario Relevant example scenario Training for the networked battlefield Training for the networked battlefield

3. Supervisory control Sheridan (1976, 1997) Concept of supervisory control Increasingly, human operator as supervisor Function: monitor automation

4. Allocation of function How to apportion control between human and machine?

5. Scale of allocation of function Sheridan and Verplank (1978) Scale describing a range of options “Levels of Automation”

6. Human has complete control Total automation (autonomous robots)

7. Human has complete control ? Total automation (autonomous robots)

8. Human has complete control Automation recommends actions Total automation (autonomous robots)

9. Human has complete control Automation executes actions (but informs human) Total automation (autonomous robots)

10. Deleterious effects of automation If operators are “out of the loop” then this can impact: Acquisition and maintenance of operators’ skill Acquisition and maintenance of operators’ skill

11. Paradigm Study conducted using a simulated process control task Modeled on an orange juice pasteurization plant Called “Pasteuriser” Called “Pasteuriser” Raw orange juice heated to kill bacteria Raw orange juice heated to kill bacteria

13. 3 MAJOR SUBSYSTEMS

18. Rate at which juice pumped around the system

24. GOOD JUICE = temperature correct

25. BAD JUICE (spoiled) = temperature too hot 

26. UNPASTEURISED JUICE = temperature too cold

27. Nature of the task Complexity of the operator’s task arises from: The interaction of the subsystems The interaction of the subsystems The presence of competing goals The presence of competing goals Dynamics that incorporate time lags Dynamics that incorporate time lags

28. Non-optimal automation Could design a perfect controller, but then humans have no reason to intervene

29. Added levels of automation Examine the impact of various levels of automation on training and performance

30. Low level of automatic control Manual control only All 3 subsystems set by operator All 3 subsystems set by operator

31. Medium level of automation System offers a decision aid Operator can decide to let automation control that subsystem Automatic control last 20 seconds or until operator intervenes with subsystem Automatic control last 20 seconds or until operator intervenes with subsystem

36. High level of automation System offers a decision aid Automation takes over control of the system unless cancelled Automatic control last 20 seconds or until operator intervenes with subsystem Automatic control last 20 seconds or until operator intervenes with subsystem

39. Previous data Some differences in training with variations in Level of Automation

40. Individual differences People differ in many ways Intellectual functioning Intellectual functioning Aptitude / Cognitive ability Resources to process information Resources to process information Lower ability… Lower ability… Fewer resources to devote to task Fewer resources to devote to task Acquire skills slower Acquire skills slower

41. ATI Research Cronbach (1957) proposed the idea of the Aptitude Treatment Interaction Brings together the “two disciplines of scientific psychology” Brings together the “two disciplines of scientific psychology” In an experiment, variability within a condition is error Individual characteristics may explain some of that remaining variance Individual characteristics may explain some of that remaining variance

42. ATI Research So, in ATI research The relationship between the IV and the DV are evaluated within each treatment condition The relationship between the IV and the DV are evaluated within each treatment condition Interaction occurs when the relationship between the IV and the DV differ across the conditions Interaction occurs when the relationship between the IV and the DV differ across the conditions

43. Example ATI Class Performance High Low Intelligence Test Score Low High High Structure Low Structure X X

44. Level of automation effects

45. Variation within groups

46. Individual differences General pattern emerged, but … Lots of variation within the groups Lots of variation within the groupsHypotheses: Aptitude explains variance Aptitude explains variance Performance lowest in medium automation Performance lowest in medium automation Essentially adding a secondary task Essentially adding a secondary task Ability—performance relationship strongest for medium automation Ability—performance relationship strongest for medium automation

47. We hypothesize … Post-Training Task Performance High Low Cognitive Ability Low High High Auto. Medium Auto. Low Auto. The relationship between cognitive ability and post- training performance will vary across the levels of automation training groups

48. We also hypothesize … Transfer Task Performance High Low Cognitive Ability Low High High Auto. Medium Auto. Low Auto. The relationship between cognitive ability and transfer performance will vary across the levels of automation training groups

49. Outcomes Based on the ATI findings, we can provide estimates of performance variance due to Cognitive ability Cognitive ability Level of automation Level of automation Level of automation X cognitive ability interaction Level of automation X cognitive ability interaction

50. Proposed experiments Study I: Aptitude, levels of automation and the effectiveness of training Study II: Fixed versus varying levels of automation in training

51. Current status Task relevant measures of “aptitude” Intelligence Intelligence Examining tests from the ETS kit of factor- referenced tests Examining tests from the ETS kit of factor- referenced tests Working memory Working memory Conscientiousness Conscientiousness

52. Project integration Integration of individual differences into taxonomy Relationship to training principles Relationship to training principles

53. Implications Training humans as supervisors of automation Ideal forms of automation Ideal forms of automation Training matched to individuals