1. Evaluating Human-Machine Systems
Human Factors Test & Evaluation

2. Human Factors Test and Evaluation (HFT&E)
HFT&E responsibilities
assurance that applicable HFT&E requirements are met
demonstrate conformance with design criteria
validate human performance requirements
confirm conformance with performance requirements
obtain quantitative measures of performance
determine if undesirable attributes have been introduced

3. Prerequisites Some of the following:
detail design drawings & specifications
mockups
role-playing scenarios
functional prototypes
person-in-the-loop simulation scenarios
engineering prototype
operations manuals
technical manuals

4. Some HFT&E Methods To assure that requirements are met
demonstrate conformance with design criteria
assess human performance in general
confirm conformance with performance requirements
obtain quantitative measures of performance
determine if undesirable attributes have been introduced

5. Inspection Applicability: verification of certain WHAT requirements
Procedure
identify WHAT requirements to be verified
inspect design specifications mockup/prototype
[observe enacted scenarios/simulations]
decide and record whether requirements are met
Considerations
simple
inexpensive
essential
laborious
can be subjective
ignores questions of HOW WELL

6. Direct continuous observation
Procedure
conduct scenarios/operations
observe continuously
record
Considerations
simple
inexpensive
essential
laborious
error-prone
requires skilled observer

7. Direct sampled observation
Procedure
conduct scenarios/operations
observe periodically
record
Considerations
less time-consuming

8. Technical manual functional evaluation
Procedure
review
job instructions
training manuals
job performance aids
with respect to
accuracy
completeness
adequacy
Considerations
manuals required

9. Heuristic Evaluation / Checklists (e.g., HEDGE)
Procedure
classify equipment (e.g., vehicle, weapon, electronics)
identify appropriate tasks
determine appropriate HF considerations
identify specific test criteria
prepare test plan
use checklist to conduct test
Considerations
labor intensive
time-consuming
systematic and thorough

10. Environment and engineering measurement equipment
Procedure: utilize appropriate equipment, e.g.,
photometer
sound level meter
vibration meter
thermometer
force, torque, and dimension equipment
anthropometry instruments
Considerations
not a complete HFT&E method
used in conjunction with other methods

11. Interviews Procedure discuss test events with subjects Considerations
question about specific tasks
question ASAP
focus on critical incidents

12. Questionnaires Procedure select question form open-end
dichotomous (Y/N)
multiple-choice
Likert scale (e.g., 0 = strongly disagree – 10 = strongly agree)
write questions
formulate questionnaire
pretest, revise
administer
analyze
Considerations
potentially powerful if designed well
someties the only practical way to assess
subjective

13. Videotapes Procedure conduct scenarios/operations videotape
review tapes and record
events
actions (speed, accuracy)
observations
Considerations
potentially powerful
expensive

14. Event recording Procedure conduct scenarios/operations
record system state variables
analyze
Considerations
powerful
requires specialized equipment
sensors
event recorders
computers
potentially expensive

15. Physiological instrumentation
Procedure
measure relevant physiological variables
heart rate, blood pressure
respiration rate, volume
brain waves
body temperature
body movement (e.g., eye movement)
Considerations
potentially powerful
intrusive (may interfere with performance)
expensive

16. Statistical analysis Procedure Considerations
identify independent variables
identify dependent variables
identify appropriate statistical methods
t-tests
ANOVA
regression analysis
non-parametric methods
design experiment
collect data
analyze data
Considerations
useful for comparisons
potentially powerful
only way to determine if differences really exist
requires skilled analysts
time consuming

17. HFT&E Examples
C-Section Facilitator
AgendaManager

18. Usability Assessment for C-Section Facilitator Prototype
Lin-Hui Huang
Chatpong Phinyawat
Robin Feuerbacher
Prototype Developer: Yu-chih Hsiao
IE 545 Dr. Funk December 1, 2006

19. CS Facilitator Overview
Status displays on touch-screen computer screens
Prepare Patient, Team and Operating Room (OR)
Serves as status communication and process memory aid
Dial-up page system
Significant time savings but not modeled in prototype
Initial Summons Data for team paging input by Ward Nurse/Clerk into the CS Facilitator
PDA devices used for paging and responses
Patient data entry
Patient data input via Tablet PC’s
Drop-down menus facilitate input

20. Initial Team & OR Status - Original

21. Problems Indicated - Original

22. Usability Assessment
Demonstrated C-Section Facilitator to doctors, nurses at PeaceHarbor Hospital
Followed scripted scenario.
failure of a team member to respond in a timely manner
miscommunication of an assigned task which is subsequently not done
problem with equipment in the OR
Interviewed participants as to system's
efficiency
learnability
memorability
satisfaction
Conducted visibility check from 20 feet for 37 inch diagonal screen

23. Scripted Scenario 0:00 C-Section decision made
0:03 Input summons data sent
0:03 Informed consent signed
0:04 Baby warmer assigned to RNA but miscommunicated
0:07 Patient prepped through Bicitra
0:07 Pediatrician or Family Practice Doctor not responding

24. Scripted Scenario 0:09 Catheter insertion delayed
0:10 All members respond except Pediatrician or Family Practice Doctor
0:13 Kiwi vacuum not functioning
0:15 All team members arrived except Pediatrician or Family Practice Doctor
0:15 Baby warmer being prepared

25. Initial Team & OR Status - Revised

26. Problems Indicated - Revised

27. Evaluation Results General Findings Well received by doctors, nurses
potential to reduce preparation times for emergency C- sections
Some Recommended Changes
Start white for prep team; turn red if not responding to page within 3 minutes (timer to be implemented in future version)
Added scrub and circulating nurses to team status plus miscellaneous title changes
Added “Prophylactics and Antibiotics” to Prepare Patient Status
Increased font sizes for titles

28. Proposed Simulation Evaluation (Test Plan Outline)
Participants: doctors, nurses, “patient”
Apparatus: new prototype Facilitator
Testing and Evaluation Method: role-playing simulation trials
Multiple Scenarios (equivalent)
Independent Variable: aided/unaided
Dependent Variables (Human Performance Measures)
Speed
synthetic summons-arrival times
decision-to-incision time
Accuracy
patient prep error rate
OR prep error rate
User satisfaction (Likert, open-ended)
Testing Procedure/Schedule
Unaided simulation trials / 4 weekends, one per month
“Washout” period / 6 months
Training / one weekend
Aided Simulation trials / 4 weekends, one per month
Analysis: ANOVA, t-tests

29. Ken Funk, Oregon State University
The AgendaManager: A Knowledge-Based System to Facilitate the Management of Flight Deck Activities
Ken Funk, Oregon State University
Rolf Braune, Braune and Associates, Inc.

30. Statement of Needs and Requirements Definition
CTM aid shall
maintain a current model of aircraft state and current cockpit tasks,
monitor task state and status,
compute task priority,
remind the flightcrew of all tasks that should be in progress, and
suggest that the flightcrew attend to tasks that do not show satisfactory progress.
leave the pilot in control

31. Simulator (with EICAS)

32. AMgr Display (replaced EICAS)

33. Simulator (with AMgr)

34. AMgr Operation simulator runs
pilot declares goals via ATC acknowledgements
System & Actor Agents instantiate Goal Agents
Goal Agents watch for goal conflicts
Function Agents assess function status
AgendaManager informs pilot via display

35. AgendaManager Display Design
extinguish L engine fire not OK -> continuing
descend to 7,000 ft A/F alt goal conflict
maintain 070 deg
slow to 240 kt fast -> accelerating
correct fuel balance L heavy -> increasing

36. extremely important, urgent goals
(highest priority)
trend info
aviate goals
(high priority)
extinguish L engine fire not OK -> continuing
descend to 7,000 ft A/F alt goal conflict
system goals
(lower priority)
maintain 070 deg
slow to 240 kt fast -> accelerating
correct fuel balance L heavy -> increasing
gray = OK
amber = not OK
red = important/urgent not OK

37. Test and Evaluation (1)
Objective: compare AMgt performance (AMgr vs EICAS)
Apparatus
flight simulator
AMgr
Subjects: 8 line pilots
Scenarios:
EUG to PDX
PDX to Eugene
Primary factor: monitoring and alerting condition
EICAS

38. Scenario from another flight simulator study

39. Test and Evaluation (2) General Procedure subject introduction
automatic Speech Recognition system training
flight training (using MCP)
subsystem training (fault correction)
EICAS/AMgr training
Trials
Scenario 1 (EICAS/AMgr)
experimenter/ATC controller gives clearances, induces faults, induces goal conflicts
subject acknowledges clearances, flies simulator, corrects faults, detects and resolves goal conflicts
Scenario 2 (AMgr/EICAS)

40. Evaluation Results

41. HFT&E (Review) HFT&E responsibilities
assurance that applicable HFT&E requirements are met
demonstrate conformance with design criteria
validate human performance requirements
confirm conformance with performance requirements
obtain quantitative measures of performance
determine if undesirable attributes have been introduced

42. Elements of a Human Factors Testing and Evaluation Plan: Descriptions of
Participants
Apparatus, Instruments, e.g.,
Mockups
Prototypes
Recording equipment
Timers
Measurement equipment
Questionnaires, checklists, etc.
Testing and Evaluation Methods
Scenarios
[ Independent Variables (e.g., with/without, system/baseline) ]
Human Performance Measures
Speed
Accuracy
Training Time
Satisfaction
Testing Procedure(s)
Analysis
Schedule