1. ACI/HFES, Baltimore, October 1-3, 2007 Sponsored by US Army Research Office SBIR Phase II: Wearable Physiological Sensor Suite For Early Detection Of Cognitive Overload US Army Aberdeen Test Center: Remote Neurological Monitoring Program

2. ACI/HFES, Baltimore, October 1-3, 2007 Stress, Overload, and Performance

3. ACI/HFES, Baltimore, October 1-3, 2007 Mental State Estimation General Cognitive Status Work- load Mental Fatigue Non- specific Factors Engage- ment Biosignal Sources Work- load Mental Fatigue Non- specific Factors Engage- ment

4. ACI/HFES, Baltimore, October 1-3, 2007 Definitions Engagement: selection of a task as the focus of attention and effort Workload: significant commitment of processing resources to an engaged task Visual, Auditory, Haptic Psychomotor Cognitive (memory, executive) Overload: task demands outstrip available processing resources Mental Fatigue: desire to withdraw attention and effort from an engaged task associated with extended performance (~45 min) General Cognitive Status Work- load Mental Fatigue Non- specific Factors Engage- ment

5. ACI/HFES, Baltimore, October 1-3, 2007 Overload Patterns in Multimodal Signals

6. ACI/HFES, Baltimore, October 1-3, 2007 Multimodal Classifier Testbed

7. ACI/HFES, Baltimore, October 1-3, 2007 Multimodal Classifier Testbed

8. ACI/HFES, Baltimore, October 1-3, 2007 Multimodal Classifier Testbed

9. ACI/HFES, Baltimore, October 1-3, 2007 Multimodal Classifier Testbed

10. ACI/HFES, Baltimore, October 1-3, 2007 Multimodal Classifier Testbed C2 C1

11. ACI/HFES, Baltimore, October 1-3, 2007 Engagement/Workload Related EEG Sources Passive viewing: theta alpha Engaged 5: theta alpha

12. ACI/HFES, Baltimore, October 1-3, 2007 Fatigue-Related EEG Sources Black = Alert Red = Mentally Fatigued Fz Pz Frontal Theta Parietal Alpha

13. ACI/HFES, Baltimore, October 1-3, 2007 Other Sources SourceEffect of Workload Heart rateIncrease Heart rate variability (and HFQRS)Decrease Vertical and horizontal EOG (eye movements)Increase BlinksMay decrease for intake Pupil diameterIncrease Skin conductance, SCR, GSRIncrease EMG (frontalis, temporalis, trapezius)Increase

14. ACI/HFES, Baltimore, October 1-3, 2007 Experimental Controls Task learning Time of day and time on task Test day Food consumption Neurotoxic effects Test environment Inadequate measurement of physiological variance Inadequate definition of ground truth workload levels: Expert analysis and scoring of replayed videos Logging all user inputs Measuring reaction times to probes

15. ACI/HFES, Baltimore, October 1-3, 2007 Validation of Workload Manipulation NASA - TLX questionnaires P300

16. ACI/HFES, Baltimore, October 1-3, 2007 Within-day Results

17. ACI/HFES, Baltimore, October 1-3, 2007 Day-to-day Results

18. ACI/HFES, Baltimore, October 1-3, 2007 Stabilizing Classifiers

19. ACI/HFES, Baltimore, October 1-3, 2007 Stabilized Day-to-day Results

20. ACI/HFES, Baltimore, October 1-3, 2007 Summary Biosignals exhibit high sensitivity to mental states, such as engagement, workload, and fatigue Accurate biosignal-based models or gauges can be developed under controlled conditions and extended to new conditions However, cognitive gauges are not very stable over time, due to behavioral, strategic, and physiological variability Multimodal models capture a wide range of behavioral and physiological variability, improving robustness of gauges over time and conditions Signal processing and computational methods help, but are not enough to yield stable models Some recalibration or model adaptation is currently required We seek ways to stabilize models with a minimum of recalibration