1. Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu http://www.ecse.rpi.edu/homepages/qji

3. Visual Behaviors Visual behaviors that typically reflect a person's level of fatigue include –Eyelid movement –Head movement –Gaze –Facial expressions

4. Eyelid Movements Tracking Eyes Develop techniques that can robustly track eyes under different face orientations, illuminations, and large head movements. Compute Eye movement parameters PERCLOS Average Eye Closure/Open Speed (AECS)

5. Eyes tracking demo

6. PERCLOS measurement over time

7. Average Eye Closure Speed Over time

10. Gaze (Pupil Movements) Real time gaze tracking No calibration is needed and allows natural head movements !. Gaze parameters Spatial gaze distribution overtime Ratio of fixation time to saccade time.

11. Gaze distribution over time while alert

12. Gaze distribution over time under fatigue

14. Head Movement Real time head pose tracking Perform 3D face pose estimation from a single uncalibrated camera. Head movement parameters Head tilt frequency over time Percentage of side views (PerSideV)

17. Facial Expressions Tracking facial features Recognize certain facial expressions related to fatigue like yawning. Building a database of fatigue expressions.

18. Facial expression demo

19. Fatigue Modeling Knowledge of fatigue is uncertain and from different levels of abstraction. Fatigue represents the affective state of an individual, is not observable, and can only be inferred.

20. Overview of Our Approach Propose a probabilistic framework based on Bayesian Networks (BN) to model fatigue. systematically integrate various sources of information related to fatigue. infer and predict fatigue from the available observations and the relevant contextual information.

21. Bayesian Networks Construction A BN model consists of target hypothesis variables (hidden nodes) and information variables (information nodes). Fatigue is the target hypothesis variable that we intend to infer. Other contextual factors and visual cues are the information nodes.

22. Causes for Fatigue Major factors to cause fatigue include: Sleep quality. Circadian rhythm (time of day). Physical conditions. Working environment.

23. Bayesian Network Model for Monitoring Human Fatigue

24. Interface with Vision Module An interface has been developed to connect the output of the computer vision system with the information fusion engine. The interface instantiates the evidences of the fatigue network, which then performs fatigue inference and displays the fatigue index in real time.

25. Conclusions Developed non-intrusive real-time computer vision techniques to extract multiple fatigue parameters related to eyelid movements, gaze, head movement, and facial expressions. Develop a probabilistic framework based on Bayesian networks to model and integrate contextual and visual cues information for fatigue monitoring.