1. Attention detection in driver simulator project #4 INPG : L. Bonnaud, A. Caplier UCL : D. trevisan, B. Macq Participants : A. Benoit, G. Chanel, P. Ngo, V. Levacic, C. Thillou Guest star : L. Lawson

2. 6/3/20152 Objectives Multimodal detection of user state –Hypo vigilance –Stress level –In the context of driving simulator Alarm => multimodal feedback to the user Test of OpenInterface as an integration platform

3. 6/3/20153 Signals used for user state assessment Video (camera) Physiological (biopac) HeadEyesMouthECGGSR Hypovigilance ?? Stress ?

4. 6/3/20154 General architecture Significative rotation ? High frequency ? Both eyes closed ? Video Mouth opening detector ECG analysis Head rotation estimation Eyes blinks detector GSR analysis GSRECG Yawn ? Fusion (redundancy, equivalence) High heart rate variation ? High GSR ? Local Peak ? Fission (redundancy, equivalence) Sonic feedback Graphical feedback Haptic feedback Screenloudspeaker Vibrating wheel Dialog controller

5. 6/3/20155 Face motions analysis

6. 6/3/20156 Physiological signals “database” 3 subjects : thanks to participants! 3 situations : –rest / relaxation –Stress stimuli while reading a newspaper Hand clapping Light in eyes Answering simple question Telephone call, “your dead” (killer game) –Playing with the driving simulator (difficult tracks)

7. 6/3/20157 GSR signal

8. 6/3/20158 GSR signal

9. 6/3/20159 GSR signal

10. 6/3/201510 ECG signal : preprocessing

11. 6/3/201511 ECG : stress assessment

12. 6/3/201512 Simplified architecture of 1st prototype Significative rotation ? High frequency ? Both eyes closed ? Video Mouth opening detector ECG analysis Head rotation estimation Eyes blinks detector GSR analysis GSRECG Yawn ? Fusion (redundancy, equivalence) High heart rate variation ? High GSR ? Local Peak ? Fission (redundancy, equivalence) Sonic feedback Graphical feedback Haptic feedback Screenloudspeaker Vibrating wheel Dialog controller Network messages (XML) 1st PC, Matlab, Linux 2nd PC, Simulator, Windows

13. 6/3/201513 Realization of 1st prototype Significative rotation ? High frequency ? Both eyes closed ? Video Mouth opening detector ECG analysis Head rotation estimation Eyes blinks detector GSR analysis GSRECG Yawn ? Fusion (redundancy, equivalence) High heart rate variation ? High GSR ? Local Peak ? Fission (redundancy, equivalence) Sonic feedback Graphical feedback Haptic feedback Screenloudspeaker Vibrating wheel Dialog controller Network messages (text)

14. 6/3/201514 Conclusion First working prototype (demo available) –Firewire camera Matlab interface under Linux –Modification of the simulator (multi threading, TCP server, message display) –TCP client in Matlab First try with Open Interface integration  problem with Matlab integration  to be solved by Lionel next week Usefulness of physiological signals Reliable online face motions analysis Still a lot of work to do !!!