1. Closing conference of SYSIASS – June 17 th 2014 Multimodal Bio-signal based Control of Intelligent Wheelchair Professor Huosheng Hu Leader of Activity 3 University of Essex, U.K.

2. Closing conference of SYSIASS June 17 th 2014 SYSIASS: Autonomous and Intelligent Healthcare System 2

3. Closing conference of SYSIASS June 17 th 2014 Aim and Objectives  Multimodal Human-Machine Interface aims to make wheelchairs more user-friendly.  The key challenge is the understanding of  the user who interacts with the wheelchair  the system (the computer and sensor technology & their usability)  the interaction between the user and the wheelchair.  A proper balance between  Functionality that is defined by the set of actions or services that it provides to its users, and  Usability that is the range and degree by which the system can be used efficiently and adequately to accomplish certain goals for certain users. 3

4. Closing conference of SYSIASS June 17 th 2014 System Configuration 4 Multimodal HMI Voice, gesture, EMG, EEG

5. Closing conference of SYSIASS June 17 th 2014 Detection of Human Intension  Inertial Sensing: to sense the position and motion of the human hand and other body parts for use in HMI.  Audio Sensing: Using microphones to sense the sound is to interpret speech, which is the most natural modality.  Visual Sensing: Using cameras to detect the human motion, such as gestures, lip motion, gaze, facial expressions, head & other body movements.  Touch and force: This is especially important for building a proper feel of “realism” in human intension.  Muscle and Brainwave: These bio-signals are obtained noninvasively from the surface of the scalp and skin; used for wheelchair control. 5

6. Closing conference of SYSIASS June 17 th 2014 Wireless Voice based Control  To use human voice commands;  Pre-defined control phrases: “forward”, “turn left”, “turn right”, “backwards”, “stop”;  Wireless wearable headphone device;  Fast voice recognition and no pre- training  Plug and play wireless and comfort control  Obstacle avoidance with laser scanner  Unlimited voice control commands available 6 Video show

7. Closing conference of SYSIASS June 17 th 2014 Visual Head Gesture based Control  To help disabled and elderly who cannot use joystick  To realise hands-free control of the wheelchair  Obstacle avoidance to ensure user safety  Functionalities: 7  Visual detection of head gestures of users  Pre-defined commands (left, right, forward, backward, stop) for controlling a wheelchair  Laser based obstacle detection & avoidance Video show

8. Closing conference of SYSIASS June 17 th 2014 Facial Expression & Head Movement  Facial expressions are detected using the cognitiv suite of Emotiv.  Head movements are detected by the gyroscope of Emotiv.  Pre-defined commands (left, right, forward, backward, and stop)  A simple and safe hands-free control with flexible configurations for users.  An ideal alternative to the joystick control for users with severe disability. 8 2F+1H3F+1H

9. Closing conference of SYSIASS June 17 th 2014 Wireless IMU in a Baseball Cap  Detect head motion for wheelchair control  Minimum user head motion required  Adapt to people with weak neck abilities  Obstacle avoidance with laser scanner  Totally replace joystick control interface 9 Video show

10. Closing conference of SYSIASS June 17 th 2014 Minimally Invasive Intra-Oral Palate for wheelchair User  A minimally invasive way to control wheelchair  It is comfortable for users and allows prolonged use.  It is based on an intra-oral dental retainer clip with  9 force sensitive resistors  10 Bit 200Hz data capture  Opto-isolation is deployed.  Connectivity to GPSB 10

11. Closing conference of SYSIASS June 17 th 2014 Conformal RFID Sensing for wheelchair control  Strain gauge RFID tag is designed to sense the stretch on skin (neck/eyebrow movement)  A tongue controlled RFID tag is designed to sense the tongue position in the mouth  To provide an on/off switch and control mechanism for controlling the wheelchair. 11

12. Closing conference of SYSIASS June 17 th 2014 Conclusion  Human-Machine Interaction (HMI) aims to achieve information full accessibility.  Human-Machine Interaction plays an key role in assistive technology.  Hands-free control of wheelchairs are extremely useful to general public without programming skills.  Multimodal HMI is necessary to overcome the limitations of single modality HMI. 12  Activity 3 of SYSIASS has been conducted successfully.