Presentation is loading. Please wait.

Presentation is loading. Please wait.

Using a Low-Cost Electroencephalograph for Task Classification in HCI Research Johnny C. Lee Carnegie Mellon University Desney S. Tan Microsoft Research.

Similar presentations


Presentation on theme: "Using a Low-Cost Electroencephalograph for Task Classification in HCI Research Johnny C. Lee Carnegie Mellon University Desney S. Tan Microsoft Research."— Presentation transcript:

1 Using a Low-Cost Electroencephalograph for Task Classification in HCI Research Johnny C. Lee Carnegie Mellon University Desney S. Tan Microsoft Research UIST 2006, Montreux Switzerland

2 National Geographic, March 2005 NY Times Magazine, October 16, 2005

3 Brain-Computer Interfaces (BCI) A direct technological interface between a brain and computer not requiring any motor output from the user Example Conferences/Journals with BCI interests: Neural Information Processing Systems (NIPS) IEEE Transactions on Biomedical Engineering IEEE Transactions on Neural Systems and Rehabilitation Engineering

4 Why is this relevant to UIST or HCI? BCI research traditionally focuses on exploratory neuroscience and rehabilitation engineering. Brain sensing could provide valuable data about: - engagement - cognitive work load - surprise - satisfaction - frustration Potentially helpful to Context Sensitive or Evaluation Systems

5 Values of BCI Values of HCI VS To use any means necessary to demonstrate that brain-computer interaction is possible. To use reasonable means to achieve a practical benefit to many users. use equipment costing $100K to +$1 million USD use highly invasive surgical procedures require hours or days of operant conditioning remove data from poor performing subjects It is okay to: use fairly affordable and accessible equipment be safe for repeated and extended use be usable without requiring significant user training use data from all subjects to evaluate its performance Wed like to:

6 Where do we start?

7 Brain Sensing/Imaging Technologies MRI CT ECoG SPECT PET MEG fMRI EROS/fNIR EEG Currently Impractical for HCI - Safe, easy, no medical expertise

8 EEG – Electroencephalograph the neurophysiological measurement of the electrical activity of the brain by recording from electrodes placed on the scalp (skipping the lower level neurophysiology) -Measures the voltage difference between two locations on the scalp -Only picks up gross, macroscopic, coordinated, and synchronized firing of neurons near the surface of the brain with perpendicular orientation to the scalp. (thus majority of activity is hidden) Analogous to holding a thermometer up to the side of a PC case

9 EEG Devices Manufacturer: BioSemi Channels: 64-128 Cost: ~$30K USD Manufacturer: EGI Systems Channels: 128-512 Cost: $100K-$250K USD

10 Lowest cost FDA approved device Designed for home and small clinical use. Only $1500 USD Specs: –2-channels –8-bit at 4µV resolution –256 samples/sec Has yet to be validated for BCI research work. If it works, it lowers the entry bar for BCI research. The Brainmaster

11 Validating the Device (and ourselves) 1. Validate the device Can we get useful data from such a low-end device? 2. Validate ourselves To explore this space, we must be able to collect our own data.

12 Validating the Device (and ourselves) Keirn, Z., A New Mode of Communication Between Man and His Surroundings, IEEE Transactions on Biomedical Engineering, Vol. 37, No. 12, 1990. Data is available for download Data has not been reproduced in the past 15 years –Some computational BCI researchers have just used this data. –State of the art does is not a great deal better.

13 Reproducing the Keirn Data We adapted procedure from Keirn to better control potential confounds. 3 tasks: Rest (Baseline): Relaxation and clearing of mind Math: Mental arithmetic, prompted with 7 times 3 8 5 Rotation: Mentally rotate an object, prompted with peacock Tasks from the original paper were designed to elicit hemispheric differences.

14 Experimental Procedure User is instructed to keep eyes closed, minimize body movement, and not to vocalize part of the tasks. For each task, a computer driven cue is given: Rest, Math, Rotate Following Math and Rotate, the experimenter says either the math problem or object

15 Experimental Procedure RestMathRot RestMath RestRotMath RotMathRest MathRestRot MathRotRest session task (14 seconds) trial Block design adapted from Kiern

16 Experimental Procedure RestMathRot RestMath RestRotMath RotMathRest MathRestRot MathRotRest MathRot RestMath RestRotMath RotMathRest MathRestRot MathRotRest MathRot RestMath RestRotMath RotMathRest MathRestRot MathRotRest 3 sessions per subject Many short tasks prevent correlation with EEG drift

17 Experimental Procedure RestMathRot RestMath RestRotMath RotMathRest MathRestRot MathRotRest MathRot RestMath RestRotMath RotMathRest MathRestRot MathRotRest MathRot RestMath RestRotMath RotMathRest MathRestRot MathRotRest Subjects: 8 subjects (3 female) 29-58 years of age All were cognitively and neurologically healthy All right handed

18 EEG Setup International 10-20 EEG electrode placement system Two channels placed on P3 and P4 with both references tied to Cz. Electrodes are held in place using conductive paste. 5-10 minute preparation.

19 Processing the Data

20 Data Processing 14 secs Rot task (14 seconds)

21 Data Processing 14 secs Task Cue Rot task (14 seconds)

22 Data Processing 14 secs Experimenter Prompt Rot task (14 seconds)

23 Data Processing 14 secs Task Onset Rot task (14 seconds)

24 Data Processing 14 secs Performing Task Rot task (14 seconds)

25 Data Processing 14 secs Performing Task ~4 secs Rot task (14 seconds)

26 Data Processing 10 secs Performing Task Rot task (14 seconds)

27 Removing time for machine learning Most machine learning algorithms dont handle time series data very well. 10 seconds

28 Removing time for machine learning Divide the 10 seconds into 2 sec windows that overlap by 1 sec Perform signal processing on each of the 9 windows to get our time less feature set 2 secs

29 Removing time for machine learning Divide the 10 seconds into 2 sec windows that overlap by 1 sec Perform signal processing on each of the 9 windows to get our time less feature set 2 secs

30 Removing time for machine learning Divide the 10 seconds into 2 sec windows that overlap by 1 sec Perform signal processing on each of the 9 windows to get our time less feature set 2 secs

31 Removing time for machine learning Divide the 10 seconds into 2 sec windows that overlap by 1 sec Perform signal processing on each of the 9 windows to get our time less feature set 2 secs

32 Removing time for machine learning Divide the 10 seconds into 2 sec windows that overlap by 1 sec Perform signal processing on each of the 9 windows to get our time less feature set 2 secs This provides 486 windows per participant

33 Signal features for each window Generic signal features such as mean power, peak frequency, peak frequency amplitude, etc. Features frequently used in EEG signal analysis.

34 Common EEG Features Raw EEG Spectral Power Gamma (30Hz-50Hz) Beta High (20Hz-30Hz) Beta Low (12Hz-20Hz) Alpha (8Hz-12Hz) Theta (4Hz-8Hz) Delta (1Hz-4Hz)

35 Feature Processing and Selection The 39 base features from each window are mathematically combined to create 1521 total features. We used a feature preparation and selection process similar to [Fogarty CHI05] to reduce the number of features: 23 features for 3-task classification (486 examples) 16.4 features for pair-wise classification (324 examples)

36 Baseline Results – 3 cognitive tasks 3 task Math v. Rotate Rest v. Math Rest v. Rotate user 167.9%83.3%88.0%85.8% user 270.6%82.7%91.4%84.3% user 377.6%88.3%93.8%86.7% user 463.6%69.4%84.9%86.7% user 566.5%91.0%81.2%80.9% user 659.3%80.6%80.2%68.5% user 771.4%87.3%90.4%86.7% user 869.8%87.7%82.4%83.6% Average 68.3%83.8%86.5%82.9% BayesNet classifier Chance: 33.3% 50% 50% 50%

37 2 secs 86.5% 68.3% 83.8% 82.9 %

38 We can do better… ???

39 Throwing time back in… Math We can average over temporally adjacent windows to improve classification accuracy

40 Averaging with Task Transitions Task transitions result in conflicting data in averaging window. High density of transitions will result in lower accuracy.

41 Averaging with Task Transitions Fewer task transitions will yield better classification accuracy.

42 Averaging with Task Transitions No transitions and averaging over all data will be the even better.

43 Classification Accuracy with Averaging +5.1 to +15.7% for 3-tasks Error bars represent standard deviation

44 So, can we really read minds? Error bars represent standard deviation Possibly not, we might be really detecting subtle motor movements….

45 Cognitive/Motor Fabric Does this matter to neuroscience?Yes Does this matter to HCI? Maybe not Tasks of varying cognitive difficultly are involuntarily coupled with physiological responses, such as minute imperceptible motor activity. [Kramer 91] Therefore, it is impossible to completely isolate cognitive activity neurologically intact individuals.

46 Cognitive/Motor Fabric If motor artifacts are reliably correlated with different types of tasks or engagement, why not use those to help the classifier? Requiring users to not move is also very impractical. Non-Cognitive Artifacts detected by EEG: –Blinking –Eye movement –Head movement –Scalpal GSR –Jaw and facial EMG –Gross limb movements –Sensory Response Potentials

47 Experiment 2 – Game Task To explore this idea of using non-cognitive artifacts to classify tasks using EEG, we chose a PC-based video game task. Halo, a PC-based first person shooter game produced by Microsoft Game Studios. Navigate a 3D environment in an effort to shoot opponents using various weapons. Relatively high degree of interaction with mouse and keyboard input controls.

48 Game Tasks Rest – baseline rest task, relax, fixate eyes on cross hairs on center of screen, do not interact with controls. Game elements do not interact with participant. Solo – navigate environment, interact with elements in the scene, and collect ammunition. Opponent controlled by expert did not interact with participant. Play – navigate environment and engage opponent controlled by expert. Expert instructed to play at a level just slightly above skill of participant to optimally challenge them.

49 Game Experimental Procedure Setup, design, and procedure was similar to first study. Participants had tutorial and practice time with game controls. 3 tasks repeated 6 times (counterbalanced) Tasks were 24 seconds to allow navigation time. Only 2 sessions were run for each participant Same 8 participants from first study were run in this study. Same data preparation and machine learning procedure.

50 Results – Game Tasks Error bars represent standard deviation 93.1%

51 Conclusion This experimental design and data processing procedure can be applied to a much wider range of applications/tasks. Our two experiments were just two examples at different ends of a spectrum. Compelling results can be achieved with low-cost equipment and without significant medical expertise or training. Non-cognitive artifacts (inevitable in realistic computing scenarios) can be embraced improve classification power. To make BCI relevant to HCI, we must challenge traditional assumptions and creatively work with its limitations.

52 Thanks! Johnny Chung Lee johnny@cs.cmu.edu Desney Tan desney@microsoft.com Thanks to MSR and the VIBE Group for supporting this work.

53 Cross-user Classifier

54

55 Brain Sensing/Imaging Technologies MRI – only anatomical data CT – only anatomical data ECoG SPECT PET MEG fMRI EROS/fNIR EEG

56 Brain Sensing/Imaging Technologies MRI – only anatomical data CT – only anatomical data ECoG – highly invasive surgery SPECT – radiation exposure PET – radiation exposure MEG fMRI EROS/fNIR EEG

57 Brain Sensing/Imaging Technologies MRI – only anatomical data CT – only anatomical data ECoG – highly invasive surgery SPECT – radiation exposure PET – radiation exposure MEG – extremely expensive fMRI – extremely expensive EROS/fNIR EEG

58 Brain Sensing/Imaging Technologies MRI – only anatomical data CT – only anatomical data ECoG – highly invasive surgery SPECT – radiation exposure PET – radiation exposure MEG – extremely expensive fMRI – extremely expensive EROS/fNIR – currently expensive, still in infancy EEG – safe, easy, no medical expertise

59 Other cool things you can do with an EEG device…

60 Event Related Potentials (ERP) Electrical activity related to or in response to the presentation of a stimulus Very well studied Relatively robust Used daily in clinical settings to check sensory mechanisms, typically in infants Requires averaging over 30-100 windows synchronized with to see response.

61 ERP - AEP ERP: Auditory Evoked Potential Used in clinics/hospitals to check hearing. Response to clicks in the ear Bold Lines = no clicks Thin Line = with clicks AEP response

62 ERP - VEP ERP: Visual Evoked Potential Focusing on a flashing target, the visual cortex will resonate at the stimulus frequency. Stimulus Frequency Harmonics

63 ERP – Auditory and Visual P300 Well known/studied potential related to attention or surprise Presented with 2 stimuli and instructed to count one of the stimuli Positive response will follow the stimulus of interest.

64 Side note: EEG as ECG ECG - Electrocardiogram placing an electrode on the chest provides a clear measure of cardiac activity. translation to BPM is a simple autocorrelation Heart beats Single Beat period 0.4 µV units

65 EEG as EMG EMG - Electromyography Measures muscular activity Wrist relaxation (return to straight position) Tension holding Wrist inward contraction (toward inner forearm) Wrist rest state NOTE: The magnitude of the spikes seem to be proportional to the acceleration involved with the movement. 0.4 µV units

66 EEG as Blink Detector Electrical activity due to muscle movement involved with eye blinks propagate through the head. Similarly, eye movements also affect the EEG recording Blinks

67 Task Classification Background Previous work is split primarily into two camps: Operant ConditioningPattern Recognition

68 Task Classification Background Previous work is split primarily into two camps: Operant ConditioningPattern Recognition Human learns how the machine works Machine learns how the human works

69 Task Classification Background Previous work is split primarily into two camps: Operant ConditioningPattern Recognition Human learns how the machine works Machine learns how the human works Relatively new Early dabbling in the late-80s Most work has happened in last 5 years.

70 Pattern Recognition Data Collection & Experimental Design Signal Processing & Feature Generation Machine Learning & Improving Accuracy

71 EEG Setup International 10-20 EEG electrode placement system Two channels placed on P3 and P4 with both references tied to Cz. Locations selected based on pilot recordings. Attaching electrodes: Prepare the site with a cleaner, use conductive paste to improve connection and hold electrode in place. P Paste rinses out with water, non-toxic. 5-10 minute preparation.


Download ppt "Using a Low-Cost Electroencephalograph for Task Classification in HCI Research Johnny C. Lee Carnegie Mellon University Desney S. Tan Microsoft Research."

Similar presentations


Ads by Google