Introduction to Brain Computer Interface (BCI) Systems Todor Mladenov CSNL, GIST 2011

Outline Introduction Background EEG signals Noninvasive EEG methods BCI Applications Communication Issues Research Issues Conclusions

Introduction Machines that could be controlled by one's thoughts. Brain computer interface devices (BCI) detect and translate neural activity into command sequences for computers and prostheses. Electrodes recording from the brain are used to send information to computers so that mechanical functions can be performed. BCI devices aim to restore function in patients suffering from loss of motor control e.g. stroke, spinal cord injury, multiple sclerosis (MS) and amyotrophic lateral sclerosis (ALS). BCI will broaden repertoire of neurosurgical treatments available to patients previously treated by non-surgical specialists

Background 1875 - Richard Caton discovered electrical properties of exposed cerebral hemispheres of rabbits and monkeys. 1883 - Marxow discovers evoked potentials. 1924 - German Psychiatrist Hans Berger discovered alpha waves in humans and invented the term “electroencephalogram”(EEG). 1929 - Berger records electrical activity from the skull. 1936 - Gray Walter finds abnormal activity with tumors. 1950s - Grey Walter developed “EEG topography” - mapping electrical activity of the brain. 1970s - Research that developed algorithms to reconstruct movements from motor cortex neurons, which control movement. 1980s - Johns Hopkins researchers found a mathematical relationship between electrical responses of single motor-cortex neurons in rhesus macaque monkeys and the direction that monkeys moved their arms (based on a cosine function). 1990s - Several groups able to capture complex brain motor centre signals using recordings from neurons and use these to control external devices.

EEG Pioneers In 1929, Hans Berger Recorded brain activity from the closed skull Reportet brain activity changes according to the functional state of the brain Sleep Hypnothesis Pathological states (epilepsy) In 1957, Gray Walter Makes recordings with large numbers of electrodes Visualizes brain activity with the toposcope Shows that brain rhythms change according to the mental task demanded

What is EEG An electroencephalogram (EEG) is a measure of the brain's voltage fluctuations as detected from scalp electrodes. It is an approximation of the cumulative electrical activity of neurons.

Physical Mechanism EEGs require electrodes attached to the scalp with sticky gel Require physical connection to the machine

Electrode Placement (1) Standard “10-20 System” Spaced apart 10-20% Nasion – point between the forehead and the skull Inion – Bump at the back of the skull Letter for region F - Frontal Lobe T - Temporal Lobe C - Center O - Occipital Lobe Number for exact position Odd numbers - left Even numbers - right

Electrode Placement (2) International 10/20 system

Electrode Placement (3) The EEG measures not action potentials not summation of action potentials but summation of graded Post Synaptic Potentials (PSPs) (only pyramidal cells: dipoles between soma and apical dendrites)

EEG Channels Channel: Recording from a pair of electrodes (here with a common reference: A1 – left ear) Multichannel EEG recording: up to 40 channels recorded in parallel

Brain “Features” User must be able to control the output: use a feature of the continuous EEG output that the user can reliably modify (waves), or evoke an EEG response with an external stimulus (evoked potential)

Brain “Features” Generally grouped by frequency: (amplitudes are about 100µV max) Type Frequency Location Use Delta <4 Hz everywhere occur during sleep, coma Theta 5-7 Hz temporal and parietal correlated with emotional stress (frustration & disappointment) Alpha 8-12 Hz occipital and parietal reduce amplitude with sensory stimulation or mental imagery Beta 12-36 Hz parietal and frontal can increase amplitude during intense mental activity Mu 9-11 Hz frontal (motor cortex) diminishes with movement or intention of movement Lambda sharp, jagged occipital correlated with visual attention Vertex higher incidence in patients with epilepsy or encephalopathy

Brain Wave Transforms Wave-form averaging over several trials Auto-adjustment with a known signal Fourier transforms to detect relative amplitude at different frequencies seperates spontaneous EEG signal to component frequencies and amplitudes high frequency resolution demands long (in the range of seconds) analysis windows

Alpha Rhythm Frequency: 8 – 12 Hz Amplitude: 5 – 100 microVolt Location: Occipital, Parietal State of Mind: Alert Restfulness Source: Oscillating thalamic pacemaker neurons Alpha blockade occurs when new stimulus is processed

Beta Rhythm Frequency: 12 – 36 Hz Amplitude: 2 – 20 microVolt Location: Frontal State of Mind: Mental Activity Reflects specific information processing between cortex and thalamus

Delta Rhythm Frequency: 1 – 4 Hz Amplitude: 20 – 200 microVolt Location: Variable State of Mind: Deep sleep Oscillations in Thalamus and deep cortical layers Usually inibited by ARAS (Ascending Reticular Activation System)

Theta Rhythm Frequency: 5 – 7 Hz Amplitude: 5 – 100 microVolt Location: Frontal, Temporal State of Mind: Sleepiness Nucleus reticularis slows oscillating thalamic neurons Therefore diminished sensory throughput to cortex

Mu Waves Studied since 1930s Found in Motor Cortex Amplitude suppressed by Physical Movements, or intent to move physically (Wolpaw, et al 1991) trained subjects to control the mu rhythm by visualizing motor tasks to move a cursor up and down (1D) (Wolpaw and McFarland 2004) used a linear combination of Mu and Beta waves to control a 2D cursor. Weights were learned from the users in real time. Cursor moved every 50ms (20 Hz) 92% “hit rate” in average 1.9 sec

Alpha and Beta Waves Studied since 1920s Found in Parietal and Frontal Cortex Relaxed - Alpha has high amplitude Excited - Beta has high amplitude So, Relaxed -> Excited means Alpha -> Beta

Importance of EEG for HCI (1) People with disabilities may have no other option for HCI than BCI. Combining EEG with motion information to improve wearable activity recognition systems. Performance improvement in both humans and artificial systems strongly relies in the ability of recognizing erroneous behavior or decisions. EEG activity evoked by erroneous gesture recognition can be used as a feedback for HCI. [1] Hybrid Human Computer Interaction Systems (Hybrid – HCI). Thoughts recognition [2] Emotions recognition [3]

Importance of EEG for HCI (2) Recognizing the emotional state of a person and its thoughts are directly applicable to the automotive industry – automatic braking, sleep detection, tiredness level monitoring, anxiety and tension monitoring. Cars controlled by thoughts - AutoNOMOS Labs at Freie Universität Berlin. New possibilities for gaming experience utilizing EEG controls. Application of EEG systems in education. Example are the apps based on NeuroSky’s ( light a comfortable EEG sensors with custom fully integrated ASIC chip) equipment.

Reading the Brain (1) Direct Neural Contact ( Invasive ) Most accurate method Highly invasive Not possible with current technologies Perhaps possible in future with e.g. nanobots

Reading the Brain (2) Electroencephalogy (EEG) Non-invasive Measures electical activity in brain Non-invasive Susceptible to noise Easy to use + low cost + portable Most commonly used device in BCIs

Reading the Brain (3) Magnetoencephalogy (MEG) Measures magnetic fields produced by electrical activity in brain Non-invasive Very accurate High equiment requirements and maintenance costs

Reading the Brain (4) Functional Magnetic Resonance Imaging (fMRI) Measures blood flow in brain using MRI (haemodynamics) Blood flow correlates to neural activity Studies the brain‘s function Very accurate Very high costs due to MRI

Direct (noninvasive) interfaces in EEG The on-going electrical activity of the brain measured from scalp electrodes is called the electroencephalogram or EEG. An event-related potential (ERP) is any measured brain response that is directly the result of a thought or perception. More formally, it is any stereotyped electrophysiological response to an internal or external stimulus. Direct Interfaces via EEG VEP – Visual Evoked Potential SSVEP – Steady-State Visual Evoked Potential P300 – ERP elicited by infrequent, task-relevant stimuli. ERS/ERD – Event related synchronization/desynchronization

Event Related Potentials (ERP) Averaging of trials following a stimulus Noise reduction: The noise decreases by the squareroot of the number of trials Far field potentials require up to 1000 measurements Assumption: no habituation occurs (participants don‘t get used to stimulation)

Visual Evoked Potential (VEP) Caused by Visual Stimulus Occurs with flashing lights (3-5 Hz) Have been used to monitor function during surgery for lesions involving the pituitary gland, optic nerve and chiasma. Application:

Steady-State Visual Evoked Potential (SSVEP) SSVEP are signals that are natural responses to visual stimulation at specific frequencies. When the retina is excited by a visual stimulus ranging from 3.5 Hz to 75 Hz, the brain generates electrical activity at the same (or multiples of) frequency of the visual stimulus. Excellent signal-to-noise ratio and relative immunity to artifacts. Applications: SSVEP-controlled robots (Boston University) User-friendly interface ( NeuroSky) [4]

P300 (1) P300 is thought to reflect processes involved in stimulus evaluation or categorization. It is usually elicited using the oddball paradigm in which low-probability target items are inter-mixed with high-probability non-target (or "standard") items. Results in a positive curve on EEG after 300ms. Strongest signal at parental lobe.

P300 (2) (Farwell and Donchin 1988) 95% accuracy at 1 character per 26s

ERS/ERD (1) Event-related desynchronization (ERD) and event-related synchronization (ERS) is the change of signal's power occurring in a given band, relative to a reference interval. People have naturally occurring brain rhythms over areas of the brain concerned with touch and movement. When people imagine moving, these brain rhythms first become weaker, then stronger. These two changes are called ERD and ERS, respectively. ERS oscillatory power increase associated with activity decrease? ERD oscillatory power decrease associated with activity increase?

ERS/ERD (2) The imagination of either a left or right hand movement results in(5): An amplitude attenuation (Event-Related Desynchronization (ERD)) of μ (8-12Hz) and central beta EEG-rhythms (13-30Hz) at the contralateral sensorial motor representation area and, in some cases, in an amplitude increase (Event-Related Synchronization (ERS)) within the γ-band (30-40Hz) at the ipsi-lateral hemisphere(6). EEG recorded from C3 electrode.

ERS/ERD (3)

ERS/ERD (4) Time Time Pre-Stimulus Post-Stimulus Pre-Stimulus Typical Theta response Apha response ERS Amplitude Theta Amplitude Alpha ERD Time Time Pre-Stimulus Post-Stimulus Pre-Stimulus Post-Stimulus Stimulus Stimulus

ERS/ERD (5) ERD ERS Theta Alpha Decrease Increase Power Test power Reference power Decrease Increase Power Decrease Increase Power ERD ERS Reference power Test power

ERS/ERD (6) Decrease Increase Power

ERS/ERD (7) Decrease Increase Power Visuo-spatial semantic LTM task Semantic task: upper alpha relevant High activity (ERD red) is related to good performance Visuo-spatial semantic LTM task Decrease Increase Power Working and short term memory task upper alpha is less relevant Inhibition (ERS) of irrelevant semantic processes leads to good performance Visuo-spatial learning WM task

Communication Issues Typical training time versus communication bitrate for the three main types of noninvasive BCIs.

BCI Applications (1) For people with certain disabilities ( paraplegia, amyotrophia,.. ) BCI might be the only way of communication. Human enhancements: Cybernetic Organisms Brainwave Synchronization Exocortex ( intelligence booster) Human manipulation Mind-control “Neurohacking”: unwanted reading of information from brain. Neuroprosthetics: Surgically implanted devices used as replacement for damaged neurons.

BCI Applications (2) BCI – operated robot

BCI Applications (3)

BCI Applications (4)

BCI Applications (5)

Research Issues (1) 1. Make BCI easy, convenient and fun to use for the most popular consumer devices and apps.

Research Issues (2) 2. Energy budget analysis – Is it feasible to spend energy preprocessing data at the BCI headset in order to reduce the amount of transmitted data and save energy from the communication link. 3. Improve the signal to noise ration (SNR) for dry sensor electrodes. 4. Deliver EEG signals with medical systems quality for consumer applications and low price. New signal processing methods and algorithms.

Research Issues (3) 5. Currently, research is only beginning to crack the electrical information encoding the information in a human subject's thoughts. Understanding this “neural code” can have significant impact in augmenting function for those with various forms of motor disabilities.

Conclusions BCI research has tremendous implications to the field of medecine. Lot’s of on-going active research driven by the enormous interest. Consumer Electronics companies such as Emotive and Neurosky are coming up with user friendly headsets. New areas of application outside of medecine – gaming, control, education.

Thank you! Q&A

References “Adaptation of Hybrid Human-Computer Interaction Systems using EEG Error-Related Potentials”, Ricardo Chavarriaga, Andrea Biasiucci, Killian F¨orster, Daniel Roggen, Gerhard Tr¨oster and Jos´e del R. Mill´an “Translating Thoughts Into Actions by Finding Patterns in Brainwaves”, Charles W. Anderson and Jeshua A. Bratman “Towards Emotion Recognition from Electroencephalographic Signals”, Kristina Schaaff and Tanja Schultz “A user-friendly SSVEP-based brain–computer interface using a time-domain classifier”, An Luo and Thomas J Sullivan Pfurtscheller G., et al., 1993, Brain Computer Interface a new communication device for handicapped people. Journal of Microcomputer Applications, 16:293-299. Neuper, C. et al., 1999. Motor imagery and ERD. Related Desyncronization, Handbook of Electroencepalography and Clinical Neurophysiology Vol. 6. Elsevier, Amsterdam, pp. 303-525. Grabner, R. H., Stern, E., & Neubauer, A. C. (2003). When intelligence loses is impact: neural efficiency during reasoning in a familiar area. International Journal of Psychophysiology, 49, 89-98. Brain-Computer Interfaces, Fabien Huske, Markus A. Kollotzek, Alexander Behm EEG/MEG: Experimental Design & Preprocessing, Lena Kastner, Thomas Ditye Classic EEG (ERP) / Advanced EEG, Quentin Noirhomme The ElectroEncephaloGram, Cognitive Neuropsychology, January 16th, 2001 Brain-Computer Interface, Overview, methods and opportunities, Emtiyaz (Emt), CS, UBC The emerging world of motor neuroprosthetics: a neurosurgical perspective, Neurosurgery. 2006 Jul; 59(1):1-14. Workshop on direct brain/computer interface & control, Febo Cincotti, August 2006

List of Abbreviations EEG – Electroencephalography ERP – Event Related Potential ERS – Event-Related Synchronization ERD – Event-Related Desynchronization HCI – Human Computer Interface BCI – Brain Computer interface VEP – Visual Evoked Potential SSVEP – Steady-State Visual Evoked Potential P300 – An ERP signal type ECG – Electrocardiography EMG – Electromyography fMRI – Functional Magnetic Resonance Imaging MEG – Magnetoencephalogy