1. Department of Computer Science Development and Application of an EEG-based Brain- Computer Interface By Aleksey Tentler Committee: Chair: Dr. Jessica Bayliss Reader: Dr. Roger Gaborski Observer: Dr. Ankur Teredesai

2. Department of Computer Science Overview Introduction Related Work System Development Experiments Problems Encountered Future Work Summary

3. Department of Computer Science Introduction There exist diseases of the nervous system that gradually cause the body’s motor neurons to degenerate –Example: Amyotropic Lateral Sclerosis (ALS) Eventually causes total paralysis The affected individual becomes trapped in his own body, unable to communicate

4. Department of Computer Science Introduction (continued) A Brain-Computer Interface (BCI) enables communication under such circumstances Using data recorded from the brain, the BCI processes it, interprets the intention of the user, and acts on it.

5. Department of Computer Science Thesis Work We develop and demonstrate a simple version of a BCI, capable of detecting an answer to a Yes/No question –The BCI has a robust and flexible design that can be expanded in the future to encompass more complex communication schemes. We present a panel of experiments that attempt to determine what effect, if any, the methods of stimulus presentation have on communication accuracy

6. Department of Computer Science BCI defined A BCI generally consists of three main components: 1.A Signal-acquisition module 2.A Signal-processing module 3.A Control module

7. Department of Computer Science BCI defined (continued) Signal-acquisition module 1.Records data from the brain 2.Does some low level filtering 3.Passes the data on to be interpreted

8. Department of Computer Science BCI defined (continued) Signal-processing module 1.Receives data from the acquisition module 2.Detects whether a particular feature or a potential is present in the data

9. Department of Computer Science BCI Defined (Continued) Control module 1.Anything from a hardware device to a software application on a computer screen 2.Gives feedback to the user 3.Can present him with choices of commands or actions at his disposal

10. Department of Computer Science BCI Defined (Continued) Control module (continued) 4.Can communicate with the signal acquisition and/or signal processing to control when to start recording and when to expect a particular feature to occur 5.Can, in some cases, also perform some analysis and/or display of the data to increase the accuracy of detection –Functions in conjunction with or instead of the signal- processing module.

11. Department of Computer Science Signal Acquisition Signal Processing Control Device FeedbackData from Brain Filtered Data Detected Features Control (Optional) Control (Optional) Filtered Data (optional) A Model of a generalized BCI BCI Defined (Continued)

12. Department of Computer Science BCI Defined (Continued) BCI Types: –Dependent –Uses the activity in the brain’s normal output pathways –Independent –Not based on the brain’s normal output pathways –Based on cognitive brain activity http://www.innovativeoandpofla.com/p rosthetic_Arm_Pictures.htm

13. Department of Computer Science BCI Defined (Continued) BCI Types: –Invasive –Involves attaching electrodes directly to the brain tissue –The patient’s brain gradually adapts its signals to be sent through the electrodes –Non-invasive Involves putting electrodes on the scalp of the patient, and taking readings http://spiderman.sonypictures.com/downloads/ http://www.tcnj.edu/~leynes/lab/tour.html

14. Department of Computer Science BCI Defined (Continued) Non-invasive EEG BCI –Poor spatial resolution –High temporal resolution –Relatively inexpensive equipment

15. Department of Computer Science BCI Defined (Continued) Non-invasive EEG BCI –Signal Types: –Signals requiring training by operant conditioning –Slow cortical potentials, alpha, mu, and beta rhythms –Signals not requiring training –P3 evoked potentials

16. Department of Computer Science Non-invasive EEG P3 BCI P3 potential types: 1.P3a –Evoked whenever a novel, unexpected, and unrecognizable stimulus is inserted into a random sequence of frequent and infrequent stimuli SeptemberFebruaryDecemberMarchJulyNovember 2.П3б –Наблюдается когда появляется что-то ожидаемое, имеющее отношение к тому над чем человек думает 2.P3b –Evoked when a task-relevant stimulus that has been expected occurs in a random sequence of frequent and infrequent stimuli

17. Department of Computer Science Non-invasive EEG P3b BCI P3b General Form A positive wave that peaks about 400 ms after a visual stimulus and about 300 ms after an auditory stimulus Might be less pronounced, or completely absent, if the user is not focused on the task http://p300.scripps.edu/default.htm

18. Department of Computer Science Non-invasive EEG P3b BCI Accuracy Issues –Software-dependent –Affected by how well the software can classify a P3 potential when it’s present in the data –User-dependent –Affected by whether the user is focused on task –Habituation is possible

19. Department of Computer Science Related Work Wadsworth Brain-Computer Interface Center, Albany –Use signals such as mu or beta rhythms that require training, which is often difficult and takes time. –Developed an application called BCI2000 University of South Florida –E. Donchin –Using the BCI2000 software –Working with the P3 evoked potentials

20. Department of Computer Science Related Work (continued) P. Kennedy, Atlanta and by J. Donoghue, Brown University –Invasive methods, requiring surgery –Allows the patient to directly interface with a computer through implanted electrodes –Requires a long training period

21. Department of Computer Science Demand for Software Currently working with an ALS patient –Tried to get help from Wadsworth, USF, and Atlanta It seems that the current available software does not meet all the necessities of flexibility, stability, and accuracy

22. Department of Computer Science Software Development Designed as a platform framework –Allows future designers and researchers to easily add or replace modules as needed –Don’t have to be knowledgeable in the low- level signal handling and transfer

23. Department of Computer Science BCI Structure Acquisition Module (Server) Signal Processing Module (Client) User Application (Client) Event Time Event Epoch Signal Detected (as a short) Event Epoch Signal Detected (as a short) EEG Visualization Module (Client) Event Epoch

24. Department of Computer Science Acquisition Module Acquisition GUI Acquisition Thread EEG Data Server and Event Receiver Listeners (over network) User Application (over network) Hardware Circuit & Filter

25. Department of Computer Science Signal Processing Module Data Receiver / Signal Sender Client Signal Processor Signal Processing Console Acquisition Module (over network)

26. Department of Computer Science User Application Module Application Thread Event Sender / Signal Receiver Client Application GUI Acquisition Module (over network) BCIGui Specific App handleEvent() handleData() handleSignal() handleDecision()

27. Department of Computer Science Experiments Subjects Procedure Online Analysis Offline Analysis Results

28. Department of Computer Science Subjects Nine total subjects –8 high-school to college-age males –1 female ALS patient To increase the motivation of non-ALS subjects, upon the completion of the experiment they were given candy or the opportunity to play computer games in the AI Lab.

29. Department of Computer Science Procedure The background and goals of the experiment were explained to each subject. The subject was fitted with an electrode cap and face electrodes

30. Department of Computer Science Procedure (continued) In the experiment, the subject was asked to use the three application GUIs The order of use of each GUI was determined randomly.

31. Department of Computer Science GUIs 2-Button Application 1-Button Application

32. Department of Computer Science Procedure (continued) For each GUI, seven trials were conducted consecutively Each trial was composed of 34 total flashes of the stimuli. The target response for each trial alternated between “Yes” and “No”.

33. Department of Computer Science Procedure (continued) For each trial, the subject was told they can either: –Count the number of times that the target response flashed –Pick a question which had the target response as the answer, and visualize achieving the answer each time the target response flashed. Before the official trials started, the subject was given 4 sample runs to get used to the system

34. Department of Computer Science Procedure( continued) At the end of each trial, the system’s response was shown to the subject. At the end of the experiment, the subject was asked for his/her opinion on which of the three GUIs was the easiest or hardest to use and for what reason(s)

35. Department of Computer Science Online Analysis The system presented 34 total events to the user, keeping track of the average response epoch for each event type. Epochs in which the eye channel electrodes showed peaks over 80  v or below –80  v were not included in the average –Avoided false detections and misdetections due to eye movements.

36. Department of Computer Science Online Analysis (continued) After all the events have been flashed, the system would analyze the two average event type epochs, and try to determine which one of them seemed closer to a P3 potential. Examined several measurements of the two epochs –Positive peak at about 400 ms after the stimulus –Negative peak before the positive peak at about 300 ms –Positive peak before the negative peak at about 100 ms

37. Department of Computer Science Online Analysis (continued) Compare the maximums/minimums and means of the two average epochs for those areas Each comparison of a mean or a maximum/minimum counted as one “vote”.

38. Department of Computer Science Online Analysis (continued) Compare the ratio of votes for each possible answer to make a decision If the numbers of votes for each epoch were equal, the system would announce that no answer was detected.

39. Department of Computer Science Offline Analysis Data was further analyzed offline using MATLAB –Used several more analysis techniques: Peak-picking Peak aligning –Attempt to get better support for the results and to analyze which, if any, analysis methods work better than others.

40. Department of Computer Science Results with the Max/Min Finding Algorithm Overall accuracy - 93.46 % 90 % for the 1-Button GUI 92.31 % for the 2-Button GUI 98.04 % for the 2-Button Novelty GUI 93.62% for the first GUI 90.74% for the second GUI 96.15% for the third GUI.

41. Department of Computer Science Results

42. Department of Computer Science Undecided Rates

43. Department of Computer Science Overall Grand Average

44. Department of Computer Science GUI Averages by Type

45. Department of Computer Science GUI Averages by Order Figure 15: Overall P3 Grand Average Figure 16: 1-Button Grand Averages Figure 17: 2-Button Grand Averages Figure 18: 2-Button Novelty Grand Averages Figure 19: First GUI Grand Averages Figure 20: Second GUI Grand Averages

46. Department of Computer Science Peak-Picking Results % Positive identified correctly= = True Positive / (True Positive + False Negative) And % Negative identified correctly= =True Negative / (True Negative + False Positive)

47. Department of Computer Science % Positive identification = =True Positive / (True Positive + False Positive) And % Negative identification= = True Negative / (True Negative + False Negative)

48. Department of Computer Science Peak-Picking Results by Order

49. Department of Computer Science Discussion Objective Results were not statistically significant enough to prove difference in GUIs Subjects’ comments –1-button application was harder to use than the 2-button one and caused eye strain for 4 out of 5 subjects who felt a difference –For the 2-button GUI, once the participant knew which button was “yes” and which button was “no,” the distinction could be made at the slightest flash of the stimulus detected out of the corner of the eye –The 2-button GUI with the color change in between, was, according to the participants, almost the same as the regular 2- button GUI For some seemed to boost their attention (2 out of 5) For others distracting. (2 out of 5)

50. Department of Computer Science Problems Encountered Acquisition –With USB 1.0 connection, the system could not have access to streaming data, making it necessary to get the data 1 second at a time. Each data request sent to the acquisition board resulted in a minimal delay. –The computer system that was used had the Windows 2000 operating system on it Windows is not designed to be a real-time processing system and therefore, the absence of OS-generated delays could not be guaranteed.

51. Department of Computer Science Problems Encountered Signal-processing –The signal detection method depended on the P3 potentials being in the right place at the right time. –The method can fail in cases where for certain reasons the P3 potential is delayed. Could have occurred toward the end of each experiment as the participant was getting fatigued Also with certain types of GUIs, such as the 1-Button GUI, which required the participants to read the stimulus text before making a decision

52. Department of Computer Science Problems Encountered User application –Timing problems caused by delays that can be expected to be present on a Windows 2000 system –The flash of the stimulus often occurred later than the time that the command to flash was added to the system event queue. –The delay was variable and inconsistent, offsetting the detection algorithm.

53. Department of Computer Science Peak-Aligning An attempt to correct for the variable delay –Align peaks using the Visual Evoked Potential (VEP) –VEP occurs at around 100 ms after the stimulus

54. Department of Computer Science Peak-Aligning Results Overall accuracy – 88.67 % 86.53 % for the 1-Button GUI 93.33 % for the 2-Button GUI 86.31 % for the 2-Button Novelty GUI 77.48% for the first GUI 92.78% for the second GUI 95.93% for the third GUI

55. Department of Computer Science Peak-Aligning Results

56. Department of Computer Science Overall Grand Average

57. Department of Computer Science GUI Averages by Type

58. Department of Computer Science GUI Averages by Order

59. Department of Computer Science Undecideds Between Peak- Aligning and the Original Algorithm

60. Department of Computer Science Problems Encountered Other Issues –No easy way to tell the cause of an incorrect result Algorithm Error User Inattention –Visually analyzed epochs to improve algorithm –Custom fitted offline algorithm for each subject to avoid mistakes due to individual differences

61. Department of Computer Science Future Work Acquisition –Can be made to allow easy pluggability of modules that interface with different types of hardware –Use a photocell to avoid timing errors Signal Processing –A signal-processing algorithms should be created that can achieve a decision with as few stimulus flashes as possible –The module should be enhanced to allow the user to calibrate it to his/her individual brain waves and P3 potentials, making the module learn from past data and modify its detection routines accordingly

62. Department of Computer Science Future Work (continued) User Application –Better and more efficient User Applications should be developed –Displaying the choices to minimize the complexity of the task –Turning BCI On/Off –Use a graphics package instead of windows forms to get better control of when stimuli flash

63. Department of Computer Science Future Work (continued) Intermodule Communications –The networking communications should be analyzed to improve efficiency and increase the useful content that is passed across Visualization Module –The visualization module should be updated to be able to support the display of stimulus flashes and perhaps the detected P3 peaks

64. Department of Computer Science Future Work (continued) –Run more experiments with larger numbers of subjects –Other variable differences should be tested for Size of text and stimuli boxes Effect of one color vs. another Effect of flashing text vs. background –Determine what is the minimum number of flashes needed to achieve a recognizable P3

65. Department of Computer Science Future Work (continued) –Applications We are currently working with the couple that contacted us, to create a viable BCI for a patient with advanced ALS. We are also currently trying to make contact with other organizations that might be able to put us in touch with patients who would agree to participate in our experiments.

66. Department of Computer Science Conclusion We have created a robust, modular, BCI platform We have tested it by running a panel of experiments, using an application that was developed based on this platform. We have also set up a useable EEG/BCI Lab in the AI Lab, which, in the future, can be used both for continued development of BCI applications, and for other EEG/EMG research at RIT

67. Department of Computer Science References “Amyotropic Lateral Sclerosis (ALS)” WebMD Portal, November 25, 2002 (March 10, 2004) Bayliss, J. D., A Flexible Brain-Computer Interface, TR756 and Ph.D. Thesis, Computer Science Dept., U. Rochester, 2001

68. Department of Computer Science References Bayliss, J. D., Use of the evoked potential P3 component for control in a virtual apartment, IEEE Trans Neural Syst Rehabil Eng, 2003 Jun; Vol. 11 (2), pp. 113-6 Black, M.J., Serruya, M., Bienenstock, E., Gao, Y., Wu, W., and Donoghue, J.P. Connecting brains with machines: the neural control of 2D cursor movement. Proceedings of the First International IEEE EMBS Conference on Neural Engineering, 19-22 March 2003, Capri Island, Italy.

69. Department of Computer Science References R.M. Chapman and H.R. Bragdon, Evoked responses to numerical and nonnumerical visual stimuli while problem solving, Nature, v.203, pp. 1155—1157, 1964. Donchin, E., Spencer, K. M., & Wijesinghe, R. The Mental Prosthesis: Assessing the Speed of a P300-Based Brain-Computer Interface, IEEE Trans. Rehab. Engineering, 2000,8, 174-179 Gloor, Pierre; Hans Berger On the Electroencephalogram of Man: The Fourteen Original Reports on the Human Electroencephalogram, N. Elsevier Science Publishers, Amsterdam, 1969. Inverso, Samuel. Automatic Error Recovery Using P3 Response Verification for a Brain-Computer Interface, Masters Thesis. RIT Department of Computer Science, July 2004

70. Department of Computer Science References Katayama J; Polich J. Stimulus context determines P3a and P3b. Psychophysiology 1998 Jan; 35 (1), pp. 23-33. Kennedy P. R., Direct control of a computer from the human central nervous system. IEEE Trans Rehabil Eng, 2000 Jun; Vol. 8 (2), pp. 198-202 McFarland D. J. EEG-based communication and control: speed- accuracy relationships. Appl Psychophysiol Biofeedback, 2003 Sep; Vol. 28 (3), pp. 217-31 Ravden and Polich, 1999. D. Ravden and J. Polich, On P300 measurement stability, habituation, intra-trial block variation, and ultadian rhythms. Biological Psychology 51 (1999), pp. 59–76

71. Department of Computer Science References Riggins, Brock R.; Polich, John; Habituation of P3a and P3b from visual stimuli. Korean Journal of Thinking & Problem Solving, Vol 12(1), Apr 2002. pp. 71-81 Smith, E. J. “Introduction to EEG” ©2002 (June 12 th, 2004)

72. Department of Computer Science References S. Sutton and M. Braren and J. Zublin and E. John, Evoked potential correlates of stimulus uncertainty, Science, v.150, pp. 1187—1188, 1965. Wolpaw JR. Brain-computer interfaces for communication and control, Clinical Neurophysiology, 2002 Jun; Vol. 113 (6), pp. 767-91

73. Department of Computer Science Contact: Aleksey Tentler axt8828@cs.rit.edu Вопросы? שאלות? Fragen? Preguntas? Domande? Questions?