1. Adaptive Stimulation Design for the Treatment of Epilepsy Joelle Pineau School of Computer Science, McGill University Montreal, QC CANADA Jointly with Robert Vincent, Aaron Courville, Massimo Avoli SAMSI Program on Challenges in Dynamic Treatment Regimes and Multistage Decision-Making June 21, 2007

2. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Background Vagus nerve and deep brain stimulation are used to treat various neurological disorders, including epilepsy. Images from www.epilepsyfoundation.org and www.cyberonics.com

3. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Project goal Problem: Existing devices offer limited control and do not adapt to the patients condition. Idea: Create an improved class of devices with closed-loop control.

4. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy A reinforcement formulation Objective: Minimize occurrence of seizures and overall amount of stimulation. The MDP model: –States, s t : recordings of electrical activity –Actions, a t : stimulation (frequency, voltage, location) –Transitions, P(s t |s t-1, a t ) : unknown –Rewards, r t : large cost for seizures, small cost for stimulation s t-1 atat a t+1 … r t-1 stst rtrt s t+1 r t+1 a t-1

5. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Technical challenges 1.Investigate supervised learning methods for automatic seizure detection to inform choice of good state representation. 2.Design a computational (generative) model of epilepsy. 3.Run reinforcement learning using online data from the computational model. 4.Run reinforcement learning using batch data from an in-vitro model.

6. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Recordings of electrical activity Recorded from single sensing electrode in in-vitro model of epilepsy. Raw data: 4096-sample frames windowed, normalized, FFT x t = 83 real-valued features: mean, range, energy, 80 FFT magnitudes y t ={normal, spike, seizure} hand-labeled for each frame.

7. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Seizure detection Long literature on seizure detection using supervised learning (e.g. RBFs, wavelets, neural networks, energy methods). »Usually uninterpretable results. Related literature in time-series prediction (e.g. HMMs, CRFs). »Requires feature selection. Instead, focus on ensemble boosting methods: –Extend standard multi-class Adaboost [Schapire&Singer, 1999] to a recurrent formulation, in which y t = f (x t, y t-1, y t-2, …, y t-k ). »Relatively interpretable results (assuming simple learners). »Requires no prior model of the distribution of features.

8. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Classification accuracy y t = f (x t, y t-1 )

9. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Feature extraction = insight into state design In all recurrent examples, first weak hypothesis recruited was frequency band 62 or 63 (= 76-77 Hz). –High value normal –Low value spike Frequency bands 6-8 (=7-10 Hz) also consistently recruited early. –High value spike –Low value normal Often recruited in the first 20 rounds is Energy. –High value spike In recurrent Adaboost, prior label often recruited early and acts as memory. –High spike prior spike etc.

10. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Technical challenges 1.Investigate supervised learning methods for automatic seizure detection to inform choice of good state representation. 2.Design a computational (generative) model of epilepsy. 3.Run reinforcement learning using online data from the computational model. 4.Run reinforcement learning using batch data from an in-vitro model.

11. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy A computational model of epilepsy Aims of computational modeling: –To understand basic mechanisms of epilepsy sufficiently to design a good state representation. –To understand the appropriate class of policies to consider. –To have an inexpensive testing environment for RL algorithms. A word of caution: –Epilepsy is a complex disease. Many animal models are found in the literature. We focus on model by [ Avoli et al., 2002], which we will use for online exploration in-vitro.

12. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Model overview (a) Assume a simple stochastic neuron model (leaky integrate-and-fire). (b) Connect many neurons in small-world network configuration. (a) Leaky integrate-and-fire neuron model(b) Network structure

13. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Model overview (contd) (c) Define a sensor model: –Voltage measurement over a patch of adjacent neurons (neuron contribution falls off in inverse-square relationship with distance from the patch center). (d) Define the stimulation model: –Input current applied uniformely to a patch of neural units. i stim v sensor

14. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Traces from the computational model # neurons firing sensor voltage

15. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Results from fixed stimulation strategies 0 Hz 0.5 Hz 1.0 Hz 2.0 Hz 4.0 Hz 5.0 Hz

16. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Technical challenges 1.Investigate supervised learning methods for automatic seizure detection to inform choice of good state representation. 2.Design a computational (generative) model of epilepsy. 3.Run reinforcement learning using online data from the computational model. 4.Run reinforcement learning using batch data from an in-vitro model.

17. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Reinforcement learning agent States: s t = {sensor reading} x {# time steps since last stimulation} discretized into small number of (independent) features Actions: a 1 = Stimulator on for 1 time-step. a 2 = Stimulator off for 1 time-step. Rewards: R(s,a) = -100for a seizure (i.e. firing count > 50) R(s,a) = -10for a stimulation R(s,a) = 0otherwise Training details: »Online data, finite horizon (60 x 100-sec traces), -greedy exploration, Sarsa( ) with eligibility

18. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Results: Learning (a) After 1 learning episode(b) After 10 learning episodes

19. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Results: Expected return

20. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Results: Policy

21. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Technical challenges 1.Investigate supervised learning methods for automatic seizure detection to inform choice of good state representation. 2.Design a computational (generative) model of epilepsy. 3.Run reinforcement learning using online data from the computational model. 4.Run reinforcement learning using batch data from an in-vitro model.

22. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Learning from batch in-vitro data We have data of the type used in the classification task, but which includes fixed-policy stimulation. States: 83 real-valued features (electrical signal sampled + FFT) Actions: {0 Hz, 0.5 Hz, 1.0 Hz, 2.0 Hz, 5.0 Hz} Training: batch (~30 x 60-sec traces), tree-based regression. No results

23. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Wrap-up Preliminary evidence for: –ability to detect seizures using in-vitro data. –generation of synthetic data with epileptiform behavior. –controllability of the stimulator in the computational model. Results apply to a specific model of epilepsy - generalization to other models is unknown. Issues I did not discuss today: –Designing the reward function. –Using the features found in Adaboost within the RL agent. –Transferring what we have learned in the computational model to the biological model. –Learning from very few data points.

24. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy Available data Recordings used for classification (no stimulation) –http://www.cs.mcgill.ca/~jpineau/datasets/epilepsy.tar.gz Recordings used for batch reinforcement learning (fixed policy stimulation) –Subject to approval by neuroscientists (1-2 weeks delay). Generative model of epilepsy –Still under development, but publicly available soon (1-2 months).