1. Modeling fMRI data generated by overlapping cognitive processes with unknown onsets using Hidden Process Models Rebecca A. Hutchinson (1) Tom M. Mitchell (1,2) (1) Computer Science Department, Carnegie Mellon University (2) Machine Learning Department, Carnegie Mellon University Statistical Analyses of Neuronal Data (SAND4), May 30, 2008

2. Hidden Process Models HPMs are a new probabilistic model for time series data. HPMs are designed for data that is: –generated by a collection of latent processes that have overlapping spatial-temporal signatures. –high-dimensional, sparse, and noisy. –accompanied by limited prior knowledge about when the processes occur. HPMs can simultaneously recover the start times and spatial-temporal signatures of the latent processes.

3. t t d1 … dN … Process 1: t t d1 … dN … Process P: … d1 … dN Prior knowledge: An instance of Process 1 begins in this window. An instance of Process P begins in this window. An instance of either Process 1 OR Process P begins in this window. There are a total of 6 processes in this window of data. Example

4. Simple Case: Known Timing T D = 1 0 0 0 0 0 0 0 0 0 1 0 1 0 0 1 0 0 0 0 1 0 1 0 0 1 0 0 0 0 0 0 1 0 0 1 … … … p1p3p2 p1 p3 p2 D W(1) W(2) W(3) Y Apply the General Linear Model: Y=XW Data Y Convolution Matrix XUnknown parameters W [Dale 1999]

5. Challenge: Unknown Timing T D = 1 0 0 0 0 0 0 0 0 0 1 0 1 0 0 1 0 0 0 0 1 0 1 0 0 1 0 0 0 0 0 0 1 0 0 1 … … … p1p3p2 p1 p3 p2 D W(1) W(2) W(3) Y Uncertainty about the processes essentially makes the convolution matrix a random variable.

6. fMRI Data … Signal Amplitude Time (seconds) Hemodynamic Response Neural activity Features: 10,000 voxels, imaged every second. Training examples: 10-40 trials (task repetitions).

7. Goals for fMRI To track cognitive processes over time. –Estimate process hemodynamic responses. –Estimate process timings. Allowing processes that do not directly correspond to the stimuli timing is a key contribution of HPMs! To compare hypotheses of cognitive behavior.

8. Our Approach Model of processes contains a probability distribution over when it occurs relative to a known event (called a timing landmark). When predicting the underlying processes, use prior knowledge about timing to limit the hypothesis space.

9. Study: Pictures and Sentences Task: Decide whether sentence describes picture correctly, indicate with button press. 13 normal subjects, 40 trials per subject. Sentences and pictures describe 3 symbols: *, +, and $, using ‘above’, ‘below’, ‘not above’, ‘not below’. Images are acquired every 0.5 seconds. Read Sentence View PictureRead Sentence View PictureFixation Press Button 4 sec.8 sec.t=0 Rest [Keller et al, 2001]

10. Process 1: ReadSentence Response signature W: Duration d: 11 sec. Offsets  : {0,1} P(  ): {  0,  1 } One configuration c of process instances  1,  2, …  k : (with prior  c ) Predicted mean: Input stimulus  : 11  Timing landmarks : 2 1 22 Process instance:  2 Process h: 2 Timing landmark: 2 Offset O: 1 (Start time: 2 + O) sentence picture v1 v2 Process 2: ViewPicture Response signature W: Duration d: 11 sec. Offsets  : {0,1} P(  ): {  0,  1 } v1 v2 Processes of the HPM: v1 v2 + N(0,  1 ) + N(0,  2 )

11. HPM Formalism HPM = H =, a set of processes (e.g. ReadSentence) h =, a process W = response signature d = process duration  = allowable offsets  = multinomial parameters over values in  C =, a set of configurations c =, a set of process instances  =, a process instance (e.g. ReadSentence(S1)) h = process ID = timing landmark (e.g. stimulus presentation of S1) O = offset (takes values in  h )  =, priors over C  =, standard deviation for each voxel [Hutchinson et al, 2006]

12. Encoding Experiment Design Configuration 1: Input stimulus  : Timing landmarks : 2 1 ViewPicture = 2 ReadSentence = 1 Decide = 3 Configuration 2: Configuration 3: Configuration 4: Constraints Encoded: h(  1 ) = {1,2} h(  2 ) = {1,2} h(  1 ) != h(  2 ) o(  1 ) = 0 o(  2 ) = 0 h(  3 ) = 3 o(  3 ) = {1,2} Processes:

13. Inference Over configurations Choose the most likely configuration, where: C=configuration, Y=observed data,  =input stimuli, HPM=model

14. Learning Parameters to learn: –Response signature W for each process –Timing distribution  for each process –Standard deviation  for each voxel Expectation-Maximization (EM) algorithm to estimate W and . –E step: estimate a probability distribution over configurations. –M step: update estimates of W (using reweighted least squares) and  (using standard MLEs) based on the E step. –After convergence, use standard MLEs for 

15. Uncertain Timings Convolution matrix models several choices for each time point. 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 … … … 0 0 1 0 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 0 0 0 0 1 0 0 1...… … PD t=1 t=2 … t=18 … T’>T S Configurations for each row: 3,4 1,2 3,4 1,2 … 3 4 1 2 …

16. Uncertain Timings 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 … … … PD e1 e2 e3 e4 … S Y = W 3,4 1,2 3,4 1,2 … Configurations:Weights: e1 = P(C=3|Y,W old,  old,  old ) + P(C=4|Y,W old,  old,  old ) Weight each row with probabilities from E-step.

20. Potential Processes Can group these many ways to form different HPMs

21. Comparing HPMS Cross-validated data log-likelihood. All values are *10 6. Participant

22. Are we learning the right number of processes? For each training set, the table shows the average (over 30 runs) test set log- likelihood of each of 3 HPMs (with 2, 3, and 4 processes) on each of 3 synthetic data sets (generated with 2, 3, and 4 processes). Each cell is reported as mean ± standard deviation. NOTE: All values in this table are *10 5.

23. Ongoing Research Regularization for process response signatures (adding bias for temporal and/or spatial smoothness, spatial priors, spatial sparsity). Modeling process response signatures with basis functions. Allowing continuous start times (decoupling process starts from the data acquisition rate) A Dynamic Bayes Net formulation of HPMs.

24. References Dale, A.M., Optimal experiment design for event- related fMRI, 1999, Human Brain Mapping, 8, 109-114. Hutchinson, R.A., Mitchell, T.M., & Rustandi, I., Hidden Process Models, 2006, Proceedings of the 23 rd International Conference on Machine Learning, 433-440. Keller, T.A., Just, M.A., & Stenger, V.A., Reading span and the time-course of cortical activation in sentence-picture verification, 2001, Annual Convention of the Psychonomic Society.