1. Learning to Identify Overlapping and Hidden Cognitive Processes from fMRI Data Rebecca Hutchinson, Tom Mitchell, Indra Rustandi Carnegie Mellon University

2. Decide whether consistent How can we track hidden cognitive processes? Read sentence View picture Cognitive processes: ? Observed fMRI: cortical region 1: cortical region 2: Observed button press:

3. Typical BOLD response Signal Amplitude Time (seconds) At left is a typical averaged BOLD response Here, subject reads a word, decides whether it is a noun or verb, and pushes a button in less than 1 second.

4. Related Work General linear model (GLM) applied to fMRI –E.g., [Dale 1999]; SPM; –Accommodates multiple, overlapping processes, –But not unknown process timing Dynamic Bayesian Networks –Family of probabilistic models for time series –E.g., Factorial HMMs [Ghahramani & Jordan 1998] –Accommodate hidden timings/states –But do not capture convolution of overlapping states –Require learning detailed next-state function

5. General Linear Model Common fMRI data analysis approach Define ‘design matrix’ X which describes timing of input stimuli y = X h + ε Observed fMRI time series Design matrix (stimulus timing) Gaussian noise Responses to individual stimuli HPM’s correspond to assuming X describes both stimuli and hidden mental processes, and is partially unknown

6. Approach: Hidden Process Models Probabilistic model –Can evaluate P(model | data), P(data | model) Describe hidden processes by their –Type, duration, start time, fMRI signature Algorithms for learning model, interpreting data –Learn maximum likelihood models and data interpretations

7. Hidden Process Models Process ID = 3 Process ID = 2 Process Instances: Observed fMRI: Decide whether consistent View picture Processes: ID: 1 Timing: P(start= +O) Response: ID: 2 Timing: P(start= +O) Response: ID: 3 Timing: P(start= +O) Response: Process ID = 1  Time landmarks: ¢ 1 ¢ 2 ¢ 1 ¢ 3

8. Process: ViewPicture Duration d: 11 sec. P(Offset times): ,  Response signature W: Configuration C of Process Instances h  1,  2, … i Observed data Y: Input Stimulus  : 11 44  Timing landmarks : ¢ 2 ¢ 1 ¢ 3 22 Process instance:  2 Process h: ViewPicture Timing landmark : 2 Offset time O: 1 sec Start time ´ + O sentence picture sentence 33 Hidden Process Models

9. HPMs More Formally… Process h = h d, ,  W i Process Instance  = h h,, O i Configuration C = set of Process Instances Hidden Process Model HPM = h H, , C,  i H: set of processes  : prior probs over H C: set of candidate configurations  : h  1 …  v i voxel noise model

10. HPM Generative Model Probabilistically generate data using a configuration of N process instances with known landmarks: 1.Generate a configuration C of process instances: For i=1 to N, generate process instance  i Choose a process h i according to P(h| i,  ) Choose an offset O i according to P(O|  (h) ) 2.Generate all observed fMRI data y tv given C:

11. HPM Inference Given: –An HPM, including a set of candidate configurations we typically assume processes known, but not timing –Observed data Y Determine: –The most probable process instance configuration c –P(C=c|Y, HPM)  P(Y|C=c, HPM) P(C=c | HPM)

12. Inference: Example Configuration 1: Observed data ProcessID=1, S=1 ProcessID=2, S=17 ProcessID=3, S=21 Configuration 2: ProcessID=2, S=1 ProcessID=1, S=17 ProcessID=3, S=23 Prediction 1 Prediction 2

13. Learning HPMs with unknown timing O(  ), known processes h(  ) EM (Expectation-Maximization) algorithm E-step –Estimate the conditional distribution over start times of the process instances given observed data, P(O(  1 )…O(  N ) | Y, h(  1 )… h(  N ), HPM). M-step –Use the distribution from the E step to get maximum-likelihood estimates of the HPM parameters. * In real problems, some timings are often known

14. HPMs are learnable from realistic amounts of data

15. Figure 1. The learner was given 80 training examples with known start times for only the first two processes. It chooses the correct start time (26) for the third process, in addition to learning the HDRs for all three processes. true signal Observed noisy signal true response W learned W Process 1Process 2Process 3

16. fMRI Study: Pictures and Sentences Each trial: determine whether sentence correctly describes picture 40 trials per subject. Picture first in 20 trials, Sentence first in other 20 Images acquired every 0.5 seconds. Read Sentence View PictureRead Sentence View PictureFixation Press Button 4 sec.8 sec.t=0 Rest

17. Decide whether consistent HPM model for Picture-Sentence Comparison Read sentence View picture Cognitive processes: ? Observed fMRI: cortical region 1: cortical region 2: Observed button press:

18. Learned HPM with 3 processes (S,P,D), and R=13sec (TR=500msec). P P SS D? observed Learned models: S P D D start time chosen by program as t+18 reconstructed P P SS D D D?

19. HPMs provide more accurate classification of unknown processes than earlier methods (e.g., Gaussian Naïve Bayes (GNB) classifier)

20. Standard classifier formulation View Picture Or Read Sentence Or View Picture Fixation Press Button 4 sec.8 sec.t=0 Rest picture or sentence? 16 sec. GNB: Standard formulation of classification problem (e.g., Gaussian Naïve Bayes (GNB)): Train on labeled data: known Processes, known StartTimes Test on unlabeled data: unknown Processes, known StartTimes

21. HPM classifier accounts for overlap View Picture Or Read Sentence Or View Picture Fixation Press Button 4 sec.8 sec.t=0 Rest picture or sentence? 16 sec. GNB: picture or sentence? HPM:

22. View Picture Or Read Sentence Or View Picture Fixation Press Button 4 sec.8 sec.t=0 Rest picture or sentence? 16 sec. GNB: picture or sentence? HPM: Results HPM with overlapping processes improves accuracy by 15% on average.

23. HPMs allow detecting and examining hidden processes with unknown timing

24. Decide whether consistent Two cognitive processes, or three? Read sentence View picture Cognitive processes: ? Observed fMRI: cortical region 1: cortical region 2: Observed button press:

25. Choosing Between Alternative HPM Models Train 2-process HPM 2 on training data Train 3-process HPM 3 on training data Test HPM 2 and HPM 3 on separate test data –Which predicts process identities better? –Which has higher probability given the test data? –(use n-fold cross-validation for test)

26. 2-process HPM, 3-process HPM, GNB

27. Summary Hidden Process Model formalism Superiority over earlier classification methods Basis for studying hidden cognitive processes

28. Future Directions Add temporal and/or spatial smoothness constraints to process fMRI signatures Allow variable duration processes Give processes input arguments, output results Feature selection for HPMs Process libraries, hierarchies