1. Hidden Process Models: Decoding Overlapping Cognitive States with Unknown Timing Rebecca A. Hutchinson Tom M. Mitchell Carnegie Mellon University NIPS Workshops: New Directions on Decoding Mental States from fMRI Data December 8, 2006

2. 2 Overview Open questions we address: –Treating fMRI as the time series that it is. –Allowing the testing of hypotheses. Open questions we do NOT address: –Interpretability of time series or spatial representation of activity. This talk –Motivation –HPMs (in 1 slide!) –Preliminary results

3. 3 Motivation Goal: connect fMRI to cognitive modeling. Cognitive Model: –Set of cognitive processes hypothesized to occur during a given fMRI experiment. Cognitive Process: –Spatial-temporal hemodynamic response function. –Timing distribution relative to experiment landmarks (like stimulus presentations and behavioral data).

4. 4 Study: Pictures and Sentences Task: Decide whether sentence describes picture correctly, indicate with button press. 13 normal subjects, 40 trials per subject. Sentences and pictures describe 3 symbols: *, +, and $, using ‘above’, ‘below’, ‘not above’, ‘not below’. Images are acquired every 0.5 seconds. Read Sentence View PictureRead Sentence View PictureFixation Press Button 4 sec.8 sec.t=0 Rest

5. 5 One Cognitive Model Read Sentence View PictureRead Sentence View PictureFixation Press Button 4 sec.8 sec.t=0 Rest ViewPicture –begins when picture stimulus is presented ReadSentence –begins when sentence stimulus is presented Decide –begins within 4 seconds of 2nd stimulus ViewPicture or ReadSentence Decide

6. 6 Hidden Markov Models (HMMs) Hidden Process Models (HPMs) States (1 time point)Processes (time window) Latent state sequenceLatent process instances No external inputUse experiment design and behavioral data State transition matrixProcess-specific timing distributions State-specific emission distributions Process-specific response signatures 1 hidden Markov chain governs observed data Process instances can overlap in space and time Forward-backward training algorithm EM training algorithm

7. 7 ViewPicture in Visual Cortex

8. 8 ReadSentence in Visual Cortex

9. 9 ViewPicture

10. 10 ReadSentence

11. 11 Decide 00.511.522.533.5 00000.0250.050.0750.85 Seconds following the second stimulus Multinomial probabilities on these time points

12. 12 Comparing Models HPMAvg. Test Set LL PS-1.0784 * 10^6 PSD-1.0759 * 10^6 PS+S-D-1.0742 * 10^6 PSD+D--1.0742 * 10^6 PSDB-1.0741 * 10^6 PSDyDn-1.0737 * 10^6 PSDyDnDc**-1.0717 * 10^6 PSDyDnDcB-1.0711 * 10^6 5-fold cross-validation, 1 subject P = ViewPicture S = ReadSentence S+ = ReadAffirmativeSentence S- = ReadNegatedSentence D = Decide D+ = DecideAfterAffirmative D- = DecideAfterNegated Dy = DecideYes Dn = DecideNo Dc = DecideConfusion B = Button ** - This HPM can also classify Dy vs. Dn with 92.0% accuracy. GNBC gets 53.9%. (using the window from the second stimulus to the end of the trial)

13. 13 Conclusions Simultaneous estimation of spatial- temporal signature (HRF) and temporal onset of cognitive processes. Framework for principled comparison of different cognitive models in terms of real data.