1. 1 Classifying Instantaneous Cognitive States from fMRI Data Tom Mitchell, Rebecca Hutchinson, Marcel Just, Stefan Niculescu, Francisco Pereira, Xuerui Wang Carnegie Mellon University November, 2003

2. 2 …

3. 3

4. 4 Cognitive state sequence COGNITIVE TASK “Virtual sensors” of cognitive state 1.Does fMRI contain enough information? 2.Can we devise learning algorithms to construct such “virtual sensors”?

5. 5 Learning Virtual Sensors Learn fMRI(t,t+k)  CognitiveState Classifiers: –Gaussian Naïve Bayes, SVM, kNN Trained per subject, per experiment Feature selection/abstraction –Select subset of voxels (by signal, by anatomy) –Select subinterval of time –Average activities over space, time –Normalize voxel activities

6. 6 Study 1: Pictures and Sentences Trial: read sentence, view picture, answer whether sentence describes picture Picture presented first in half of trials, sentence first in other half Three possible objects: star, dollar, plus Collected by Just et al.

7. 7 It is true that the star is above the plus?

8. 8

9. 9 + --- *

10. 10

11. 11 Is Subject Viewing Picture or Sentence? Learn fMRI(t,t+8)  {Picture, Sentence} Leave two out cross-validation was used to assess the performance of the classifiers SVMs and GNB worked better than kNN Some Details: –12 subjects, 40 pictures, 40 sentences –1397 - 2864 voxels per subject, 7 ROIs –fMRI snapshot taken every half second

12. 12 Error for Single-Subject Classifiers Error computed by averaging over all subjects 95% confidence intervals per subject are ~ 10% large Error of default classifier is 50% Dataset \ ClassifierGNBSVM1NN2NN5NN Picture vs Sentence0.160.090.200.180.19

13. 13 Approach: define supervoxels based on anatomically defined regions of interest –Normalize per voxel activity for each subject Each value scaled now in [0,1] –Abstract to seven brain region supervoxels –16 snapshots for each supervoxel Train on n-1 subjects, test on nth –Leave one subject out cross validation Can We Train Subject-Indep Classifiers?

14. 14 NO Feature Selection used in this experiment 95% confidence intervals approximately 5% large Error of default classifier is 50% Error for Cross Subject Classifiers Dataset \ ClassifierGNBSVM1NN2NN5NN Cross-Subject Pict vs Sent0.300.250.360.330.32

15. 15 Study 2: Word Categories Family members Occupations Tools Kitchen items Dwellings Building parts 4 legged animals Fish Trees Flowers Fruits Vegetables

16. 16 Word Categories Study Stimulus: –12 blocks of words: Category name (2 sec) Word (400 msec), Blank screen (1200 msec); answer … –Subject answers whether each word in category –20 words per block, nearly all in category

17. 17 Training Classifier for Word Categories Learn fMRI(t)  Word Category Training methods: kNN, GNB Leave one example out from each class used to assess performance Some Details: –10 subjects, 20 examples per class –8470 - 11,136 voxels per subject, 30 ROIs –fMRI snapshot taken every second

18. 18 Study 2: Results Classifier outputs ranked list of classes Evaluate by the fraction of classes ranked ahead of true class –0=perfect, 0.5=random, 1.0 unbelievably poor Dataset \ ClassifierGNB1NN3NN5NN Words 0.080.300.200.16

19. 19 Study 3: Syntactic Ambiguity Is subject reading ambiguous or unambiguous sentence? –“The experienced soldiers warned about the dangers conducted the midnight raid.” –“The experienced soldiers spoke about the dangers before the midnight raid.” Almost random results if no feature selection used With feature selection: –SVM - 77% accuracy –GNB - 75% accuracy –5NN – 72% accuracy

20. 20 Four feature selection methods: Active (n most active available voxels compared to baseline fixation activity, according to a t-test) RoiActive (n most active voxels in each ROI) RoiActiveAvg (average of the n most active voxels in each ROI) Disc (n most discriminating voxels according to a trained classifier) Active works best Feature Selection

21. 21 Dataset Feature Selecti on GNBSVM1NN3NN5NN Picture Sent No 0.290.320.430.410.37 Active 0.160.090.200.180.19 WordsNo 0.10N/A0.40 Active 0.08N/A0.300.200.16 Synt Amb No 0.430.380.500.460.47 Active 0.250.230.29 0.28 Feature Selection

22. 22 Summary Proved that there is enough information in the fMRI signal to allow decoding of Cognitive States Successful training of classifiers for instantaneous cognitive state in three studies Cross subject classifiers trained by abstracting to anatomically defined ROIs Feature selection and abstraction are essential

23. 23 Research Opportunities Learning temporal models –HMM’s, Temporal Bayes Nets Learn to discriminate whether a subject has certain mental disease Discovering useful data abstractions –ICA, PCA, hidden layers in Neural Nets Merging data from multiple sources –fMRI, ERP, reaction times