Neural representation and decoding of the meanings of words

How is language represented in the brain? Two different questions: Which brain areas are involved in language? How is language represented? Meanings of words Speech sounds Syntax

Decoding as a test of whether we really understand anything about the encoding How does the brain represent the meanings of words? Test: see if we can neurally decode the words

Video from CBS 60 Minutes https://www.youtube.com/watch?v=8jc8URRxPIg

Semantic similarity: A quick demo to show that your brain cares about it You will be presented with a list of words. Try to remember as many as you can. pie cake tart heart candy chocolate soda nice sugar sour bitter good tooth taste honey Were any of the following words in that list? soft short sweet smooth Deese-Roediger-McDermott effect

People’s attempts so far to neurally decode word-meanings Important contribution: Tom Mitchell et al., Science, 2008 fMRI data and semantic features publicly available at http://www.cs.cmu.edu/~tom/science2008

Modeling a continuous space Standard decoding: Neural responses for Stim A Neural responses for Stim B Present some test-set neural data: Q: Was it elicited by A or by B? Problem: What if it is a new stimulus, C?

Interpolating between stimuli, using a model of the stimulus space Pattern-information analysis: from stimulus decoding to computational-model testing. Kriegeskorte N. Neuroimage. 2011 May 15;56(2):411-21.

Stimuli Snodgrass, J. G., & Vanderwart, M. (1980). A standardized set of 260 pictures: norms for name agreement, image agreement, familiarity, and visual complexity. Journal of experimental psychology: Human learning and memory, 6(2), 174.

The 60 words used as stimuli

The Mitchell study: word stimuli and semantic features Stimuli: concrete nouns E.g. hammer, shirt, dog, celery (60 words in all) 12 categories (tools, clothing, food, etc.), each with 5 words Semantic features: action verbs E.g. push, move, taste, see (25 semantic features in all) Each noun has a 25-element semantic feature vector of its co-occurrence freqs with the verbs, from Google text- corpus hammer = 0.13*break + 0.93*touch + 0.01*eat + … celery = 0.00*break + 0.03*touch + 0.84*eat + …

Example co-occurrence features Features for cat: say said says (0.592), see sees (0.449), eat ate eats (0.435), run ran runs (0.303), hear hears heard (0.208), open opens opened (0.175), smell smells smelled (0.163), clean cleaned cleans (0.146), move moved moves (0.088), listen listens listened (0.075), touch touched touches (0.075) … http://www.cs.cmu.edu/~tom/science2008/semanticFeatureVectors.html

Mitchell et al. (2008) What do the semantic features look like in the brain?

Predicting brain activation

The Mitchell study: training and testing the model Training (carried out separately in each subject) Stage 1: represent each noun as weighted sum of semantic features Stage 2: learn a linear mapping between those semantic features onto the word-elicited fMRI patterns in each person's brain. Testing: can the model predict neural activation for untrained words?

The Mitchell study: decoding results “Leave two out” testing strategy Remove each pair of words in turn from 60-word set Train the model on the remaining 58 words Predict neural patterns for the two test-words Match those predictions against the test-words’ actual elicited activations Success rate for decoding left-out word-pairs: 77% Chance-level is 50% Tried making a hand-crafted feature-set: 80.9%