1. Neural representation and decoding of the meanings of words

2. How is language represented in the brain?
Two different questions:
Which brain areas are involved in language?
How is language represented?
Meanings of words
Speech sounds
Syntax

3. Decoding as a test of whether we really understand anything about the encoding
How does the brain represent the meanings of words?
Test: see if we can neurally decode the words

4. Video from CBS 60 Minutes

5. Semantic similarity: A quick demo to show that your brain cares about it
You will be presented with a list of words. Try to remember as many as you can.
pie
cake
tart
heart
candy
chocolate
soda
nice
sugar
sour
bitter
good
tooth
taste
honey
Were any of the following words in that list?
soft short sweet smooth
Deese-Roediger-McDermott effect

6. People’s attempts so far to neurally decode word-meanings
Important contribution: Tom Mitchell et al., Science, 2008
fMRI data and semantic features publicly available at

7. Modeling a continuous space
Standard decoding:
Neural responses for Stim A
Neural responses for Stim B
Present some test-set neural data:
Q: Was it elicited by A or by B?
Problem:
What if it is a new stimulus, C?

8. Interpolating between stimuli, using a model of the stimulus space
Pattern-information analysis: from stimulus decoding to computational-model testing. Kriegeskorte N. Neuroimage May 15;56(2):

9. Stimuli
Snodgrass, J. G., & Vanderwart, M. (1980). A standardized set of 260 pictures: norms for name agreement, image agreement, familiarity, and visual complexity. Journal of experimental psychology: Human learning and memory, 6(2), 174.

10. The 60 words used as stimuli

11. The Mitchell study: word stimuli and semantic features
Stimuli: concrete nouns
E.g. hammer, shirt, dog, celery (60 words in all)
12 categories (tools, clothing, food, etc.), each with 5 words
Semantic features: action verbs
E.g. push, move, taste, see (25 semantic features in all)
Each noun has a 25-element semantic feature vector of its co-occurrence freqs with the verbs, from Google text- corpus
hammer = 0.13*break *touch *eat + …
celery = *break *touch *eat + …

12. Example co-occurrence features
Features for cat:
say said says (0.592),
see sees (0.449),
eat ate eats (0.435),
run ran runs (0.303),
hear hears heard (0.208),
open opens opened (0.175),
smell smells smelled (0.163),
clean cleaned cleans (0.146),
move moved moves (0.088),
listen listens listened (0.075),
touch touched touches (0.075) …

13. Mitchell et al. (2008) What do the semantic features look like in the brain?

14. Predicting brain activation

15. The Mitchell study: training and testing the model
Training (carried out separately in each subject)
Stage 1: represent each noun as weighted sum of semantic features
Stage 2: learn a linear mapping between those semantic features onto the word-elicited fMRI patterns in each person's brain.
Testing: can the model predict neural activation for untrained words?

16. The Mitchell study: decoding results
“Leave two out” testing strategy
Remove each pair of words in turn from 60-word set
Train the model on the remaining 58 words
Predict neural patterns for the two test-words
Match those predictions against the test-words’ actual elicited activations
Success rate for decoding left-out word-pairs: 77%
Chance-level is 50%
Tried making a hand-crafted feature-set: 80.9%