Lexical Access: Generation & Selection

Lexical Access: Generation & Selection

Main Topic Listeners as active participants in comprehension process
Model system: word recognition

The mental lexicon sing door carry turf turtle gold turk turkey turn
sport figure sing door carry turf turtle gold turk turkey turn water turbo turquoise turnip turmoil

How do we recognize words?
The Simplest Theory Take a string of letters/phonemes/syllables, match to word in the mental lexicon (That’s roughly how [cheap] word processors work) …is it plausible?

Robust in noisy environments
I want to start with a couple of basic facts that give us a clearer picture of the cognitive ability that we’re trying to explain. First, language understanding is quite robust, even in noisy environments, like when you’re having a conversation on a metro train. (I’ll bet you that Siri on your iPhone won’t work so well in that setting.) Robust in noisy environments

Word Recognition is Fast
Intuitively immediate - words are recognized before end of word is reached Eye-tracking studies indicate effects of access within ms Speech shadowing at very brief time-lags, ~250ms (Marslen-Wilson 1973, 1975)

Fast ‘the dog was big and scary’ 3-5 words/second 200-400 msec/word
Third, language comprehension is fairly fast. These are waveforms and a spectrogram that show the acoustic information that we have to decipher in order to understand a simple sentence like “the dog was big and scary”. And the words come at us at a rate of around 3-5 words per second. That’s about milliseconds per word. Reading speeds are pretty similar to that. I don’t know whether that sounds fast or slow to you. But consider what you have to do with each word as it comes in. [LIST] Ok – big deal – so how long do those steps take? Well we also know something about that. At each word … 1. visual/acoustic processing 2. phoneme recognition 3. word recognition 4. syntactic analysis 5. semantic interpretation

Electrical/magnetic brain activity Word access: ~250-400 msec
Halgren et al. 2002 Electrical/magnetic brain activity Word access: ~ msec One thing that we know is that successful word recognition generally takes on the order of milliseconds from when you see or hear a word. We know this from lots of different sources of evidence, including work in cognitive neuroscience. You can’t just look at brain recordings and say “oh look – that’s word recognition right there”. But there’s a clever strategy that allows you to come pretty close to that. What you do is record people’s brains when they’re reading or hearing two types of words, maybe *common* and *uncommon* words, or *expected* and *unexpected* words in a sentence context. And then you look for when the pattern of brain activity for the two word types starts to differ. The initial activity is just the same for the two types of words – because the sensory systems don’t care about the lexical properties of the words – but when the activity patterns start to diverge, that’s pretty good evidence that word recognition has happened. (I should point out that this Is *not* fMRI, but rather the electrophysiological techniques that we use here that allow us to track the precise time-course of brain activity with millisecond precision.) Now the time estimate that we get from this basically uses up all of the time that we have before the next word comes along, often even more. And so that doesn’t leave us with any time for fitting the word into the ongoing sentence or for updating our discourse interpretation. So normal language understanding, which *seems* to be effortless and immediate, actually involves a fairly severe computational bottleneck. How can we deal with that timing bottleneck? Well, in order to think about this, I find it useful to think of an analogy from a very different domain. Computational Bottleneck Updating interpretation at each word requires much more time than is available. Pylkkänen et al. 2002

Marslen-Wilson 1975 Speech shadowing involves on-line repetition of a speaker… Speech shadowing involves on-line repetition of a speaker… ms Shadowing latency The new peace terms have been announced… They call for the unconditional surrender of … universe of … already of … normal semantic syntactic

Marslen-Wilson 1975

“If the interaction between higher and lower levels of of analysis takes place only after the initial phonetic and lexical identification of the word, then restoration of disrupted words should be equally frequent in all Context conditions. The shadower would have no basis, in his initial repetition, for rejecting contextually anomalous restorations. However, if immediate identification does interact on-line with the semantic and syntactic context, then it becomes possible for context variables to determine word restoration frequency.” (Marslen-Wilson, 1975, p. 226)

“The high incidence of WR errors in Normal2 illustrates the speed and the precision with which structural information can be utilized. If the first syllable indicates a word that matches the context, then the close shadower can immediately start to restore that word in his repetition. This implies, first, that the constraints derived from the preceding items of the string are available to guide the analysis of even the first syllable of the target word. Second, these constraints can specify the permissible form-class and meaning of the target word with sufficient precision to enable the shadower to assess the appropriateness of just its first syllable.” (Marslen-Wilson 1975, p. 227).

Lexical Access is Robust
Succeeds in connected speech Succeeds in fast speech Survives masking effects of morphological affixation and phonological processes Deleted or substituted segments Speech embedded in noise

But… Speed and robustness depends on words in context sentence --> word context effects In isolation, word recognition is slower and a good deal more fragile, susceptible to error …but still does not require perfect matching

Questions How does lexical access proceed out of context?
Why is lexical access fast and robust in context? When does context affect lexical access? does it affect early generation (lookup) processes? does it affect later selection processes? What is the neurophysiological implementation of this interaction?

Classic Experimental Paradigms

Reaction Time Paradigms
Lexical Decision Priming

Looking for Words List 1 sickle cathartic torrid gregarious oxymoron atrophy List 2 parabola periodontist preternatural pariah persimmon porous

Looking for Words List 1 sickle cathartic torrid gregarious oxymoron atrophy List 2 parabola periodontist preternatural pariah persimmon porous Speed of look-up reflects organization of dictionary

Looking for Words +

Looking for Words DASH

Looking for Words +

Looking for Words RASK

Looking for Words +

Looking for Words CURLY

Looking for Words +

Looking for Words PURCE

Looking for Words +

Looking for Words WINDOW

Looking for Words +

Looking for Words DULIP

Looking for Words +

Looking for Words LURID

(Embick et al., 2001)

Looking for Words Semantically Related Word Pairs doctor nurse hand finger speak talk sound volume book volume

Looking for Words In a lexical decision task, responses are faster when a word is preceded by a semantically related word DOCTOR primes NURSE Implies semantic organization of dictionary

Active Recognition System actively seeks matches to input - does not wait for complete match This allows for speed and robustness, but …

Cost of Active Search… Many inappropriate words activated
Inappropriate choices must be rejected Two Stages of Lexical Access activation vs. competition recognition vs. selection proposal vs. disposal

The mental lexicon sing door carry turf turtle gold turk turkey turn
sport figure sing door carry turf turtle gold turk turkey turn water turbo turquoise turnip turmoil

The mental lexicon TURN sing door carry turf turtle gold turk turkey
sport figure sing door carry turf turtle gold turk turkey turn water turbo turquoise turnip turmoil TURN

Automatic activation TURN turnip turmoil sing door carry
sport figure sing door carry turf turtle gold turk turkey water turn turbo turquoise turnip turmoil TURN

Lateral inhibition TURN turnip turmoil sing door carry
sport figure sing door carry turf turtle gold turk turkey water turn turbo turquoise turnip turmoil TURN

What is lexical access? Activation Competition Selection/Recognition
TURN TURNIP level of activation TURF TURTLE resting level time Stimulus: TURN (e.g. Luce et al. 1990, Norris 1994)

Cohort S song story sparrow saunter slow secret sentry etc.

Cohort SP spice spoke spare spin splendid spelling spread etc.

Cohort SPI spit spigot spill spiffy spinaker spirit spin etc.

Cohort SPIN spin spinach spinster spinaker spindle

Cohort SPINA spinach

Cohort SPINA spinach word uniqueness point

Cohort SPINA spinach spinet spineret

Cross-Modal Priming

Evidence for Cohort Activation
KAPITEIN KAPITAAL (Marslen-Wilson, Zwitserlood)

KAPITEIN KAPITAAL KAPIT… (Marslen-Wilson, Zwitserlood)

KAPITEIN KAPITAAL BOOT KAPIT… GELD (Marslen-Wilson, Zwitserlood)

KAPITEIN KAPITAAL BOOT BOOT KAPIT… KAPITEIN GELD GELD (Marslen-Wilson, Zwitserlood)

CAPTAIN CAPTIVE SHIP SHIP CAPT… CAPTAIN GUARD GUARD (Marslen-Wilson, Zwitserlood)

Cohort Model Partial words display priming properties of multiple completions: motivates multiple, continuous access Marslen-Wilson’s claims Activation of candidates is autonomous, based on cohort only Selection is non-autonomous, can use contextual info. How, then, to capture facilitatory effect of context?

Gating Measures Presentation of successive parts of words
SPI SPIN SPINA… Average recognition times Out of context: ms In context: 200ms (Grosjean 1980, etc.)

Word Monitoring Listening to sentences - monitoring for specific words
Mean RT ~240ms Identification estimate ~200ms Listening to same words in isolation Identification estimate ~300ms (Brown, Marslen-Wilson, & Tyler)

Cross-Modal Priming The guests drank vodka, sherry and port at the reception (Swinney 1979, Seidenberg et al. 1979)

Cross-Modal Priming The guests drank vodka, sherry and port at the reception WINE SHIP (Swinney 1979, Seidenberg et al. 1979)

Cross-Modal Priming The guests drank vodka, sherry and port at the reception WINE I’m worried about whether this really shows (i) multiple access, and (ii) is a simple reflection of access. SHIP (Swinney 1979, Seidenberg et al. 1979)

Generation and Selection
Investigating the dependence on ‘bottom-up’ information in language understanding ‘Active’ comprehension has benefits and costs Speed Errors Overgeneration entails selection Sources of information for generating candidates Bottom-up information (e.g., lexical cohorts) ‘Top-down’ information (e.g., sentential context) Questions about whether context aids generation or selection

Cross-modal Priming Early: multiple access Late: single access
…i.e., delayed effect of context

CMLP - Qualifications Multiple access observed
when both meanings have roughly even frequency when context favors the lower frequency meaning Selective access observed when strongly dominant meaning is favored by context (see Simspon 1994 for review)

Context vs. frequency The guests drank wine, sherry, and port at the reception. The violent hurricane did not damage the ships which were in the port, one of the best equipped along the coast.

Frequency in Reading Rayner & Frazier (1989): Eye-tracking in reading
measuring fixation durations in fluent reading ambiguous words read more slowly than unambiguous, when frequencies are balanced, and context is unbiased unbalanced words: reading profile like unambiguous words when prior context biases one meaning dominant-biased: no slowdown due to ambiguity subordinate-biased: slowdown due to ambiguity contextual bias can offset the effect of frequency bias how can context boost the accessibility of a subordinate meaning?

Why multiple/selective access?
How could context prevent a non-supported meaning from being accessed at all? (Note: this is different from the question of how the unsupported meaning is suppressed once activated) Possible answer: selective access can only occur in situations where context is so strong that it pre-activates the target word/meaning

Summary so far … Active generation of lexical hypotheses (generation vs. selection) Lexical processing is more robust in context But RT evidence suggests that initial activation is driven by bottom-up information on word forms Marslen-Wilson: activation of cohort members is autonomous, selection among candidates makes use of context What is the source of autonomy in lexical activation? (i) architectural (ii) practical Processing of bottom-up input may be different when specific lemmas are available ahead of the input, due to (i) monitoring, (ii) strongly constraining context Context clearly aids robustness of word recognition; not so clear how/whether it impacts speed of word recognition

Speed of Integration If context can only be used to choose among candidates generated by cohort… context can choose among candidates prior to uniqueness point but selection must be really quick, in order to confer an advantage over bottom-up information [… or recognition following uniqueness point must be slow in the absence of context.]

Tanenhaus & Lucas 1987

Cross-Modal Lexical Access
Seidenberg, Tanenhaus, Leiman, & Bienkowski (1982) Cross-modal naming They all rose vs. They bought a rose Probes: FLOWER, STOOD Immediate presentation: equal priming; 200ms delay: selective priming Prather & Swinney (1977): similar w/ cross-modal lexical decision Tanenhaus & Donnenworth-Nolan (1984): similar, w/ extra delay in presenting target word

Experiment 1

Experiment 2

cost no cost Tanenhaus, Swinney et al. showed no effect of syntactic category in cross-modal tasks – and argued that this made sense. But these folks did find effects of syntactic category. So what gives? Could the activation that feeds cross-modal naming or associate priming fail to lead to ambiguity/uncertainty effects on eye-movements in reading? They suggest that both meanings become active, but that the effect of the syntactic constraint is to eliminate effects of competition. Good opportunity to discuss link between findings and conclusions.

Eye-tracking

Frequency in Object Recognition
X “Pick up the be..” (Dahan, Magnuson, & Tanenhaus, 2001)

lobster bench X bell bed “Pick up the be..” (Dahan, Magnuson, & Tanenhaus, 2001)

Timing estimates Saccadic eye-movements take ms to program Word recognition times estimated as eye-movement times minus ~200ms

(Dahan, Magnuson, & Tanenhaus, 2001)

Cohort Model Partial words display priming properties of multiple completions: motivates multiple, continuous access Marslen-Wilson’s claims Activation of candidates is autonomous, based on cohort only Selection is non-autonomous, can use contextual info. How, then, to capture facilitatory effect of context…

Cohort SPINA spinach

Cohort SPIN spin spinach spinster spinaker spindle

CAPTAIN CAPTIVE SHIP SHIP CAPT… CAPTAIN GUARD GUARD (Marslen-Wilson, Zwitserlood)

Matches to other parts of words
Word-ending matches don’t prime honing [honey] bij [bee] woning [apartment] foning [--]

Disagreements Continuous activation, not limited to cohort, as in TRACE model (McClelland & Elman, 1986) Predicts activation of non-cohort members, e.g. shigarette, bleasant

BIG BAT DOG Words B I G A T R Phonemes Feedback vs. Decision Bias

Non-Cohort Competitors
“Pick up the…” beaker beetle (onset) speaker (non-onset) carriage (distractor) (Allopenna, Magnuson, & Tanenhaus, 1998)

“Look at the cross. Now look at the {flower/flour}.”
Shape competitor: pillow/lollipop

Biased: The baker had agreed to make several pies for the large event today, so he started by taking out necessary ingredients, like milk, eggs, and flour. Neutral: As Jenny walked into the house after school, she looked at the table and was surprised to see the {flower/flour}.

I wanted to point out a minor difference in your interpretation of Allopenna, Magnuson, & Tanenhaus (1998) and mine. Allopenna et al. is cited on p. 75 as one of the "estimates in the literature [that] the earliest processes involved in lexical access often fall in the 200 ms range". But eye tracking data of the sort we presented actually gives a strikingly different estimate. What we find again and again in studies using the visual world paradigm is that there is an approximately ms lag between events in the speech signal and changes in fixation proportions. However, this should not suggest that it takes 200 ms for processes of lexical access to kick in. Rather, given that it takes at least about 150 ms to plan and launch an eye movement to a point of light in a darkened room, this means we can roughly subtract 150 msecs of the lag and attribute it to saccade planning. This leaves us with only about 50 msecs to attribute to the very earliest processes of access that are indexed by the eye movements. (This seems too short by 1-2 dozen msecs, but note that only a very small proportion of trials include such early eye movements, and statistically reliable differences between related and unrelated items emerge another ~25-50 msecs later.) [ message, 6/26/07] Jim Magnuson, UConn

Discrepancy 1: fMRI & MEG localizations of N400s contrast
MEG: N400 in pMTG – lexical access fMRI: lexical effect varies, sentence effect typically frontal Resolution 1: there’s consistency in lexical effects, taking into account SOA, MTG-lexical Resolution 2: sentences, fMRI integrates across time, may highlight different features This leads to a picture where MTG N400 is lexical area with activity modulated top-down Debate: does N400 reflect lexical or higher-level processes? Lexical evidence: Lau et al – anatomical argument Lexical evidence: Federmeier/Kutas 1999 – priming of inappropriate words Lexical evidence: Lau et al – comparison of N400 to anomalous & unprimed Higher level evidence: van Berkum, discourse effects Higher level evidence: Hagoort et al … Higher level evidence: Hagoort et al. 2009, coercion Eye-tracking: more evidence of context insensitivity in lexical access Discrepancy 2: characterization of interactions different in behavioral and cog-neuroscience literatures – what gives? Option 1: no conflict. less activation of unambig ≠ activation of irrelevant Option 2: no top-down effect, just less bottom-up signal needed Option 3: contexts are different, i.e., N400 contexts stronger than CMLD contexts

Cognitive Neuroscience
There is a current debate in the cognitive neuroscience of language that has much in common with this … though the two areas are rarely compared How the questions are framed: Behavioral studies: Does biasing context prevent consideration of contextually inappropriate word meanings? Cognitive neuroscience: Does the N400 reflect lexical or higher-level interpretive processes?

Electrophysiology of Sentence Comprehension
Semantic anomaly I drink my coffee with cream and sugar I drink my coffee with cream and socks N400 Kutas & Hillyard (1980)

Left Anterior Negativity (LAN)
Electrophysiology of Sentence Comprehension Left Anterior Negativity (LAN) P600 he mows he *mow

N400 & Cloze Probability Lots of work has shown that the N400 isn’t just a response to semantic anomaly. It’s a more general response to is seen to all words, that varies as a function of cloze probability in sentence contexts, and importantly is also seen in a wide variety of lexical tasks. The N400 is modulated by priming, by word repetition, by lexical frequency, and by a host of other factors. e.g., Kutas & Hillyard, 1984, Nature Also modulated by (i) priming, (ii) word repetition, (iii) lexical frequency, etc.

and priming Kutas & Federmeier, 2000, TICS

Amy bought the napkins that the café manager diligently folded in the booth.
Amy bought the napkins that the café manager diligently baked in the booth. Now there’s no disputing that it’s pretty cool what you can record with these different tools, and I must admit that it still has a bit of a science fiction feel to me. This figure is from an MEG study that was led by Henny Yeung when he was in our lab a couple of years ago, and it shows the detailed time course of activity from a source in left temporal cortex while people are reading semantically normal and semantically anomalous sentences. What we’re tracking here is the MEG counterpart of the classic N400 response component at each word, and you can see that the response comes in clearly after each open class word, and much less clearly after function words. … and this is just from putting a magnetometer next to the head and recording the fields that come out of the head. Pretty neat. But we have to make sure not to get carried away by all this. It’s not hard to find overblown claims in the media about where all this technology is going. MEG counterpart of N400 at successive word positions in sentence comprehension. (Yeung et al., 2004)

Funct. Magn. Res. Imaging (fMRI) Magnetoencephalography (MEG)
Electroencephalography (EEG/ERP) The main measures that I’ll be talking about are … functional MRI – this is a great tool for localizing brain activity, … then there’s electroencephalography, which has been around for a long while and hence lacks sex appeal, but I think it still gives excellent information about the timing of brain activity … and I’ll also talk about magnetoencephalography or MEG, which LOOKS more like fMRI (you put your head in the toilet seat here), but it’s actually very closely related to EEG, and it holds the promise of offering both timing and localization information … though more on that to come.

Funct. Magn. Res. Imaging (fMRI) Magnetoencephalography (MEG)
Strong external magnetic field: 3 Tesla Small brain-generated fields: ~10-14 Tesla So the basic idea is this. What happens if you look at the kinds of designs that give N400s in ERP studies, and compare the localization evidence from MEG and fMRI studies? Although the machinery looks kinda similar, the techniques are quite different. Changes in blood oxygenation (BOLD) Extracellular currents from neurons External field targets specific locations Localizations inferred from scalp distrib. Time course: a few seconds Time course: milliseconds

SEM > SYN SYN > SEM fMRI – inferior frontal
Rüschemeyer et al. 2006, Neuroimage The first pass generalization is not too encouraging. In MEG studies of semantic anomalies, almost all studies find sources in left posterior temporal cortex. In fMRI studies of the same things you find activations all over the place, but the one consistent location is left inferior frontal cortex. That’s not so great. MEG – posterior temporal Most consistent localizations in sentences with semantic anomalies Helenius et al. 1999, J. Cogn. Neurosci.

(Halgren et al. 2002)

fMRI MEG Friederici et al., 2003, Cer. Cor.
Kuperberg et al. 2003, JoCN Friederici et al., 2003, Cer. Cor. fMRI Halgren et al., 2003, NeuroImage MEG The starting point of this was a project that we had a few years ago with Kuni Sakai, who’s going to be talking here tomorrow. We were already working with MEG, and we wondered whether we might be able to make some headway by combining data from MEG and fMRI. As you know, fMRI is GREAT for localization of brain activity, but it’s not so hot for timing information. On the other hand, MEG is great for isolating activity in time, and we can make serious attempts to localize momentary MEG activity. But the kinds of localizations that you get from MEG are really not in the same league as the localizations you get from fMRI. If you just care about localization, then fMRI is your friend. But we wanted to know whether we could use localization information from fMRI to “seed” localizations of our MEG data, possibly giving us the best of both worlds.

The people flocked to Washington to attend a… statistics workshop
We looked at a few different sentence conditions. But the one that I’ll tell you about here involves a standard semantic anomaly paradigm. So imagine that you read a sentence onset like this [….] then you might expect the continuation to be “rally” or “Redskins game”, so you might be surprised to encounter [this] completion. [This wasn’t actually one of our sentences.] It’s well known that unexpected sentence final words like this elicit a robust N400 effect in ERP studies. But what about the localization?

Semantic Anomaly MEG/EROS  MTG Helenius et al., 1999, JoCN
Halgren et al., 2002, NeuroImage Well, in studies using time-sensitive measures, the findings are pretty consistent. If you look across a range of different MEG studies – whether it’s this one by Liina and Brian, or this one by Riita Salmelin’s group in Helsinki, or this one by Eric Halgren – what you find is that the response to semantic anomaly localizes to somewhere in the posterior temporal lobe. The localization accuracy isn’t perfect, but we’ll call it the medial temporal gyrus, or MTG, for the sake of convenience. You find the same thing if you look at results from event-related optical signaling –or EROS – this is one of those freaky techniques where you somehow localize brain activity by shining infra-red lights into the head. Here’s a study from the Illinois group from a couple of years ago, and their localization of the N400 lines up perfectly with the MEG studies. Pylkkänen & McElree, 2007, JoCN Tse et al., 2007, PNAS

Semantic Anomaly fMRI  IFG Baumgaertner et al., 2002, NeuroImage
But things are rather different if we look at fMRI studies of the same phenomena. Here it seems that sentences with semantic anomalies lead to increased activation in areas of left inferior frontal cortex. You can see that in this study by Peter Hagoort’s group, in this study by Gina Kuperberg, and in this study by Baumgaertner and colleagues – there are lots of different patches of color here, but it’s the blue that we care about. Not sure if you can see it at the back, but I added a white circle to show where it should be. Hagoort et al., 2004, Science Kuperberg et al., 2008, NeuroImage

Sentence-level fMRI Studies
And I’m not just cherry picking here, as you can see from this table. When you look across a larger group of fMRI studies, you find that almost all of them – 15 of 17 – show activation in one or more subregions of left inferior frontal cortex. Lau, Phillips, & Poeppel, 2008, Nat. Revs. Neurosci.

Sentence-level fMRI Studies
But only about a third of them show temporal lobe activation. So that’s kind of odd. The area that is most reliably localized in the MEG studies shows up pretty rarely in fMRI studies. So what’s going on! Lau, Phillips, & Poeppel, 2008, Nat. Revs. Neurosci.

Direct Comparison fMRI: MEG: inferior frontal posterior temporal
More closely matched design Same individuals Same materials (more trials w/ MEG) Same presentation parameters Conflicting localizations persist fMRI: inferior frontal MEG: posterior temporal Well, one possibility is that the MEG and fMRI studies weren’t as closely matched as we’d like, so that the results don’t line up properly. So we tested this. You can’t do simultaneous MEG and fMRI – the magnet in the case of fMRI is about 14 orders of magnitude stronger than the brain magnetic fields that you measure in the case of MEG. But we can do the next best thing. XXX We took the same people, and the same pool of materials, and the exact same presentation parameters, and we had people do both studies – one in our MEG lab, and the fMRI down the road at NIH. … XXX And we found the same contrast again. The effect of semantic anomaly localized to posterior temporal cortex in MEG, and to inferior frontal cortex in fMRI. So it seems that it’s something about the techniques. … But it’s not simply that fMRI struggles to detect activity in posterior temporal cortex. Lau, Yeung, Hashimoto, Phillips, & Braun, ‘06

fMRI vs. MEG comparison Identical pool of materials Same participants
Congruous: … the napkins that the café manager diligently folded … Incongruous: … the napkins that the café manager diligently baked … Incorrect: … the napkins that the café manager diligently fold … items per participant per condition All incongruous words appear as congruous words in another item set Same participants Identical presentation parameters, task, etc. So we thought that maybe the problem was that the previous studies just weren’t closely matched. So we fixed that. We ran a study using congruous, incongruous sentences using exactly the same materials, same people, same task and presentation parameters, using MEG and fMRI. It was a pain to run, but we hoped that it would resolve the discrepancies. (We also had syntactically incorrect sentences, but I’m not concerned with those here.) Lau et al. 2008

Congruous: Amy bought the napkins that the café manager diligently folded in the booth.
Incongruous: Amy bought the napkins that the café manager diligently baked in the booth. R L R L fMRI And the results. No cigar. In the fMRI study we found nice inferior frontal activations for semantic anomalies. And in the MEG study our most consistent sources were in posterior temporal cortex. Not so encouraging for aligning measures. MEG

fMRI studies of semantic priming
What’s more, if you just look at the fMRI literature, you find huge variability. Fortunately, there are a lot of studies to compare. This figure shows localizations from studies of semantic priming. The activations are all over the map. My inclination as an electrophysiology fan would be to just give up. So much for fMRI. fMRI studies of semantic priming Lau, Phillips, & Poeppel, 2008, Nature Rev. Neurosci.

Lexical Priming fMRI  MTG Gold et al., 2006, J. Neurosci.
… because if we look at results from a different class of “N400-like” studies, we find that fMRI and MEG line up much better. It’s been clear for a long time that lexical priming also gives rise to an N400 effect, and the electrophysiological effect seems to come from the same place as the N400 that you find in semantic anomalies. But what about fMRI? Well, here we find that fMRI studies also consistently show modulation of activity in posterior temporal areas. This figure is from a nice study by Gold from a couple of years ago, and this red splotch shows the effect of priming. And so what is going on with the temporal areas in these studies? Lau, Phillips, & Poeppel, 2008, Nat. Revs. Neurosci.

Controlled Processing
Gold et al., 2006, J. Neurosci. Lexical Priming fMRI  MTG Controlled Processing fMRI  IFG Well here it seems that they are only activated in studies that test priming with longer SOA’s between words. If there’s a long SOA, you consistently see inferior frontal activation, and if there’s a short SOA you don’t. This suggests that the temporal areas are involved in more general lexical access processes, and that the inferior frontal areas are specifically engaged by some kind of controlled processes that only take place when there’s a long gap between primes and targets. Lau, Phillips, & Poeppel, 2008, Nat. Revs. Neurosci.

Why the contrasts? Lexical priming effects in MEG & fMRI: same localization  MTG Sentence level lexical expectancy in MEG  MTG Consistent with ‘lexical account’ of N400 effect … though not necessary for current argument (Kutas & Federmeier, 2000; van Berkum, 2008; Lau et al., 2008) The processes that are modulated by lexical expectancy/anomaly are likely engaged by every (open class) word of a sentence  poor fit to hemodynamic response function (HRF) So how does this fit together? XXX When we look at lexical priming effects, we find that MEG and fMRI are in agreement – the same temporal areas are involved. XXX And in MEG we find that manipulation of sentence level expectancy gives effects in the same region. XXX This is consistent with the so-called “lexical” account of the N400 effect, though that’s not necessary for the point I’m making here.XXX What IS important is that it’s probably the case that whatever processes are modulated in cases of semantic anomaly are the same normal processes that are engaged by every word of a sentence. So we have the same area engaged a whole bunch of times over the course of a sentence. … And that’s the kind of thing that could easily get in the way of localizing the specific effect of semantic anomaly.

Overlapping HRFs So you may know that the standard blood flow response to a stimulus is really long, about 12 seconds. And what we generally try to do in fMRI studies is to fit this hemodynamic response function to variation in the ongoing BOLD signal. But if you’ve got a sentence study where words are presented at least twice a second, this response is going to overlap very tightly with the response to the previous words. XXX-XXX-XXX And that’s going to work against the possibility of isolating any variation in the response that is specific to your unexpected word. So the problem here is that fMRI integrates information across an interval that for psycholinguists is pretty long, and so makes it hard to pick out one amongst a whole bunch of similar events that happen within that interval.

Automatic vs. Controlled Processes

DOCTOR NURSE DINRUP NURSE COUCH NURSE
Semantic association  facilitation [consistent] No association  inhibition [sometimes] Controlled/strategic effects Long SOA (Stimulus Onset Asynchrony), e.g. > 500ms Explicit pairing of words High proportion of associated pairs

(Automatic) Spreading Activation

High ambig: The shell was fired towards the tank.
Rodd, Davis, & Johnsrude, 2005, Cereb. Cortex High ambig: The shell was fired towards the tank. Low ambig: Her secrets were written in her diary.

Lau, Phillips, & Poeppel, 2008, Nature Rev. Neurosci.
So this then leads to a picture where we can separate the role of different brain areas in processing words in context. Under this view, the basic N400 effect seen with ERPs reflects primarily lexical processes, and it has primarily a posterior temporal source. But these lexical processes are enormously influenced by interaction with a variety of other mechanisms, including combinatorial representations of sentence context in anterior temporal areas and the angular gyrus, and control mechanisms in the inferior frontal lobe. This can make sense of the discrepancies in the lexical priming studies. But what about the sentence anomaly studies? Lau, Phillips, & Poeppel, 2008, Nature Rev. Neurosci.

fMRI-MEG discrepancy for sentence-level anomalies
Inferior frontal Posterior temporal fMRI consistent inconsistent MEG cccasional† fMRI integrates activity across several seconds Posterior temporal: effect of context on lexical retrieval at target word minor relative to many lexical retrieval events in whole sentence Inferior frontal: control processes engaged by anomaly stand out relative to sentence Why do we find that in sentence anomaly studies inferior frontal activations are dominant with fMRI and posterior temporal activations in MEG? To understand this it’s useful to go back to the word-by-word MEG activity figure that I showed you right at the start. Here we’re seeing the series of lexical retrieval events that span the entire sentence. There are a lot of these lexical retrieval events. And relative to those the effect of context on just one word is pretty small, and that may be why it’s harder to see in fMRI. In contrast, the control processes engaged by anomaly processing, and reflected in inferior frontal activations, may stand out relative to the sentence as a whole. Amy bought the napkins that the café manager diligently folded in the booth. Amy bought the napkins that the café manager diligently baked in the booth.

Interim conclusion … N400 reflects activity in pMTG associated with lexical activation This activity is modulated by context

Discrepancy: behavioral vs. EEG/MEG evidence
Generalization #1: context facilitates word recognition Generalization #2: but homophone processing shows early context-independent profile  generation of lexical candidates is based on bottom-up information alone Generalization #3: N400 effect, which is strongly modulated by cloze probability (= context?) is associated with lexical access  lexical access is directly influenced by context Possible resolutions: 1. Yes – we have a problem! 2. No – timing difference 3. No – the measures show differential sensitivity 4. No – these are different notions of ‘context effect’ 5. No – the interpretation of the N400 is incorrect

Category associations
Fischler et al A robin is not a bird. A robin is not a car. which elicits the larger N400?

Lau et al. 2009

The day was breezy so the boy went out to fly …
deLong, Urbach, & Kutas, 2005, Nature Neurosci.

(Kutas & Federmeier 2000)

‘baseball’ is not at all plausible here, yet it elicits a smaller N400 - why?
(Kutas & Federmaier 2000)

Federmeier & Kutas 1999

van Berkum et al. 1999

But see Delogu et al …

Next … Generating and selecting words in context
Fixed expressions Lexical associations Richer use of context … somehow Eyes and brainwaves - done Odd expectations Negation Associate boost Role reversal “astonish the article” “tell the suitcase” Moses illusion

Generating predictions
The gardener talked as the barber trimmed the __________ … The barber talked as the gardener trimmed the __________ … mustache hedge Just as an example, try to complete this sentence: [CLICK] The gardener talked as the barber trimmed the __________ … mustache? beard? How about this? [CLICK] The barber talked as the gardener trimmed the __________ … hedge? tree? branches? This exercise shows that we CAN make predictions about upcoming input. Further, it shows that our predictions can be quite sophisticated, such that even small changes in the context can lead us to make very different predictions. This kind of task (aka a “cloze task”) is typically used to find out what kinds of predictions comprehenders make, and we tend to assume that the responses that people give in a cloze task reflect the output of predictions. But where do these outputs come from? [CLICK]

Category associations
Fischler et al A robin is not a bird. A robin is not a car. which elicits the larger N400?

Nieuwland & Kuperberg, Psych. Sci., 2008

Pragmatically licensed negation Pragmatically unlicensed negation
(mean ERP values, ms, averaged across 12 posterior electrodes)

“Unfortunately, neither the memory retrieval account nor the integration account of the N400 component is particularly well-specified. For example, under the memory retrieval hypothesis, what remains to be specified is how exactly rich pragmatic context can facilitate or hinder retrieval of word meanings.” (van Berkum 2008) Jos van Berkum

I’m not one to attribute every activity of man to climate change
I’m not going to solely blame all of man’s activities on changes in climate I’m not one to attribute every activity of man to climate change 9/30/08 I don’t know if there’s a systematic literature on the recognition – or failure to recognize – speech errors. Question – how representative/broad are such cases? Role-reversal errors in comprehension are real, but how pervasive are they? syntax-free interpretive mechanism recovery from error when intended meaning known 10/02/08

Generating predictions
The gardener talked as the barber trimmed the __________ … The barber talked as the gardener trimmed the __________ … mustache hedge Just as an example, try to complete this sentence: [CLICK] The gardener talked as the barber trimmed the __________ … mustache? beard? How about this? [CLICK] The barber talked as the gardener trimmed the __________ … hedge? tree? branches? This exercise shows that we CAN make predictions about upcoming input. Further, it shows that our predictions can be quite sophisticated, such that even small changes in the context can lead us to make very different predictions. This kind of task (aka a “cloze task”) is typically used to find out what kinds of predictions comprehenders make, and we tend to assume that the responses that people give in a cloze task reflect the output of predictions. But where do these outputs come from? [CLICK]

“Thematic P600s” P600 Evidence for independent semantic composition
__ The hearty meal was devoured … … The hearty meal was devouring … P600 Kim & Osterhout 2005, J. Mem. Lang. For breakfast the boys would only eat toast and jam. For breakfast the eggs would only eat toast and jam. Kuperberg et al., 2003, Cogn. Br. Res.; see also Kolk et al. 2003, Hoeks et al. 2004 Reviews: Kolk 2006; Kuperberg 2007; Bornkessel-Schlesewsky & Schlesewsky 2008 Evidence for independent semantic composition

N400 P600 __ The hearty meal was devoured …
… The hearty meal was devouring … … The dusty tabletop was devouring … Kim & Osterhout 2005, J. Mem. Lang. P600 N400 Motivation for independent semantic composition P600 as reflection of conflict between interpretation and surface form Evidence that the effect is specific to cases of semantic attraction

Three things to explain
Presence of P600 in cases of thematic/selectional anomaly (many demonstrations) Consistent lack of N400 effect in sentences with reversed thematic roles (Kolk et al. 2003; Hoeks et al. 2004; van Herten et al., 2005; van Herten et al. 2006; Kuperberg et al. 2003; Kuperberg et al. 2007; Kim & Osterhout 2005; Stroud & Phillips 2011; Chow & Phillips 2010; Ye & Zhou 2008; Vissers et al. 2007) Disappearance of P600 in absence of semantic attraction i.e., meal – devour vs. tabletop – devour (1 prior demonstration: Kim & Osterhout 2005)

No N400 P600 van Herten, Kolk, & Chwilla, 2005, Cog Br Res

Role Reversal The thief that the cop arrested. High cloze The cop that the thief arrested. Low cloze N400 amplitude typically tracks cloze. Not here. So we shifted to focusing on the N400 in the context of role reversals. The typical case is like this: in [X] the verb “arrest” is very likely. And if you take the same words and reverse their roles, the verb “arrest” is very unlikely. Normally we’d expect this to have a big effect on the N400. Because N400 tracks cloze. But not here.

N400’s blindness to role-reversals
Observed quite often: English: Kim & Osterhout (2005); Kuperberg et al. (2003; 2007), etc. Dutch: Kolk et al. (2003); Van Herten et al. (2005; 2006) Mandarin Chinese: Ye & Zhou (2008); Chow & Phillips (2013) Japanese: Oishi & Sakamoto (2010) Important People notice. The N400 does not. ___ canonical role-reversed No N400 effects! No N400 effects! No N400 effects! And this is a quite consistent finding across labs and languages. Not without exception, but quite consistent. One thing that’s really important to remember here. It’s not the case that *people* fail to notice role reversals. They consistently do, and you can see that reflected in the P600 that you almost always see in these studies. It’s just the N400 that doesn’t seem to care. If the N400 reflects the state of comprehenders lexico-conceptual expectations, then this implies that the N400 reflects a point in processing at which our lexical expectations are sensitive to word associations, but not to grammatical argument roles. That’s the part we’ve focused on since … I would like to highlight here that in all these studies the N400 seems to be modulated by lexical association only, and this contrasts very sharply with findings that, even when lexical factors are matched, the N400 is still sensitive to predictability or congruity (as in Van Berkum’s speaker identity study I mentioned above). For this talk we will take the presence of P600 effects to simply reflect that the processing system is sensitive to role-reversals at a time point later than the N400, and we won’t be discussing hypotheses about what processes underlie the P600. P600 effect P600 effect P600 effect Kim & Osterhout (2005) Van Herten et al. (2005) Kuperberg et al. (2007)

Nieuwland & van Berkum, 2005, Cog Br Res
[… story with many mentions of tourist and suitcase …] Next, the woman told the tourist that she thought he looked really trendy. Next, the woman told the suitcase that she thought he looked really trendy. No N400 P600 Nieuwland & van Berkum, 2005, Cog Br Res

B. Interference Control E. Kim & Osterhout (2005) Replication
A. Spanish The warning fue declared… The warning fue declaring… The apartment fue declaring… C-D. Interference The tall grass on the large lawn was mowed… The tall grass on the large lawn was mowing… The front porch beside the large lawn was mowing… The tall grass around the rural house was mowed… The tall grass around the rural house was mowing… The front porch of the rural house was mowing… B. Interference Control The large lawn was mowed… The large lawn was mowing… The rural house was mowing… E. Kim & Osterhout (2005) Replication The hearty meal was devoured … The hearty meal was devouring … The dusty tabletop was devouring … Seems that we get no effects of semantic attraction. Just effects of animacy violation, plus effects of lower lexical association. P600 Stroud & Phillips, 2011, in prep

English Replication Target items taken from Kim & Osterhout 2005, Exp 2, SOA changed 700ms  500ms

No unique effect of role-reversal
(effect of animacy violation on N400 is harder to assess) Mandarin Chinese Any effect of role-reversal beyond effects of animacy violation and lexical association? Chow & Phillips 2013

Dumb N400 ________ Canonical Philanthropist BA orphan adopt
________ Role-reversed Orphan BA philanthropist adopt P600 effect No N400 effects! emphasize how consistent this is across studies highlight how this surprising this is given all the evidence that the processes indexed by the N400 are highly sensitive to semantic/pragmatic/general world knowledge and the fact that they show a late positivity shows that it’s not a completely illusion, if there’s any illusion it’s just temporary. Chow & Phillips (2013) also in many previous studies

Chow, Wang, Lau, & Phillips, 2018
Evolving expectations I: Time helps Wing Yee’s study in Mandarin did a really nice job of showing the same thing. It’s about 6 years old, but it only appeared in the last month or so, because we went to 5 journals before we got lucky. (Probably half of this room has been a reviewer – so thank you!) First she showed that having low predictability or high predictability contexts doesn’t seem to matter. (Though we’ll have more on that in Lara Ehrenhofer’s talk later this afternoon.) Mandarin Chow, Wang, Lau, & Phillips, 2018

Evolving expectations I: Time helps But then Wing Yee added an adverbial phrase like “last summer” that lengthened the time between the arguments and the verb. That didn’t make a difference in the low-predictability contexts. But it did make a difference in the high predictability contexts. That suggests that time is related to prediction. Mandarin Chow, Wang, Lau, & Phillips, 2018

Evolving expectations I: Time helps And then finally she moved around the position of the adverbial phrase. If it went at the start of the sentence, so that the arguments were next to the verb, then it didn’t help, and there was still no N400 effect. If it went in between the arguments and the verb, then it did bring back the N400. Mandarin Chow, Wang, Lau, & Phillips, 2018

Chow et al. CUNY 2015 Evolving expectations I: Time helps ERP target
Cloze Probability Argument Reversals Short Long Offline The restaurant owner forgot… (a) which customer the waitress had … served 11.4% 15.1% 25.4% (b) which waitress the customer had … 3.6% 1.2% 0% Argument Substitution The secretary confirmed… (c) which illustrator the author had … hired 16.% 19.7% 27.7% (d) which readers the author had … 1.5% 0.2% The manager knew … SOA = 530ms had ISISHORT: 530ms ISILONG: 2030ms tcutoff = 900ms Audio recording (duration = 4s) Speeded Cloze Adapted from [Grant et al. ‘15] We’ve also found similar things using a version of the behavioral speeded cloze paradigm, borrowed from Adrian Staub and Meg Grant. But I won’t have time to go into that here. I don’t really have time to go into this, but ideally we’d like to be able to explore this using something cheaper and easier than EEG. We did explore this a couple of years ago, using a speeded cloze paradigm that we borrowed from Adrian Staub and Meg Grant. Basically, instead of measuring N400s, we simply gave people incomplete sentences and asked them to complete them under time pressure. We were interested in whether cloze responses would get “smarter” over time. This is roughly what we found: we elicited some role reversal completions when people were under time pressure. The rate of errors got lower with more time. And the completions in the canonical and reversed order diverged more as people had more time to respond. … This is an area we’d really like to look into further. It turns out that running the experiments is not so hard, but coding the responses is more challenging. Chow et al. CUNY 2015

Cloze via Argument Role Reversal
Evolving expectations II: Same cloze, different sources Cloze via Argument Role Reversal … which customer the waitress had served… (25.4%) … which waitress the customer had served… (zero) Cloze via Argument Substitution … which illustrator the author had hired… (27.7%) … which readers the author had hired… (zero) Another nice study by Wing Yee Chow adds a further piece of evidence, showing that different expectations are available on different time courses. What Wing Yee did in this study, thanks in large part to a heroic effort by Cybelle Smith, who’s now in Kara Federmeier’s lab, was hold cloze probability constant while varying the source of the cloze values. So, she created role reversal conditions where the canonical order verb had about 25% cloze, and the reversed order had zero cloze. So, in this case the words are identical, and the cloze difference is entirely due to the argument roles. And she also created conditions where the cloze difference was due to the choice of words. So in THIS case the cloze was around 27%, and then by substituting a different word, the cloze could be pushed to zero. When the cloze contrast was due to the words, there was a big N400 effect. When the cloze contrast was due to the argument roles, there was none. Chow et al. (2015); see Kukona et al for eye-tracking evidence.

What does extra time do? Less predictable verb More predictable verb
Time for predictions before verb Time for detection of implausibility at verb What does extra time do? Less predictable verb Canonical cloze: median 3% (Reversal cloze: 0%) More predictable verb Canonical cloze: median 62% (Reversal cloze: 0%) Experiment 3 Chow, Wang, Lau, & Phillips, 2013

Minimal Reversal Japanese
Role reversal by case-marker reversal (NOM-ACC) Two-word sentences (one argument dropped). Bee-NOM STING Fish-ACC CATCH Scholar-NOM STUDY God-ACC WORSHIP The lexical association between the words are exactly the same – what’s different is just the

Minimal Reversal Japanese
Role reversal by case-marker reversal (NOM-ACC) Two-word sentences (one argument dropped). ?Bee-ACC STING ?Fish-NOM CATCH ?Scholar-ACC STUDY ?God-NOM WORSHIP The lexical association between the words are exactly the same – what’s different is just the

Design + 蜂が刺す + 蜂が刺す Factor 1: Plausibility Plausible Implausible
Bee-NOM sting Fish-ACC catch Bee-ACC sting Fish-NOM catch Factor 2: SOA 800ms + 蜂が刺す Short + 蜂が刺す Two factors: Plausibility & SOA Long 1200ms Method details N =24, all right handed native Japanese speakers Delayed plausibility judgment; 160 Experimental (40 per condition) ‘filler’ sentences; Latin-Square

Result – Verb Short SOA Long SOA - Canonical (bee-NOM sting)
- Reversed (bee-ACC sting) Short SOA - Canonical (bee-NOM sting) - Reversed (bee-ACC sting) Long SOA

What do bees do? … and what gets done to bees? And how do we access that information? Looking in world knowledge to find events where bee-is-agent, or bee-is-patient

Prediction as memory search
Prediction = memory search Lexical prediction ~= semantic memory search Given bee-ACC … What does it takes to predict swat instead of sting? Find a predicate that typically takes bee as a patient. Can we find this item in semantic memory in one direct step? (probably not) Lexical association links: annotated for abstract roles? Event knowledge: encoded in terms of abstract roles? (agent, patient vs. stinger, stingee) Let me step back and ask – what does it take to come up with a word that is likely given bee-ACC? You want specifically a “verb that typically takes bee as a patient.”

Encoding and accessing structured information in memory

Spreading activation as search space reduction
STING BUZZ INSECT HONEY HIVE PAIN HURT SWAT .....bee It doesn’t have to be dumb – it can be for instance reflect complex event memory like schema. Quite similar to the process involved in semantic priming (= fast and automatic but not very specific)

Spotting the right items
Pain Sting Buzz Honey Swat Insect Hive For the sake of simplicity, let’s assume that spreading activation can be constrained by grammatical category (this is an interesting empirical questions)

Same pattern in verb-final structure in Chinese:
Last week policeman BA suspect arrest… Last week suspect BA policeman arrest… Policeman BA suspect ZAI last week arrest… Suspect BA Policeman ZAI last week arrest… Same cloze contrast, different source: … which customer the waitress had served… (25.4%) … which waitress the customer had served… (zero) … which illustrator the author had hired… (27.7%) … which readers the author had hired… (zero) 1) Some recent evidence by Wing Yee Chow suggest that the role reversal does NOT fail to affect prediction. It is just slow. Chow et al. (2015 ab); see Kukona et al for eye-tracking evidence.

A Classic Case for Rapid Interpretation
Garnsey, Tanenhaus & Chapman (1989) Garnsey and collegues looked at the processing of filler gap dependencies in English.

Garnsey et al. 1989 Plausible
The business man knew which customer the secretary called… Implausible The business man knew which article the secretary called… N400 effect Importantly, this is an English verb-final word order. N400 is greater in the implausible condition than in the plausible condition

What does it mean? Classic Interpretation
Plausible The business man knew which customer the secretary called… Implausible The business man knew which article the secretary called… Classic Interpretation rapid filler-gap dependency formation full interpretation prior to the verb Is this incompatible with the Chinese results? Perhaps certain types of information can impact predictive computations more quickly than others?

Experiment 4 (Inputs to prediction)
Do different types of information impact predictive computations on different time scales? Cloze probability manipulation: 26% vs. 0% in both cases Argument Role Argument Identity The restaurant owner forgot… which customer the waitress had served … which waitress the customer had served … The secretary confirmed… which illustrator the author had hired… which readers the author had hired... Argument role, just like the role-reversal manipulation I’ve shown you in Experiments 1 and 2. The same set of arguments, different order. Argument identity is a manipulation that is similar to what we saw in Garnsey’s study, manipulating the identity of the extract object NP. In both cases we have a cloze probability manipulation, such that the verb in the expected condition (in black) has on average 26% cloze, whereas the one in the unexpected condition (in red) has zero percent cloze. If comprehenders can immediately use the identity, but not the structural roles, of the arguments to predict an upcoming verb, then we should expect to see an N400 effect in the argument identity condition, but not in the argument role condition. Cybelle Smith

Experiment 4 (n=24) Argument Role Argument Identity No N400 effect
The restaurant owner forgot… which customer the waitress had served … which waitress the customer had served … The secretary confirmed… which illustrator the author had hired… which readers the author had hired... No N400 effect -7μV +7μV 1000ms N400 effect -7μV +7μV 1000ms Point to P600 (replicates previous work on role-reversed sentences) identical cloze contrasts  clearly different patterns CONCLUSION! This suggests that the identity of the arguments can impact comprehenders’ predictions more quickly than their structural roles. Comprehenders do rapidly detect the anomaly. But they fail to use the compositional interpretation of the pre-verbal NPs to generate differential verb predictions.

Experiment 5 (n=24) Argument Contrast Word Contrast N400 effect
The lawyer knew which defendant the judge had sentenced The judge knew which lawyer the judge had sentenced… John knew … which defendant the judge had sentenced… which lawyer the judge had sentenced... N400 effect N400 effect On the LEFT: the sentences have the same words, but the arguments of the verb differ On the RIGHT: the sentences have different words; the arguments of the verb differ The N400 effect is driven specifically by arguments of verb. The words are parsed successfully. It is just their argument roles that do not immediately impact predictions.

ERP Measures of Completing Filler-Gap Dependencies
So what about this classic finding about English NP-NP-V orders? The businessman knew which customer the secretary called … The businessman knew which article the secretary called … N400 ERP Measures of Completing Filler-Gap Dependencies Garnsey, Tanenhaus, & Chapman, 1989, JPR

‘Smart N400’ in Dutch V2 sentences
Kos, Vosse, van den Brink, & Hagoort (2010)

J. Cogn. Neurosci., 2010

Good fit to context Poor fit to context

State of Play Question #1: what representations/processes is the N400 sensitive to? Question #2: what representations/processes does the N400 itself reflect? Evidence on Question #1 N400 is sensitive to many high-level sources of information (e.g., van Berkum et al.) N400 is also sensitive to low-level lexical associations (Fischler et al. 1983, etc. etc.) Evidence on Question #2 – Lexical Best evidence involves sensitivity to lexical properties despite other information being available Facilitation of related-but-inappropriate words (Federmeier & Kutas 1999) Neuroanatomical localization (Lau et al. 2008) N400 insensitivity to higher-level semantic information that impacts P600 (Chow & Phillips, 2013) Evidence on Question #2 – Higher-level semantics Best evidence involves sensitivity to higher-level (in)congruity that conflicts with lexical availability Effects of semantic composition (Kuperberg et al. 2009; Baggio et al. 2009) Repeated name penalty (RNP) effects (Gordon et al.)

Discrepancy: behavioral vs. EEG/MEG evidence
Generalization #1: context facilitates word recognition Generalization #2: but homophone processing shows early context-independent profile  generation of lexical candidates is based on bottom-up information alone Generalization #3: N400 effect, which is strongly modulated by cloze probability (= context?) is associated with lexical access  lexical access is directly influenced by context Possible resolutions: 1. Yes – we have a problem! 2. No – timing difference 3. No – the measures show differential sensitivity 4. No – these are different notions of ‘context effect’ 5. No – the interpretation of the N400 is incorrect

Lexical Access and Context
Behavioral studies on interaction with context Measures: cross-modal priming/naming, eye-tracking in reading (ambiguity penalty), eye-tracking visual world (shape competitor looks) Evidence: multiple meanings of homophones considered despite context Evidence: mostly from consideration of ill-fitting meanings Conclusion: context influences generation of lexical candidates only in strongly constraining environments; all agree that context impacts selection More specific: context rarely prevents generation of semantically poor-fitting candidates Electrophysiological studies on interaction with context Measures: ERP, MEG Evidence: amplitude of N400(-like) component varies with fit to context Evidence: and there is independent reason to view this component as lexical Conclusion: context influences some measure of lexical access quite broadly More specific: context modulates difficulty of accessing unambiguous word meanings; at a higher level than familiarity-check that drives eye-movements

Putting the Pieces Together
Robustness & speed Homophone meanings (within/between categories) Types of contextual constraint Measures of access Mechanisms of context use Timing of contextual information

(Final) Assignment How does context affect word recognition? Synthesize the problem and the evidence for an intelligent non-expert. Slides will be uploaded 3pp max (single spaced) by Dec 7 (please)

M350 (based on research by Alec Marantz, Liina Pylkkänen, Martin Hackl & others)

Lexical access involves
Activation of lexical representations including activation of representations matching the input, and lateral inhibition between activated representations Followed by selection or decision involving competition among activated representations that are similar in form

M350 RESPONSE TO A VISUAL WORD 0 200 300 400 Time [msec] Sagittal view

MEG response components elicited by visually presented words in the lexical decision task
RMS analysis of component field patterns.

(Embick et al., 2001)

Neighbors & Competitors
Phonotactic probability sound combinations that are likely in English e.g. ride vs. gush Neighborhood density number of words with similar sounds ride, bide, sighed, rile, raid, guide, died, tried, hide, bride, rise, read, road, rhyme, etc. gush, lush, rush, gut, gull …

Behavioral evidence for dual effects
Same/different task (“low-level”) RTs to nonwords with a high phonotactic probability are speeded up. RT High probability: MIDE Sublexical frequency effect RT Low probability: YUSH Lexical decision task (“high-level”) RTs to nonwords with a high phonotactic probability are slowed down! High probability: RT MIDE Competition effect Low probability: RT YUSH (Vitevich and Luce 1997,1999)

Stimuli High probability Low probability Word BELL, LINE PAGE, DISH
Materials of Vitevich and Luce 1999 converted into orthographic stimuli. Four categories of 70 stimuli: High probability Low probability Word BELL, LINE PAGE, DISH Nonword MIDE, PAKE JIZE, YUSH High and low density words frequency matched. (Pylkkänen, Stringfellow, Marantz, Brain and Language, 2003)

Effect of probability/density (words)
* ** n.s. n.s. (Pylkkänen, Stringfellow, Marantz, Brain and Language, 2003)

Effect of probability/density (nonwords)
** * n.s. n.s. (Pylkkänen, Stringfellow, Marantz, Brain and Language, 2003)

M350 = 1st component sensitive to lexical factors but not affected by competition
Activation Competition Selection/Recognition TURN TURNIP level of activation TURF TURTLE resting level time Stimulus: TURN

Lexical Access: Generation & Selection

Similar presentations

Presentation on theme: "Lexical Access: Generation & Selection"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Lexical Access: Generation & Selection

Similar presentations

Presentation on theme: "Lexical Access: Generation & Selection"— Presentation transcript:

Similar presentations

About project

Feedback