Presentation is loading. Please wait.

Presentation is loading. Please wait.

Looking for a ‘gold standard’ to measure language complexity: What psycholinguistics and neurolinguistics can (and can’t) offer to formal linguistics.

Similar presentations


Presentation on theme: "Looking for a ‘gold standard’ to measure language complexity: What psycholinguistics and neurolinguistics can (and can’t) offer to formal linguistics."— Presentation transcript:

1 Looking for a ‘gold standard’ to measure language complexity: What psycholinguistics and neurolinguistics can (and can’t) offer to formal linguistics Lise Menn & Jill Duffield Linguistics Dept/Institute of Cognitive Science University of Colorado, Boulder priming (short-term effects) “Everything that speakers have experienced, planned or unplanned, externally or internally generated, over the preceding time intervals from milliseconds to minutes to decades, affects how hard it is for them to deploy grammar and lexicon at a given instant…A basic reason that general real-world knowledge influences the complexity involved in producing and comprehending words and constructions is that if a hearer has have good grounds for predicting what someone is likely to say (or write), they have more processing resources available to handle difficult syntax and obscure lexicon. One way to think of the moment-to-moment dynamics—the highly variable nature of what’s hard and what’s surprisingly easy—is in terms of reducing or increasing the competition; that is, of channeling the flow of activation more narrowly in the direction of response, or of having it spread out (when there is more competition, there is less activation going in any given direction). Having a strongly directed flow can give apparently simultaneous access to all parts of a construction, because rapid assembly looks like retrieval from storage. Conversely, being pulled in multiple directions – for example, when an utterance competes with well-established collocation patterns — slows down tasks like processing No ifs, ands, or buts. (Duffield and Menn submitted, p.18)

2 Background for our stance
Cross-linguistic work on basic morphosyntax in aphasia, and on the earliest stages of child phonology shows that these areas are loaded with individual and with language-specific differences. ‘Markedness’ keeps vanishing into the mist of unverifiability. It’s no guide to complexity. So, the issue of what’s simple and why, especially in those domains, has been a constant undercurrent and a frequent topic of discussion. Menn, L. & L.K Obler (eds.) (1990). Agrammatic Aphasia: A Cross-Linguistic Narrative Sourcebook. Philadelphia: John Benjamins. Menn, L. & M..M. Vihman (2011). Features in child phonology: inherent, emergent, or artefacts of analysis? In N. Clements and R. Ridouane, eds. Where Do Phonological Features Come From? Cognitive, physical and developmental bases of distinctive speech categories. Amsterdam: John Benjamins. 3/23/12 Grammatical Complexity Workshop

3 What we’re going to say – 1 to 4
1. What our brains find difficult is not always what grammars consider complex, partly because what’s hard for our brains is not constant; it depends on many factors. (It’s complicated.) 2. Proposed metrics for language or grammar complexity should correspond in some way to the ‘gold standard’ of what’s hard for our brains to process. 3. Language complexity measures will have to go beyond a single measure of grammar complexity, because complexity for speakers/writers is not the same as for hearers/readers. 4. Psycho- and neurolinguistics can’t provide a royal road to measuring the complexity of a grammar or a language, but they do provide tools to measure processing complexity of individual sentences/utterances for speakers vs. listeners or readers, and learners vs. skilled language users. Nicol, J,, C. Jakubowicz & M.-C. Goldblum/ Sensitivity to grammatical marking in English-speaking and French-speaking non-fluent aphasics. Aphasiology 10 (6): – both structure and frequency matter 3/23/12 Grammatical Complexity Workshop

4 What we’re going to say – 5 to 8
5, Psycho- and neurolinguistic studies indicate that a valid measure of complexity will have to integrate across many linguistic levels (including semantics) and take frequency into account. 6. This implies that construction-based and usage-based approaches to grammar can provide insights into how grammars can come closer to reflecting what our brains do. 7. But: Complexity measures must handle competition and how it gets resolved in both comprehension and production: the paradigmatic axis also plays a role in complexity. 8. Pragmatics/real-world knowledge are involved in resolving this competition. (Implications for practical applications are clear; for comparison of languages, much less so.) 3/23/12 Grammatical Complexity Workshop

5 Problematic examples to start from
Pourquoi l’aphasique peut-il dire: "Je ne peux pas le dire" et pas "Elle ne peut pas la chanter''? (Nespoulous & Lecours 1989) [Why can the aphasic person say I can’t say it but not She can’t sing it ?] Possible culprits: lexical frequency, collocation frequency (formula status?), emotional weight… Dressler’s (1991) work on Breton: A speaker with fluent aphasia tends to name pictures or examples of a single object using the plural form if the object itself is most frequently found in quantity (leaves, potatoes); using the dual if the object is usually found in pairs (eyes, hands). Goes against all notions of markedness. Relative frequency of particular inflected form of particular word must be the explanation. What brains find difficult is not always what grammar and linguists consider complex. Nespoulous, Jean-Luc, and André Roch Lecours Pourquoi l’aphasique peut-il dire: "Je ne peux pas le dire" et pas "Elle ne peut pas la chanter''? (De l'intérêt des dissociations verbales dans l'étude du « comportement verbal des aphasiques). Cahiers du Centre Interdisciplinaire des Sciences du Langage, Mélanges offerts à J. Verguin. Toulouse: Université de Toulouse-Le Mirail. Nespoulous, Jean-Luc, Chris Code, Jaques Virbel, and André Roch Lecours Hypotheses on the dissociation between “referential” and “modalizing” verbal behaviour in aphasia. Applied Psycholinguistics 19: Dressler, Wolfgang 1991 The sociolinguistic and patholinguistic attrition of Breton. In First language attrition, Herbert W. Seliger and Robert Michael Vago (eds.), Cambridge: Cambridge University Press. 3/23/12 Grammatical Complexity Workshop

6 Grammatical Complexity Workshop
Why is there a problem? 1. What our brains find difficult is not always what grammars consider complex, partly because what’s hard for our brains is not constant; it depends on many factors. We cannot equate ‘simpler’ with what is learned earlier, and the reason we cannot do it is that the neural networks change over the course of development. 3/23/12 Grammatical Complexity Workshop

7 Another example from aphasia
Aphasic verb tense production errors are often, as one might expect, substitutions of present tense for past tense. But the reverse seems to be true for at least some agrammatic aphasic speakers of Arabic (Mimouni & Jarema, 1997), Polish (Jarema & Kądzieława, 1990), and Korean (Halliwell, 2000) Mimouni, Z., & Jarema, G. (1997). Agrammatic aphasia in Arabic. Aphasiology, 11, 125–144. Jarema, G., & Kądzieława, D. (1990). Agrammatism in Polish: A case study. In L. Menn & L. K. Obler (Eds.), Agrammatic aphasia, Vol. II (pp. 817–894). Amsterdam: Benjamins. Halliwell, J. (2000). Korean agrammatic production. Aphasiology, 14, 1187–1203. 3/23/12 Grammatical Complexity Workshop

8 Grammatical Complexity Workshop
What brains find difficult is not always what grammars consider complex This is not just about speakers with brain damage. ‘Difficulty’ is tied to particular circumstances. Production/comprehension asymmetry: most obvious case: ambiguity. Speakers, who know what they intend to say, often produce utterances that are difficult for hearers because of ambiguity in their referring expressions (He did it!) and elsewhere Long history of studies of ambiguity resolution in psycholinguistics that demonstrates & relies on the processing difficulty caused by hearer’s or reader’s need to resolve ambiguity on-line. Other studies showing speakers have to put effort into being simple for their hearers (any teacher knows this!) With respect to ambiguity resolution in reading: Aro, M., & Wimmer, H. (2003). Learning to read: English in comparison to six more regular orthographies. Applied Psycholinguistics, 24, 621–635. See also Ferreira, V. S. (2008). Ambiguity, accessibility, and a division of labor for communicative success. Learning Motivations, 49, 3/23/12 Grammatical Complexity Workshop

9 Grammatical Complexity Workshop
What brains find difficult is not always what grammars consider complex Learning changes the brain (how could it fail to?), creating a learner/skilled user asymmetry So, relative ‘simple-to-complex’ rankings must shift as we learn our first languages (OT calls this constraint (re-)ranking). Phonotactics provides many uncontroversial examples Blevins’ (1995) illustrations of syllable types: Spanish and Sedang permit CCVC but not CVCC, while the reverse is true for Klamath and Finnish. Japanese speakers struggle with English /tr/ but routinely produce [tstʃ], e.g. in place name ‘Tsuchiura’) Blevins, Juliette The syllable in phonological theory. In Goldsmith, John A., The Handbook of Phonological Theory. Cambridge MA: Blackwell 3/23/12 Grammatical Complexity Workshop

10 Before we go on: An essential distinction
The ‘grammar of a language’ as an abstraction across speakers – the ‘patterns out there’ to be learned (E-language) The ‘grammar of a language’ as an abstraction across speakers isn’t directly testable by psycholinguistics/neurolinguistics. If that grammar is your main concern, what we have to say has to be mediated by your idea of the relationship between the grammar of a language and the grammar of each speaker. the grammar internal to a given speaker, which should be that speaker’s internal approximation to the ‘patterns out there’ (I-language). This is what we’re concerned with in this presentation. 3/23/12 Grammatical Complexity Workshop

11 But focusing on speaker-internal grammar is only a start
Making this distinction can handle (some) differences between learners (who have a cruder approximation to that abstract grammar) and skilled users (who have a better one). But there are more problems to deal with. One that we’ll keep coming back to: If there’s only one internal grammar, a single measure of its complexity can’t handle the the fact that what’s difficult for comprehension (e.g. ambiguity, unclear reference) is not necessarily difficult for production. 3/23/12 Grammatical Complexity Workshop

12 Grammatical Complexity Workshop
Back to our first three points, slightly elaborated: What our brains find difficult is not always what grammars consider complex, partly because what’s hard for our brains is not constant and depends on many factors. Proposed metrics for language or grammar complexity – at least for speaker-internal language or grammar – should correspond in some way to the empirical ‘gold standard’ of what’s hard for our brains to process. So language complexity measures that claim to be valid metrics for what’s in human minds will also have to go beyond a single measure of grammar or language complexity. it’s Complicated 3/23/12 Grammatical Complexity Workshop

13 And we still have the problem cases we started with:
Some aphasic people can say I can’t say it. but not She can’t sing it. A Breton speaker with fluent aphasia tends to name pictures or examples of a single object using the plural form if the object itself is most frequently found in quantity (leaves, potatoes) using the dual form if the object is usually found in pairs (eyes, hands). Breton - Dressler, ref. above 3/23/12 Grammatical Complexity Workshop

14 What can (or can’t) psycholinguistics & neurolinguistics offer?
3/23/12 Grammatical Complexity Workshop

15 What’s a ‘gold standard’? Why do linguists need one for complexity?
A rigorous standpoint, outside of particular formalisms and levels of language, to inform proposed measures of complexity Needed in order to test whether a proposed metric corresponds to measures of what our brains find effortful to process …just as a proposed metric of color must correspond to some psychophysical measure of human responses to color if it’s going to be useful in accounting for perception. A metric that is useful for calibrating printers may not do well at accounting for what colors people find similar. 3/23/12 Grammatical Complexity Workshop

16 Grammatical Complexity Workshop
Computational measures of utterance complexity need to be validated against processing measures –i.e., measures of performance Validating a particular formal analysis of processing (e.g., an analysis that can take the number of competing antecedents for a relative pronoun into account, an analysis that can take various aspects of frequency into account) puts us into the domains of psycholinguistics and neurolinguistics, as other speakers have already made clear. 3/23/12 Grammatical Complexity Workshop

17 Quotes on taking a psycholinguistic approach to grammatical complexity
…the emerging correlation between performance and grammars exists because grammars have conventionalized the preferences of performance, in proportion to their strength and in proportion to their number, as they apply to the relevant structures in the relevant language types. (Hawkins 2004) In order to test the hypothesis that typological distributions reflect processing complexity, an independently motivated, well-defined, and empirically assessable notion of processing difficulty is essential. (Jaeger & Tily 2010) …not only [should] grammatical theorists …be interested in performance modeling, but also …empirical facts about various aspects of performance can and should inform the development of the theory of linguistic competence.” –Sag & Wasow, 2011 …the competence-performance distinction acknowledges the value of the sort of work linguists do in their day-to-day research, while recognizing that this work eventually must be placed in a broader psychological context. (Jackendoff 2002) Hawkins, J. (2004). Efficiency and complexity in grammars. Oxford University Press, New York. Jaeger, T. Florian and Tily, H. (2011). Language Processing Complexity and Communicative Efficiency. WIRE: Cognitive Science 2 (3): Sag, I. A. & Wasow, T. (2011) Performance-Compatible Competence Grammar. In Robert Borsley and Kersti Borjars, eds. Non-Transformational Syntax: Formal and Explicit Models of Grammar. Oxford: Wiley-Blackwell. Jackendoff, R. (2002) Foundations of Language. Oxford University Press, New York. 3/23/12 Grammatical Complexity Workshop

18 Grammatical Complexity Workshop
Complexity measures must predict processing effort for different levels, and their interactions Psycholinguistic experiments with normal speakers Study of cognitive constraints and island effects (Hofmeister & Sag, 2010) Results: Island constraints interact with other features to affect processing effort, correlating with grammaticality judgments WH-island violations are processed more easily when the extracted element is complex (a WH-phrase) Which employee did Albert learn whether they dismissed after the annual performance review? processed more easily than Who did Albert learn whether they dismissed after the annual performance review? Hofmeister, P. & Sag, I. (2010). Cognitive constraints and island effects. Language 86(2) 3/23/12 Grammatical Complexity Workshop

19 Measuring complexity for speakers: Error rate as an example
Studies of formation of grammatical dependencies: even subject-verb agreement can be complex! Producing subject-verb agreement: a “local noun” embedded in a subject noun phrase may interfere with the production of agreement - but structure constrains that interference. (Bock & Cutting 1992) The key [PP to the wooden cabinet]… is / are… The key [PP to the wooden cabinets]… is / are… The key [RC that ___ opened the cabinet]… is / are… The key [RC that ___ opened the cabinets]… is / are… Agreement: Clausal boundaries encapsulate information—so while relative clauses may be structurally more complex, they don’t interfere with agreement relations as much as phrases: Bock, K., & Cutting, J. C. (1992). Regulating mental energy: Performance units in language production*1. Journal of Memory and Language, 31(1), attraction effect no attraction effect 3/23/12 Grammatical Complexity Workshop

20 Grammatical Complexity Workshop
And there’s a glitch: Complexity measures must be compatible with different performance metrics Speakers and listeners show different sensitivities to certain structures in processing tasks While error data Bock & Cutting (1992) showed that relative clauses isolate information interfering with agreement for speakers while prepositional phrase modifiers do not (i.e., a clause boundary effect), Tanner (2012) uses ERP and reading times to show no interaction between structure x local noun number x grammaticality (no clause bounding effect) in production The key [PP to the wooden cabinet(s)]… The key [RC that ___ opened the cabinet(s)]… in comprehension Tanner, D. (2012). Structural effects in agreement processing: ERP and reaction time evidence for comprehension/production asymmetries. Paper presented at the Annual Meeting of the Linguistics Society of America. Portland, OR. January 2012. effect of number of embedded noun differs across structure here, but not here 3/23/12 Grammatical Complexity Workshop

21 Grammatical Complexity Workshop
Glitch: Complexity measures must be compatible with different performance metrics Speakers and listeners show different sensitivities to certain structures in processing tasks Speaker-hearer asymmetries aren’t ‘just’ matters of discourse ambiguity…. which means that there can’t be just one measure of complexity of a sentence – comprehension and production may give different complexity rankings 3/23/12 Grammatical Complexity Workshop

22 Grammatical Complexity Workshop
Glitch: Different sensitivities to structures in different processing tasks Study of discourse and weight-based factors on relative clause extraposition: Francis & Michaelis (2012) used two different tasks to measure the combined effects of: Definiteness: e.g., (Some/The) research… VP length: e.g. (…was conducted/…has been conducted fairly recently) RC length: e.g., (….that refutes the existing theories/…that refutes the existing theories with very clear and convincing evidence) Judgment task: readers saw two versions of a relative clause sentence (e.g., "Further research that indicates...." vs. "Further research has been conducted...") Elicited production task: speakers were given three constituents - a subject NP, a relative clause, and a verb phrase, and asked to order those constituents in a full sentence. Francis, E. & Michaelis, L. (2012). Effects of weight and definiteness on speakers’ choice of clausal ordering in English. Paper presented at the Annual Meeting of the Linguistics Society of America. Portland, OR. January 2012. 3/23/12 Grammatical Complexity Workshop

23 Grammatical Complexity Workshop
Glitch: Speakers and listeners may show different sensitivities to structures in particular processing tasks In both experiments, indefinite subjects (e.g. "Some research" vs. "The research"). as opposed to definite ones, were more likely to be used with extraposed relative clauses. BUT: In the judgment (comprehension) task, readers preferred the extraposed version for longer relative clauses VP length, however, didn't matter In the production task, VP length did matter: shorter VPs were more likely to predict extraposed relative clauses. Relative clause length, however, didn't matter in production. No explanation for this particular pair of comprehension/production discrepancies yet… 3/23/12 Grammatical Complexity Workshop

24 Grammatical Complexity Workshop
So: We don’t have a ‘royal road’ to offer; how about a set of road-building tools? What tools do we have to measure utterance complexity? Complexity for the listener (comprehension) Neural correlates of relative effort: one current measure is Event-Related Potential (ERP); Errors, reaction time and eye-tracking can also be used. (Imaging tools. though glamorous, don’t have good enough spatiotemporal resolution yet) Complexity for the speaker (production) Error-based measures of relative effort comprise the majority of production research. Reaction time measures and eye-tracking measures also are useful. 3/23/12 Grammatical Complexity Workshop

25 Things you probably know about measuring performance
Available performance measures for both comprehension and production can only look at one word, utterance, or short passage at a time. Performance always has a large random element – minds differ, and they are simultaneously busy with many things besides the task set by the experimenter (wishing for coffee, worrying about politics or the weather…) So most measures have to be averaged over fairly large sets of similar items, and sometimes over speakers as well. 3/23/12 Grammatical Complexity Workshop

26 Grammatical Complexity Workshop
Measuring complexity for listeners: Event-Related Potential as an example An ERP study of metaphor comprehension (Lai, Curran, & Menn 2009) Comprehending conventional and novel metaphors: Lexical semantics affects comprehension effort when structure is held constant. Sense/nonsense judgment task, comparing listeners’ ERPs for these four semantic groups: Literal: Every soldier in the frontline was attacked. Conventional: Every point in my argument was attacked. Novel: Every second of our time was attacked. Anomalous: Every drop of rain was attacked. (assignment of sentences to groups checked by naïve subject ratings) Using a sensicality judgment task, we compared ERPs elicited by the same target word when it was used to end literal, conventional metaphorical, novel metaphorical and anomalous sentences. Lai, Vicky T., Tim Curran, & Lise Menn Comprehending conventional and novel metaphors: An ERP study. Brain Research 1284: 145–155. 3/23/12 Grammatical Complexity Workshop

27 Grammatical Complexity Workshop
Measuring complexity for listeners: Event-Related Potential as an example Result: Conventional metaphors required a short burst of additional processing effort when compared with literal sentences. Novel metaphors required a more sustained effort, similar to the effort observed in anomalous sentences. Literal: Every soldier in the frontline was attacked. Conventional: Every point in my argument was attacked. Novel: Every second of our time was attacked. Anomalous: Every drop of rain was attacked. Comprehension of metaphors involves an initial stage of mapping from one concept to another; such mappings are cognitively taxing, implying that complexity (as processing effort) involves more than structure. ERP matches our intuitions about complexity – and elaborates them. 3/23/12 Grammatical Complexity Workshop

28 Measuring complexity for speakers: more about agreement errors
Studies of formation of grammatical dependencies: even subject-verb agreement can be complex! Producing subject-verb agreement: a “local noun” embedded in a subject noun phrase may interfere with the production of agreement - but structure constrains that interference. (Bock & Cutting 1992) The key [PP to the cabinet]… is / are… The key [PP to the cabinets]… is / are… The key [RC that ___ locks the cabinets]… is / are… Bock, K., & Cutting, J. C. (1992). Regulating mental energy: Performance units in language production*1. Journal of Memory and Language, 31(1), 3/23/12 Grammatical Complexity Workshop

29 Measuring complexity for speakers: more about agreement errors
Producing subject-verb agreement: “Local nouns” that are embedded more deeply are less likely to interfere with agreement production (Franck, Vigliocco & Nicol, 2002) The threat [PP to the president [PP of the companies]] … is / are… The threat [PP to the presidents [PP of the company]] … is / are… so syntactic structure can directly affect production complexity – and here, the more complex structure has made one component of producing this sentence easier! Franck, J., Vigliocco, G., & Nicol, J. (2002). Subject-verb agreement errors in French and English : The role of syntactic hierarchy. Language and Cognitive Processes, 17(4), 3/23/12 Grammatical Complexity Workshop

30 Measuring complexity for speakers: Other measures:
Onset latencies – how long before speaker starts to respond Show difficulty in processing (e.g., competition between verb forms in subject-verb agreement: Haskell & MacDonald, 2003; Staub, 2009) Eye tracking Shows the interface between high-level message formulation and sentence planning (Brown-Schmidt & Tanenhaus, 2006) Directed elicitation of alternate forms Provides a measure of accessibility (e.g., production of optional complementizers allows speakers time to access upcoming constituents: Ferreira & Firato, 2002) Haskell, T. R. & MacDonald, M. C. (2003). Conflicting cues and competition in subject-verb agreement. Journal of Memory and Language, 48: Participants were asked to produce questions, e.g., “The plates for the party guests…..broken”; latencies reflect time to plan the verb Staub, A. (2009). On the interpretation of the number attraction effect: Response time evidence. Journal of Memory and Language, 60: Two-choice reaction time experiment, e.g., “The gang with the dangerous rivals…WAS/WERE”; Tested whether or not latencies were due to agreement processing or subject planning Brown-Schmidt, S. & Tanenhaus, M. (2006). Watching the eyes when talking about size: An investigation of message formulation and utterance planning. Journal of Memory and Language, 54: Ferreira, V. S. & Firato, C. E. (2002). Proactive interference effects on sentence production. Psychonomic Bulletin & Review, 9: 3/23/12 Grammatical Complexity Workshop

31 So that was the fourth point:
Psycho- and neurolinguistics can’t provide a royal road to measuring the complexity of a grammar or a language, but they do provide tools to measure processing complexity of individual sentences/utterances for speakers vs. listeners or readers, and learners vs. skilled language users. 3/23/12 Grammatical Complexity Workshop

32 Grammatical Complexity Workshop
5. A valid measure of complexity will have to integrate across many linguistic levels (including semantics), and take frequency into account. What does that mean, and what kinds of data support it? Let’s break it up into sub-claims that we can examine one at a time. 3/23/12 Grammatical Complexity Workshop

33 Grammatical Complexity Workshop
What valid complexity measures would have to do: 5. Complexity measures must predict processing effort for multiple levels, and for interactions among levels, taking frequency into account O’Grady, Wiliam To appear. Relative clauses: Processing and acquisition. In The acquisition of relative clauses: Functional and typological perspectives, Evan Kidd (ed.). Amsterdam: John Benjamins O’Grady: “… the interaction of simple elements and phenomena can yield systems and effects of a qualitatively different and more complex nature.” 3/23/12 Grammatical Complexity Workshop

34 Grammatical Complexity Workshop
First, let’s talk about processing effort for different levels and their interactions A testable general complexity measure would have to be able to make predictions about the effort for processing short individual utterances or passages, and correctly predict the relative effort needed. We’ve already seen that this effort depends partly on the choice of individual lexical items within those utterances or passages She can’t sing it; Every minute of our time was attacked. 3/23/12 Grammatical Complexity Workshop

35 Grammatical Complexity Workshop
5. Complexity measures must predict processing effort for different levels and their interactions While linguists may analyze a particular utterance into several components, the mind may store it and process it as a whole, or as both whole and analyzed. Work on idiom blends suggests that idioms are both analyzable and stored as lexical items (Cutting & Bock, 1997) Observed speech error: “Help all you want”, blended from idioms/formulas ‘Help yourself!’ and ‘Take all you want!’ - idiom blends like this respect the internal structure of each component, because their surface syntax remains well-formed, so the speaker must have access to those internal structures. As many theorists now argue, The ‘whole vs. analyzed’ opposition is much too crude, as many linguists have argued on theoretical grounds (Culicover 1999, among many others.) Idiom blends are important arguments for not assuming that idiomaticity and analyticity are opposite. Here are things that blend syntactically perfectly, and the same time that the semantics is somewhat opaquely assigned to the parts. Formal measures of complexity are typically understood to be syntactic measures. It’s not difficult to imagine some form of combination, some expansion of that, to incorporate word frequency such that you can make an utterance more difficult if it had less frequent words. But without taking into account the words and structures, the particular words appearing together in that structure; without taking constructions into account, you’re still going to miss essential contributors to both types of complexity (speaker and hearer) Cutting, J. C. & Bock, K. (1997). That’s the way the cookie bounces: Syntactic and semantic components of experimentally elicited idiom blends. Memory and Cognition, 25: Culicover, P. W Syntactic Nuts: Hard Cases, Syntactic Theory, and Language Acquisition. Oxford University Press. 3/23/12 Grammatical Complexity Workshop

36 Grammatical Complexity Workshop
5. Complexity measures must predict processing effort for different levels, and their interactions Start with some intuitive evidence for constructions Particular verbs may be specified or preferred by a particular construction, such as mind if in the politeness formula Do/will [person A] mind if [event X]? (Do you mind if I sit here?). Easy to process, and not completely fixed lexically. (This construction is nested in more general patterns, cf. Would your mother have a fit if I…) Conversely, when an unexpected word is used in a familiar construction – especially if it evokes a different construction - it can make the construction relatively harder to process (Would you care if I sit here?). so ‘effort for different levels’ needs to include ‘mixed levels’, e.g. structures with some lexical items specified 3/23/12 Grammatical Complexity Workshop

37 Grammatical Complexity Workshop
5. Complexity measures must predict processing effort for different levels and their interactions Psycho- and neurolinguistics also tell us that the ‘whole vs. analyzed’ opposition is too crude. We even have to go beyond constructions and talk about collocations. Collocations with high transitional probabilities, even when the collocations don’t form constructions (e.g., subject + aux collocations, he has or I am), are easier for speakers to produce. (That’s why we can have contractions across the NP-VP boundary!) Smooth flow across this boundary is well exemplified in fluent aphasias, e.g. French jargon aphasia (Lecours et al. 1981) Learning probabilities is a subconscious (procedural) and gradual process. Expectations that A will be followed by B, or that A will occur in structure α, become stronger over time, rather than clicking from ‘nothing’ to ‘all’.. Lecours et al. 1981 3/23/12 Grammatical Complexity Workshop

38 Grammatical Complexity Workshop
‘Agrammatic’ aphasic speakers show the effect of high sequential probabilities … forgot the wash the dishes ‘forgot that she was washing the dishes’ … I like the go home. ‘I’d like to go home.’ Once they have chosen the definite article to follow forget or like, these speakers are in trouble; both plug in familiar phrases (wash the dishes, go home) with appropriate semantic content, but in forms that cannot follow the. These utterances are difficult to explain in grammatical terms, because they show the article being substituted for the infinitive marker, and, even more strikingly, because the collocation ‘V+the’ goes across the major syntactic boundary between the verb and what should be the start of its NP object. Agrammatic aphasic speaker is SK (aka Mrs. K., Shirley), a participant for Menn in various published studies of agrammatic speech since 1991. 3/23/12 Grammatical Complexity Workshop

39 Grammatical Complexity Workshop
5. Complexity measures must predict processing effort...taking frequency into account We can’t escape dealing with usage and frequency. In the limit, there may be no empirically testable difference between ‘being stored as a unit’ and having very strong, predictable links from one sub-unit to another. So if your theory doesn’t permit multi-level or other kinds of complex ‘units’ and/or doesn’t recognize collocations that aren’t constructions, that’s not necessarily a problem. What you do need is – at least - a way to incorporate item frequencies and transition probabilities into the representation of a structure after the structure has its words filled in (or during the process of getting them filled in). 3/23/12 Grammatical Complexity Workshop

40 Grammatical Complexity Workshop
5. Complexity measures must predict processing effort for different levels, and their interactions We’ll shortly consider some more evidence for constructions, as structures lying between the extremes of lexical items and full clauses. But: the complexity of deploying a construction is not determined by construction frequency alone (e.g., Dutch word order in aphasia, Bastiaanse, Bouma & Post 2009) For example, the interpretation or deployment of a construction, such as the English subject or object cleft, may be made better or worse by the existence of similar constructions (Dick et al. 2001:772): the ‘paradigmatic axis' is relevant to processing complexity – more on that soon. Bastiaanse, Roelien, Gosse Bouma, and Wendy Post, Linguistic complexity and frequency in agrammatic speech production. Brain and Language 109: Dick, Frederick, Elizabeth Bates, Beverly Wulfeck, Jennifer Aydelott Utmann, Nina Dronkers, and Morton Ann Gernsbacher, 2001 Language deficits, localization and grammar: Evidence for a distributive model of language breakdown in aphasic patients and neurologically intact individuals. Psychological Review 108 (4) 3/23/12 Grammatical Complexity Workshop

41 Grammatical Complexity Workshop
5. Complexity measures must predict processing effort for different levels, and their interactions, taking frequency into account Purely structural complexity does affect normal and aphasic language processing (e.g., Thompson & Shapiro 2007 showed that practice on structurally more complex clause generalizes to improvement on less complex clauses, but practice on simple clauses doesn’t generalize to more complex ones). But in general: processing effort is a function of the interaction of structure and frequency at multiple levels, Let’s look at a psycholinguistic study of normal speakers and a related one of aphasic speakers which demonstrates this. Thompson, C. K., & Shapiro, L. P. (2007). Complexity in treatment of syntactic deficits. American Journal of Speech-Language Pathology, 16, 30–42. Thompson, C. K., Shapiro, L. P., Kiran, S., & Sobecks, J. (2003). The role of syntactic complexity in treatment of sentence deficits in agrammatic aphasia: The complexity account of treatment efficacy (CATE). Journal of Speech, Language, and Hearing Research, 46, 591–607. O’Grady, William. Relative clauses: Processing and acquisition. In The acquisition of relative clauses: Functional and typological perspectives, Evan Kidd (ed.). Amsterdam: John Benjamins 3/23/12 Grammatical Complexity Workshop

42 They shrank the sweater two sizes
5. Complexity measures must predict processing effort for different levels, and their interactions – taking frequency into account An example of frequency/structure interaction: Relative verb-(subcategorization) frame frequencies create a bias (reader’s expectation) that affects readers’ processing patterns and comprehension. ‘Shrink’, for example, has a syntactic bias towards the undergoer-subject argument structure – it is more frequently used in the ‘unccusative’ frame than in any other. The sweater shrank two sizes They shrank the sweater two sizes Susanne Gahl, Lise Menn, Gail Ramsberger, Daniel S. Jurafsky, Beth Elder, Molly Rewega, Audrey L. Holland (2003) Garnsey, Susan M.; Neal J. Pearlmutter; Elizabeth Myers; and Melanie A. Lotocky The contributions of verb bias and plausibility to the comprehension of temporarily ambiguous sentences. Journal of Memory & Language – 93. Gahl, S. & Garnsey, S. M Knowledge of grammar includes knowledge of syntactic probabilities. Language 82(2),

43 5. Complexity measures must predict processing effort for different levels, and their interactions – taking frequency into account The sweater shrank two sizes They shrank the sweater two sizes Eye movements during reading: verbs with similar syntactic biases pattern together, whereas verbs that are similar only in meaning do not, Garnsey et al Comprehension: Clauses that conform to a verb’s bias, - They proposed X, They suggested that Y - comprehended faster and more accurately than bias-violating sentences with the same structure - They proposed that X, They suggested Y (ibid.) Production studies supporting this: Gahl & Garnsey 2004, 2006. Syntactic frame and verb bias in aphasia: Plausibility judgments of undergoer-subject sentences. Gahl, Susanne, Menn, Lise, Ramsberger, Gail, Jurafsky, Daniel S., Elder, Elizabeth, Rewega, Molly, & Audrey L. Holland Brain and Cognition, 53: Garnsey, Susan M.; Neal J. Pearlmutter; Elizabeth Myers; and Melanie A. Lotocky The contributions of verb bias and plausibility to the comprehension of temporarily ambiguous sentences. Journal of Memory & Language – 93. Gahl, S. & Garnsey, S. M Knowledge of grammar, knowledge of usage: Syntactic probabilities affect pronunciation variation. Language 80(4), Gahl, S. & Garnsey, S. M Knowledge of grammar includes knowledge of syntactic probabilities. Language 82(2),

44 5. …taking both structure and frequency into account
Gahl et al. 2003: People with aphasia comprehend sentences better when the verb is in its preferred frame. Clauses with unaccusatives (undergoer-subjects) can be hard or easy, depending on how typical it is for the verb to be used in the unaccusative construction. This supports a processing model that has relatively direct semantic construal of the verb frames of simple clauses, rather than indirect construal involving, e.g., traces. the accessibility of a lexical item depends not only its frequency, but also on the density of its phonological and semantic neighborhoods (e.g., Gordon 2002; Mirman and Magnuson 2008), among other factors. … there is no standard approach to defining constructional neighborhood density,… constructions within a family of constructions sharing similar properties, such as subject-gap relative clauses and clefts sharing “SVO” or “NVN” word order, or the family of constructions having filler-gap dependencies (Sag 2010), may be subject to neighborhood effects similar to those of lexical items. (footnote 1, Duffield &Menn submitted)

45 Grammatical Complexity Workshop
The interaction of levels and the effect of frequency on complexity also support our sixth point: 6. Construction-based and usage-based approaches to grammar can provide insights into how grammars can come closer to reflecting what our brains do. Finally, let’s look at evidence for our last two points 3/23/12 Grammatical Complexity Workshop

46 Grammatical Complexity Workshop
a valid measure of complexity will have to integrate across many linguistic level, and furthermore… Complexity measures must be compatible with different performance metrics There can be no single performance metric. 3/23/12 Grammatical Complexity Workshop

47 Grammatical Complexity Workshop
Complexity measures must be compatible with different performance metrics There can be no single performance metric. Why not? What is effortful for speakers does not always match what is effortful for hearers Speakers and hearers have been shown to be sensitive to different structural features the same utterance types Therefore: The goal of an overall complexity measure needs to be split into sub-goals; several roughly commensurable but sometimes incompatible measures are required. Let’s look at some experimental evidence 3/23/12 Grammatical Complexity Workshop

48 Grammatical Complexity Workshop
Complexity measures must be compatible with different performance metrics Identifying referents--Production Cognitive demands can swamp information that would improve referential success (Wardlow Lane & Ferreira, 2008) Speakers are asked to name target objects (in the common ground) so that listeners can identify them, while being faced with privileged objects of varying saliency. Speakers could name the target object by only using information that is common ground (e.g., “the heart”) or by also using privileged information (e.g., “the small heart”) that is useless and possibly confusing to the hearer. Wardlow Lane, L., Ferreira, V.S., Speaker-external versus speaker-internal forces on utterance form: Do cognitive demands override threats to referential success? Journal of Experimental Psychology: Learning, Memory, and Cognition 34 (6), 3/23/12 Grammatical Complexity Workshop

49 Grammatical Complexity Workshop
Complexity measures must be compatible with different performance metrics Identifying referents—Production experiment Cognitive demands swamp information that would improve referential success (Wardlow Lane & Ferreira, 2008) Results: When the saliency of privileged information was increased, speakers made more reference to it, (e.g., identifying the target object as “the small heart”) than when such information was less salient, despite the risk of confusing the listener. In this task, cognitive demands result in a lower processing load for the speaker when producing more (complex) descriptions. Wardlow Lane, L., Ferreira, V.S., Speaker-external versus speaker-internal forces on utterance form: Do cognitive demands override threats to referential success? Journal of Experimental Psychology: Learning, Memory, and Cognition 34 (6), 3/23/12 Grammatical Complexity Workshop

50 Grammatical Complexity Workshop
Complexity measures must be compatible with different performance metrics Identifying referents—Comprehension experiment using eye-tracking (following the listener’s successive visual fixations) Visual context affects ambiguity resolution (Spivey, Tanenhaus, Eberhard & Sedivy, 2002) Four conditions: Listeners heard either Put the apple on the towel in the box or Put the apple that’s on the towel in the box Sometimes what they saw was this: Spivey, Michael J., Tanenhaus, Michael K., Eberhard, Kathleen M., & Sedivy, Julie C Eye movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution. Cognitive Psychology 45, A. ‘on the towel’ is redundant: there’s only one apple 3/23/12 Grammatical Complexity Workshop

51 Grammatical Complexity Workshop
Complexity measures must be compatible with different performance metrics Identifying referents—Comprehension experiment using eye-tracking (following the listener’s successive visual fixations) Visual context affects ambiguity resolution (Spivey, Tanenhaus, Eberhard & Sedivy, 2002) Four conditions: Listeners heard either Put the apple on the towel in the box or Put the apple that’s on the towel in the box And sometimes what they saw was this: B. ‘on the towel’ is relevant: there are two apples Spivey, Michael J., Tanenhaus, Michael K., Eberhard, Kathleen M., & Sedivy, Julie C Eye movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution. Cognitive Psychology 45, 3/23/12 Grammatical Complexity Workshop

52 Grammatical Complexity Workshop
Complexity measures must be compatible with different performance metrics Put the apple on the towel in the box. Listeners’ eye movements were recorded to see whether they treated the PP modifier (e.g., “…on the towel”) in the instruction as a goal, in which case they would look to the empty towel, or as a modifier, in which case they would look to the apple on the towel, and not to the empty towel. Spivey, Michael J., Tanenhaus, Michael K., Eberhard, Kathleen M., & Sedivy, Julie C Eye movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution. Cognitive Psychology 45, A. redundant info: one apple B. relevant info: two apples 3/23/12 Grammatical Complexity Workshop

53 Grammatical Complexity Workshop
Complexity measures must be compatible with different performance metrics Participants heard instructions in one of these two forms: - temporarily ambiguous: Put the apple on the towel in the box or - unambiguous (control): Put the apple that’s on the towel in the box Results: In the one-referent visual condition listeners were more likely to look towards the empty towel – that is, to treat the PP modifier on the towel in this temporarily ambiguous instruction as a goal - than in the two-referent instruction. Spivey, Michael J., Tanenhaus, Michael K., Eberhard, Kathleen M., & Sedivy, Julie C Eye movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution. Cognitive Psychology 45, A. redundant source info: one apple B. useful source info: two apples 3/23/12 Grammatical Complexity Workshop

54 Grammatical Complexity Workshop
Complexity measures must be compatible with different performance metrics Results - 2: In the two-referent condition (B), there was no difference between the responses to the temporarily ambiguous Put the apple on the towel in the box and the unambiguous Put the apple that’s on the towel in the box. Participants looked to the correct apple, and then to the correct goal (the box), but not to the false goal (the towel). Spivey, Michael J., Tanenhaus, Michael K., Eberhard, Kathleen M., & Sedivy, Julie C Eye movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution. Cognitive Psychology 45, A. redundantsource info: one apple B. useful source info: two apples 3/23/12 Grammatical Complexity Workshop

55 Grammatical Complexity Workshop
Complexity measures must be compatible with different performance metrics So the visual context – the second apple – in display (B) resulted in a lower processing load for the listeners who heard the temporarily ambiguous (garden-path) Put the apple on the towel in the box instruction. The structural ambiguity (does ‘on the towel’ modify the apple or describe what is to be done with it?) apparently added no complexity/processing load for them. Spivey, Michael J., Tanenhaus, Michael K., Eberhard, Kathleen M., & Sedivy, Julie C Eye movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution. Cognitive Psychology 45, 3/23/12 Grammatical Complexity Workshop

56 Grammatical Complexity Workshop
Complexity measures must be compatible with different performance metrics So we’ve seen that both speaker and hearer are influenced by their environments. Speakers may produce something that’s more complex because it’s easier to say in the context, even though that context can make the utterance more difficult for the listener. (Wardlow Lane & Ferreira, 2008) Listeners can easily comprehend something that’s more complex by making use of the visual context. (Spivey et al., 2002) Spivey, Michael J., Tanenhaus, Michael K., Eberhard, Kathleen M., & Sedivy, Julie C Eye movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution. Cognitive Psychology 45, Wardlow Lane, L., Ferreira, V.S., Speaker-external versus speaker-internal forces on utterance form: Do cognitive demands override threats to referential success? Journal of Experimental Psychology: Learning, Memory, and Cognition 34 (6), 3/23/12 Grammatical Complexity Workshop

57 Grammatical Complexity Workshop
What valid complexity measures would have to do: 7. Complexity measures must handle competition and how it gets resolved in both comprehension and production 3/23/12 Grammatical Complexity Workshop

58 Grammatical Complexity Workshop
7. Complexity measures must handle competition and how it gets resolved in both comprehension and production Processing is strongly influenced by competition – normal and aphasic speakers have both substitution errors and blends. Competing lexical items - substitutions . addressing a child using the name of their sibling tip-of-the-tongue (Bhatnagar – Baharav – Bhuvana!) aphasic word search: my right side was blunt- lump- trip- slip- Competing members of an inflectional paradigm Competing interpretations (visiting relatives…, garden paths) Competing possible continuations of a partially-produced structure. 3/23/12 Grammatical Complexity Workshop

59 Grammatical Complexity Workshop
7. Complexity measures must handle competition and how it gets resolved in both comprehension and production English-speaking aphasic speaker struggles: ‘And the boy give to a cookie—the boy give to girl a cookie,’ apparently torn between double object and prepositional constructions. More errors of article choice among people with agrammatic aphasia in German than in Italian. A German speaker struggles to find the right form: “die...der...das...die...den..den Hund” (Bates, Wulfeck, & MacWhinney 1991) More verb inflection errors among people with agrammatic aphasia in languages with richer verb paradigms. So: the ‘paradigmatic axis’, as well as the structural ‘syntagmatic axis’, affects the complexity of deploying a given structure. Bates, Wulfeck, & MacWhinney Crosslinguistic Research In Aphasia: An Overview. Brain and Language, 41, 3/23/12 Grammatical Complexity Workshop

60 Grammatical Complexity Workshop
What valid complexity measures would have to accommodate: 8. Pragmatics/real-world knowledge are involved in resolving this competition 3/23/12 Grammatical Complexity Workshop

61 Grammatical Complexity Workshop
8. Pragmatics/real-world knowledge are involved in resolving this competition Pragmatics has an enormous effect on processing We already saw this in the metaphor processing study Other speakers here have also made this point Let’s look at a clear, reasonably controlled example… Suggested: Kaiser & Trueswell (2004) The role of discourse context in the processing of a flexible word-order language Mak, Willem, Wietske Vonk & Herbert Schrieffers Discourse structure and relative clause processing. Memory and cognition 36 (1) Gibson, Edward (1998) Linguistic complexity: locality of syntactic dependencies. Cognition 68: 1-76. 3/23/12 Grammatical Complexity Workshop

62 Grammatical Complexity Workshop
What’s this? 3/23/12 Grammatical Complexity Workshop

63 Grammatical Complexity Workshop
What’s this? 3/23/12 Grammatical Complexity Workshop

64 Grammatical Complexity Workshop
What’s this? 3/23/12 Grammatical Complexity Workshop

65 Grammatical Complexity Workshop
8. Pragmatics/real-world knowledge is involved in resolving this competition Competing perspectives add to the difficulty of choosing among truth-value-equivalent constructions There’s a table with a lamp, A lamp on a table – we’re rarely tempted to say There’s a table under the lamp ! There’s a bed with a pillow at the wrong end or A pillow at the foot of a bed – we don’t hesitate to choose the larger object as the location. In the third picture, we’re slowed up by the four-way competition among possible orders of mention and choice of location: The footstool is behind the armchair The armchair has a footstool behind it There’s a footstool with an armchair in front of it There’s an armchair in front of a footstool. 3/23/12 Grammatical Complexity Workshop

66 Grammatical Complexity Workshop
8. Pragmatics/real-world knowledge is involved in resolving this competition Performance data like this lead us to conceptualize complexity as requiring an integration of fluctuating levels of competition among different types of structures, varying from moment to moment as well as from speaker to speaker. 3/23/12 Grammatical Complexity Workshop

67 Grammatical Complexity Workshop
Wrapping up 3/23/12 Grammatical Complexity Workshop

68 To evaluate complexity measures for speaker-internal grammar, we need:
Studies with systematic variation of structures and lexical items that would let us zero in on how construction frequency, lexical frequency, and other sources of processing difficulty interact. Should be carried out with normal speakers using ERP or other sensitive measures of processing load, because the number of linguistic variables is so high that a good design would impose serious burdens on a speaker with aphasia. 3/23/12 Grammatical Complexity Workshop

69 Implications for a valid complexity measure
Point 1 – the need to work across unit sizes/levels and to predict the interaction of frequency and structure (including transition probabilities) are arguments for framing a processing account in terms of a formalization of grammar that takes surface structure (constructions) and usage into account Point 2 – compatibility with different performance metrics - implies that the goal of a creating formal processing complexity metric needs to be split into sub-goals for comprehension, production, and learning. Point 3 – that competition adds to complexity – implies that the ‘paradigmatic axis’ as well as the structural ‘syntagmatic axis’ is affects the complexity of a given structure. Point 4 – the need to deal with the impact of real-world situations on processing - suggests that any formal metric, even one that satisfies the above requirements - will be incomplete. But that doesn’t mean it won’t be interesting and useful! 3/23/12 Grammatical Complexity Workshop

70 How might a formal system handle this?
The construction grammar approach treats language as an inventory of constructions (both lexical and combinatoric) (abstract entities that are the loci of constraints on the interface of form & meaning; Sag, Boas, & Kay 2012). Each construction has specific information about its properties; we don’t assume generalizations across utterances simply because of structural similarity. For example, there are unique properties of various filler-gap constructions (Sag, 2010). By using a type hierarchy, we can represent different grains of constructions. Sag, Ivan A., Hans C. Boas, and Paul Kay. Introducing Sign-Based Construction Grammar. To appear in H.C. Boas and I.A. Sag (Eds.) Sign-Based Construction Grammar. Stanford: CSLI Publications. `Final' Version of March 8, 2012, downloadable from Sag, I. A. (2010). English filler-gap constructions. Language, 86(3), 3/23/12 Grammatical Complexity Workshop

71 How might a formal system handle this?
Furthermore, constructions contain more than just syntactic information—they contain semantic and pragmatic information as well, thus formalizing these elements. These features may allow us to deal with the interaction of different levels (morphological, lexical, syntactic, semantic & pragmatic) although it does not directly address the issue of frequency and transition probabilities. The unique specification of construction properties organized in a type hierarchy could allow us to predict certain types of competition. For example, if two constructions are specified for the same lexical item, argument structure, or co-occurrence restrictions, we would expect there to be competition between those two constructions—although the competition itself is not formally represented. 3/23/12 Grammatical Complexity Workshop

72 Grammatical Complexity Workshop
it’s Complicated 3/23/12 Grammatical Complexity Workshop

73 Grammatical Complexity Workshop
Thank you! Special thanks to Laura Michaelis-Cummings for steering us to good readings and giving feedback. 3/23/12 Grammatical Complexity Workshop


Download ppt "Looking for a ‘gold standard’ to measure language complexity: What psycholinguistics and neurolinguistics can (and can’t) offer to formal linguistics."

Similar presentations


Ads by Google