Identifying Criterial Features for Reference Level Descriptions John A. Hawkins (1,2) & Luna Filipović (1) (1) University of Cambridge, (2) University.

Identifying Criterial Features for Reference Level Descriptions John A. Hawkins (1,2) & Luna Filipović (1) (1) University of Cambridge, (2) University of California Davis

The CEFR levels have been defined hitherto in functional terms, i.e. in terms of the uses to which language can be put and the various functions that learners can perform as they gradually master a second language. See e.g. the "can-do" statements of the Council of Europe (2001) document, pp.244-257.

Learners at B1 “can express opinions on abstract/cultural matters in a limited way or offer advice within a known area”. Learners at B2 “can follow or give a talk on a familiar topic”. Learners at C1 “can contribute effectively to meetings and seminars within own area of work or keep up a casual conversation with a good deal of fluency”.

By focusing on functions the CEFR attempts to define these levels in a way that is independent of the different grammatical and lexical details of the languages of Europe.

But learners who perform each of these functions may be using a wide variety of grammatical constructions and words in order to do so. And the ability to do each of the tasks does not tell us with precision which grammatical and lexical properties of English (and of other target languages) the learner actually knows and uses at each level.

We need this precision and added specificity, because the grammatical and lexical details of each language are key properties that examiners look for when they assign candidates to a proficiency level and score, because teachers need to teach these properties and because learners need to learn them.

So in this talk we ask: 1) how much of the grammar and lexicon of English do learners actually know and/or produce at each of these CEFR levels? 2) and what patterns and principles are there in these developing second language acquisition stages of English?

The way we have chosen to proceed is to search for “criterial features”. These are properties of English (constructions, words, rule types, errors and their frequencies, etc) that are distinctive and characteristic of L2 proficiency at the different levels. See Hawkins & Filipović (2011), Hawkins & Buttery (2009, 2010).

Criterial features can be found in all areas of English: syntax, morphology, phonology, the lexicon, semantics, and discourse. They enable us to distinguish higher proficiency levels from lower levels in an efficient way.

An analogy for Criterial Features The defining characteristics for recognising faces in a police identikit. You don’t need to see all the features of a person’s face in order to distinguish that person from others, just the important defining characteristics that capture essential qualities Criterial features are similar: they capture essential distinguishing properties of the CEFR proficiency levels.

Another example: Languages change over time, and when historians of English examine Old English, Middle English and Early Modern English, they focus on important differences between these stages, not on what stayed the same. So too in second language learning. We need to discover what is characteristic of each level as learners progress.

More generally, this research programme is centered on the notion of criteriality and on criterial differences between levels. I.e. we are trying to capture key changes from level to level in what has been learned and in what can be produced, based NOT on what we hope learners have learned and on what they have been taught, but based on an empirical investigation of what they actually do, in the CLC.

Electronic corpora of learner English make it possible for us to discover criterial features. The Cambridge Learner Corpus (CLC) gives us empirical evidence for developmental stages in the learning of new constructions, words and word meanings. It also gives us quantitative data on learner errors in syntax, morpho-syntax and lexical choice.

This corpus has been tagged for parts of speech and parsed using a sophisticated automatic parser ( Briscoe et al. 2006 ) permitting numerous grammatical and lexical searches to be conducted. Between a third and one half has been error-coded using codes devised by researchers at CUP.

Sample Error Codes in the CLC RNReplace noun Have a good travel (journey) RV Replace verb I existed last weekend in London (spent) MD Missing determiner I spoke to President (the) I have car (a) AGVVerb agreement error The three birds is singing (are) IVIncorrect Verb Inflection I spended last week in London (spent) FJWrong Adjective Form The situation got worst (worse) UQUnnecessary Quantifier A little bit quite common (quite common) DYDerivation of Adverb It happened fastly (fast)

Briscoe’s RASP (Robust Accurate Statistical Parser) tagging identifies parts of speech (PoS) probabilistically tagging parse generates a parse forest representation containing all possible subanalyses with associated probabilities weighted Grammatical Relations weighted Grammatical Relations yielded by the n- best parses of the input.

There are different types of criterial features (Hawkins & Buttery 2010, Hawkins & Filipović 2011). Here we refer to just two: Positive Linguistic Features These refer to positive, i.e. correct, linguistic properties of English that have been acquired at a certain L2 level and that generally persist at all higher levels. A property P (e.g. a new construction type) acquired at B2 may differentiate that level and higher levels from [A1, A2, B1] and will be criterial for the former. Or P may be acquired at C2 and differentiate this level from all lower levels.

Negative linguistic features These are incorrect properties of English, or errors, that occur at a certain level or levels and with a characteristic frequency. Both the presence versus absence of the errors, and especially their frequency (the "error bandwidth"), can be criterial for the level(s).

Examples of Positive Criterial Features The A levels (A1 and A2) Simple intransitive clauses (NP-V) and the slightly more complex transitive (NP-V-NP) sentence types are present from the beginning: He went. (NP-V) A1 He loved her. (NP-V-NP) A1

Modal auxiliary verbs like may, might, can and must appear first at A1 or A2, but only in some of their senses. Can is first attested in the PERMISSION sense at A1 and in the POSSIBILITY sense at A2: And if you want, you can bring pencils or pens. (PERMISSION) A1 In this magazine you can see all the new C.D.[s] and all the dates of the concerts. (POSSIBILITY) A2

Noun Phrase sequences of Pronoun plus Infinitive are found at A2: something to eatA2 nothing to doA2 as are postnominal modifiers with participial –ed: beautiful paintings [painted by famous Iranian painters] A2

Lexical verbs appearing at the A levels are typically among the most basic and frequent verbs of English; they appear first in their most basic and frequent senses. Verbs attested at A1 include: catch, eat, give, put, take and walk

New lexical verbs appearing at A2 include: break, cut, hit, push, stand, and fall again typically in their most basic and literal senses. For break this includes its primary physical sense: I broke a beautiful glass. A2 for cut it includes the following example in its primary sense: First I cut the cake with my mother. A2

The B Levels (B1 and B2) The new features at B1 involve more complex syntax, e.g. an “Object Control” structure such as: I ordered him [to gather my men to the hall] B1 him is both the object of ordered and the logical subject of gather here. This is a criterial construction for B1 and higher levels which distinguishes them from the A levels.

Structures like the following with finite or non-finite subordinate clauses and movement of the WH-word (how, where, etc) to the front of its clause are also first attested at B1: I don’t know [how I could have done it] B1 I did not know [where to look for it] B1

And postnominal modifiers in participial –ing become productive at B1: I received your mail [asking for the sales report] B1

Structures with a finite subordinate clause positioned to the right of predicates like is true and seems with a subject it are also criterial for B1 and higher levels: It’s true [that I don’t need a ring to make me remember you] B1 i.e. so-called “Extraposition” structures

A large number of new lexical verbs appear for the first time at B1 including: divide, fit, grab, spill, stick and tear And the meanings of the verbs that appeared first at A1 and A2 begin to expand from their basic senses.

break appears for the first time in the extended sense of INTERRUPT at B1: At last I managed to break the routine of the city … B1

Constructions that are criterial for B2 and higher levels include “secondary predications” go and paint the houses yellow and blue B2 with yellow and blue predicated of the direct object houses

“Extraposition” structures with a non-finite subordinate clause positioned to the right of its predicate are B2 It would be helpful [to work in your group as well] B2

And so-called “Pseudocleft” structures with an initial what functioning as subject of its verb: What fascinated me was [that I was able to lie on the sea surface] B2

“Subject-to-Subject Raising” constructions appear first at B2 with most of the higher verbs and adjectives that trigger this rule, for example prove: The car has proved [to be one of the most important inventions of our century] B2 Similar examples are found at B2 with other raising verbs and adjectives (The car happened to be …, The car appeared to be …, The car turned out to be …, The car is likely to be …, etc)

New lexical verbs at B2 include acquire, capture, drag, rush, spread,swallow and new meanings and uses are attested for the verbs that appeared earlier.

For break, first attested at A2, these include new collocations such as break a promise or break the law: For cut, also an A2 verb, they include new meanings at B2 such as REDUCE in cut the cost

The C Levels (C1 and C2) “Subject-to-Object Raising” constructions with the verb believe appear first at C1 and are criterial for C levels: I believe her [to be this country’s best representative] C1

Passivized Subject-to-Object Raising constructions such as the following with assumed are also criterial for C1: the low cost of membership and entry was assumed to be an advantage. C1

Sequences of two prenominal –s genitives are found at C1: in the bride’s family’s house C1 Structurally: in [[[the bride’s] family’s] house]

New lexical verbs appearing first at C1 include accumulate, boast, quote, reassure, shape and stain along with new meaning possibilities for the verbs already introduced. E.g. break appears first in the idiomatic sense of break the bank at C1.

New features appearing at C2 include less common Subject-to-Object Raising constructions with higher predicates such as presume, declare and remember: He presumed work [to be the way to live] C2

New lexical verbs at C2 include stagger, sway, limp, saunter, raid,squander New meanings for break at C2 include original figurative senses such as the attested break the wall that surrounds him.

Negative Linguistic Features One major distinguishing feature of the C levels can be seen in the low frequencies for “negative features” or errors. There appear to be significant improvements in ALL of the syntactic and morpho-syntactic error types at the C levels.

At the B levels, by contrast, improvements are relatively modest, and for many error types the scores actually get worse, especially at B2, before they get better again at C1.

The error codes defined and assigned by our CUP colleagues involve morpho- syntactic errors of inflection, derivation and grammatical form, and syntactic errors of omission, positioning and co-occurrence. It appears that learners at the C levels are increasingly mastering these rules of English, whereas B-level learners are not.

We must now ask: WHY we see these patterns in the data and why we see the criterial features changing the way they do at the different levels? In particular, WHAT is it about the features of the higher proficiency levels that makes them late acquired rather than early?

It cannot simply be that learners are imitating the words and constructions they are explicitly taught in their textbooks. First, because there are many different textbooks and teaching methods around the world. But secondly because learners learn more than they are explicitly taught, from their reading materials, papers, magazines, movies, TV, conversations, and so on.

I.e. second language learning shares many similarities with first language learning, but not all obviously.

For example, more frequently occurring words and constructions are learned before less frequent ones, and simpler words, constructions and meanings are learned before more complex ones, in both first and second language acquisition.

E.g. learning English nouns and verbs with high frequencies of use is easier than learning those with lower frequencies, because they are encountered more frequently (greater exposure); frequent lexical items are overrepresented at first in L2 English, moving gradually to L1 English norms ( see Hawkins & Buttery 2009, Hawkins & Filipović 2011)

More frequent construction types (“subcategorization” frames or “verb co- occurrences”) are acquired earlier than more infrequent ones, in general (see C. Williams 2007, Hawkins & Filipović 2011 ).

The constructions of English that are learned earliest are those that occur most frequently in the input, as reflected in e.g. the British National Corpus. This could be established by comparing the CLC with the British National Corpus (BNC), see Williams (2007).

In fact, the new constructions that are criterial for A2, B1 and B2, in Williams’ data, appear to be learned in direct proportion to their frequency in the input, as reflected in the BNC. The more exposure, the earlier the acquisition and the easier the learning. This is shown in Tables 1 and 2.

Table 1 Frequencies for Verb Co-occurrence Frames in English Corpora (including BNC) Average Token Frequencies in the BNC for the new Verb Co-occurrence Frames appearing at the learner levels A2B1B2/C1/C2 1,041,634 38,174 27,615

Table 2 Frequency Ranking Average Frequency Ranking in the BNC for the new Verb Co-occurrence Frames appearing at the learner levels A2 B1B2/C1/C2 8.238.6 55.6

These kinds of data enable us to set up the following principle of second language learning (for which there are also well-attested parallels in first language learning, see e.g. Tomasello 2003, Diessel 2004, MacWhinney 2005):

Maximize Frequently Occurring Properties (MaF) Properties of the L2 are learned in proportion to their frequency of occurrence (as measured, for example, in the BNC): more frequent exposure of a property to the learner facilitates its learning and reduces learning effort. I.e. more frequent properties will result in earlier L2 acquisition, more of the relevant properties learned, and fewer errors, in general. Infrequency makes learning more effortful, with precise predictions depending on other factors.

Another principle of second language learning (shared with first language learning) involves the relative simplicity or complexity of structures and meanings. The criterial grammatical features of earlier levels are, in general, simpler than those of later levels.

Also, in phonology simpler consonants and consonantal distinctions are acquired earlier than more complex ones ( see e.g. Eckman 1984 ). Simpler and more basic meanings for verbs are acquired earlier than more complex and derived extensions in meaning, figurative uses, etc.

The verb break in its basic physical sense at A2 break in the sense of INTERRUPT (break the routine) B1 break an agreement, promise, etc. B2 break the bank (idiomatic) C1 break the wall that surrounds him (original figurative) C2

Maximize Structurally and Semantically Simple Properties (MaS) Properties of the L2 are learned in proportion to their structural and semantic simplicity: simplicity means there are fewer properties to be learned and less learning effort is required. I.e. simpler properties will result in earlier L2 acquisition, more of the relevant properties learned, and fewer errors. Complexity makes learning more effortful, in general, since there are more properties to be learned, with precise predictions depending on other factors.

In second language learning we also see “transfer” effects from the first language, either positive (when the transfer results in a correct L2 property) or negative (when it results in an error). This is one thing that differentiates second from first language acquisition.

E.g. speakers of languages with definite and indefinite articles find it easier to acquire the article system of English than do speakers of languages without articles ( see Hawkins & Buttery 2009, 2010 )

Errors involving missing definite and indefinite articles in the L2 English of the CLC are consistently low when the L1s also have articles. Recall: MDI spoke to President (the) I have car (a)

Table 3 (next slide) shows missing determiner error rates for “the” and “a” at all proficiency levels for French, German and Spanish as first languages. All three languages have an article system. ( Data from Hawkins & Buttery 2009 ) The figures indicate the percentage of errors with respect to the total number of correct uses. For instance a percentage of 10.0% would indicate that a determiner was omitted 1 in every 10 times that it should have appeared. We see generally low error rates for these languages, without significant deviation between levels.

Table 3 Missing Determiner Error Rates for L1s with Articles Missing “the” A2 B1 B2 C1 C2 French4.76 4.675.01 3.11 2.13 German0.00 2.564.11 3.11 1.60 Spanish3.37 3.624.76 3.222.21 Missing “a” A2 B1B2 C1C2 French6.60 4.796.56 4.763.41 German0.89 2.903.83 3.62 2.02 Spanish4.52 4.287.91 5.16 3.58

Table 4 (next slide) shows missing determiner error rates for “the” and “a” at all levels for Turkish, Japanese, Korean and Russian as first languages. These languages do not have an article system. There is a general linear improvement, i.e. a decline, in error rates across the levels with increasing proficiency (shown from left to right).

Table 4 Missing Determiner Error Rates for L1s without Articles Missing “the” A2 B1B2 C1C2 Turkish22.06 20.7521.32 14.447.56 Japanese27.66 25.9118.72 13.809.32 Korean22.58 23.83 18.13 17.4810.38 Russian14.63 22.7318.45 14.62 9.57

Table 4 continued Missing “a” A2 B1B2 C1C2 Turkish24.29 27.6332.48 23.8911.86 Japanese35.09 34.8024.26 27.4115.56 Korean35.29 42.3330.65 32.5622.23 Russian21.71 30.1726.37 20.8212.69

One of the learning principles proposed in Hawkins & Filipović (2011) to account for these data is Maximize Positive Transfer:

Maximize Positive Transfer (MaPT) Properties of the L1 which are also present in the L2 are learned more easily and with less learning effort, and are readily transferred, on account of pre-existing knowledge in L1. Shared L1/L2 properties should result, in general, in earlier L2 acquisition, in more of the relevant properties being learned, and in fewer errors, unless these shared properties involve e.g. high complexity and are impacted by other factors. Dissimilar L1/L2 properties will be harder to learn by virtue of the additional learning that is required, again in general.

More generally, in ongoing work we provide a multi-factor model of learning comprising a set of interacting principles such as those illustrated. This model is a type of “complex adaptive system” ( see Gell-Mann 1992 ) and is called “CASP”, short for “complex adaptive system principles of SLA”.

These principles interact, sometimes reinforcing each other (e.g. early acquired frequent items are also often simple), sometimes competing to produce variable outputs and alternative interlanguages. Some of the principles are more general, others more specific. One of the more general principles is Minimize Learning Effort.

Minimize Learning Effort (MiL) ◦ Learners of a second language (L2) prefer to minimize learning effort when they learn the grammatical and lexical properties of the L2. Learning effort is minimized when shared properties of the L1 can be transferred directly into the L2 (MaPT), when properties of the L2 are frequently occurring in the L2 input (MaF), when structural and semantic properties of the L2 are simple rather than complex (MaS), and when there are fewer linguistic items to be learned in a given grammatical or lexical domain.

Once criterial features and transfer effects have been identified at the different proficiency levels they can be put to use for learning, teaching and assessment purposes, as illustrated in other talks in this session.

NB! Our criterial features are taken only from candidates who scored passing grades at each level. Hence learners who are studying for the relevant level can now be told explicitly what their successful peers have mastered, and teachers can incorporate these features in their teaching and materials, thereby optimizing the learners’ chances of success.

Theoretical interest of this work The criterial features we are extracting from the corpus are also of interest to theoreticians studying language acquisition. They provide a new set of empirical patterns that can be used to inform predictive and multi-factor theories of learning, such as CASP.

References Briscoe, E., J. Carroll and R. Watson (2006) ‘The second release of the RASP system’. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, Sydney, Australia. Council of Europe (2001) Common European Framework of Reference for Languages: Learning, teaching, assessment. CUP, Cambridge. Diessel, H. (2004) The Acquisition of Complex Sentences. CUP, Cambridge. Eckman, F.R. (1984) ‘Universals, typologies, and interlanguage’. In:W.E. Rutherford, ed., Language Universals and Second Language Acquisition, John Benjamins, Amsterdam, 79-105. Gell-Mann, M. (1992) ‘Complexity and complex adaptive systems’. In J.A. Hawkins & M. Gell-Mann, eds., The Evolution of Human Languages, Addison-Wesley, Redwood City, CA. Hawkins, J.A. (2004) Efficiency and Complexity in Grammars. OUP, Oxford. Hawkins, J.A. & P. Buttery (2009) ‘Using learner language from corpora to profile levels of proficiency: Insights from the English Profile Programme’. In L. Taylor & C.J. Weir, eds., Language Testing Matters, Proceedings of the 3 rd ALTE Conference 2008, CUP, Cambridge, 158-175.

Hawkins, J.A. & P. Buttery (2010) ‘Criterial features in learner corpora: Theory and illustrations’, English Profile Journal 1. Hawkins, J.A. & L. Filipović (2011) Criterial Features in L2 English: Specifying the Reference Levels of the Common European Framework. CUP, Cambridge. MacWhinney, B. (2005) ‘A unified model of language acquisition’. In J.F. Kroll & A.M.B. de Groot, eds., Handbook of Bilingualism: Psycholinguistic Approaches, OUP, Oxford. Milanovic, M. (2009) ‘Cambridge ESOL and the CEFR’, University of Cambridge ESOL Examinations Research Notes 37: 2-5. Tomasello, M. (2003) Constructing a Language: A Usage-based Theory of Language Acquisition, Harvard University Press, Cambridge, Mass. Williams, C.A.M. (2007) ‘A preliminary study into verbal subcategorisation frame usage in the CLC’, MS, RCEAL, University of Cambridge.

Acknowledgments The findings reported here would not have been possible without the assistance of many collaborators. Special thanks to: Co-author Paula Buttery (see Hawkins & Buttery 2009, 2010); Annette Capel of CUP for help with the wordlist searches; Ted Briscoe of the Cambridge Computer Lab and his colleagues for use of the RASP parser; Mike Milanovic & Nick Saville of Cambridge ESOL for theoretical and practical guidance and financial support (see below); Mike McCarthy of CUP and Penn State U for advice and input; Roger Hawkey of Cambridge ESOL for advice and English Profile Programme co-ordination; Lu Gram of the Computer Lab for help with error calculations and other searches; Caroline Williams of Cambridge University for verb subcategorisation data; and to CUP’s computational linguists who prepared "The Compleat|Complete Learner Corpus Document" 2006, from which the error codes and examples sentences in slide 18 are taken.

Financial Support The work reported here was made possible by generous financial support from Cambridge Assessment and from Cambridge University Press, within the context of the Cambridge English Profile Programme. This support is gratefully acknowledged here.

Identifying Criterial Features for Reference Level Descriptions John A. Hawkins (1,2) & Luna Filipović (1) (1) University of Cambridge, (2) University.

Similar presentations

Presentation on theme: "Identifying Criterial Features for Reference Level Descriptions John A. Hawkins (1,2) & Luna Filipović (1) (1) University of Cambridge, (2) University."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Identifying Criterial Features for Reference Level Descriptions John A. Hawkins (1,2) & Luna Filipović (1) (1) University of Cambridge, (2) University.

Similar presentations

Presentation on theme: "Identifying Criterial Features for Reference Level Descriptions John A. Hawkins (1,2) & Luna Filipović (1) (1) University of Cambridge, (2) University."— Presentation transcript:

Similar presentations

About project

Feedback