Developing CEF-related proficiency scales for the Escoles Oficials d’Idiomes Neus Figueras

The EOI system in Catalonia ●State funded language schools (+16) ●There are two levels defined Certificat de Cicle Elemental (B1 ?) Certificat d’Aptitud (B2 ?) ●13 different languages ●Standardised language proficiency certificates in place since 1995 The exams are ●developed centrally ●related to published curricula ●written according to set specifications ●submitted to pre-testing ●standardised administration and marking ●analysed statistically

Issues to be solved ●Revise EOI certificate tests after 10 years. ●Develop an item banked system. ●Link levels to Common European Framework levels (Manual). ●Is examination difficulty equivalent across time? ●Is the lower certificate consistently easier than the higher certificate? ●Are the different certificates in the different languages comparable?

Exam revisionLinking EOI & CEF levels Teaching/learning context To sum up

The EOI project Outline ●July 2003 – 2004 ●Define objectives ●Project design ●Empirical scale development ●Scale validation ●Item banking (English) starts ●July 2004 –2005 ●Publication of scales ●Item banking (English) continues ●Linkage to CEF: Procedures in the Manual (specification) ●July 2005 – 2006 ●Confirmation of linkage: ●standardisation ●empirical validation ●comparability across languages

And..... CEF adoption or adaptation? “A scale, like a test, has validity in relation to contexts in which it has been shown to work” (2001:22)

Scale Development Step 1: selecting level descriptors (per skill) ●CEF ●Curriculum objectives for EOI ●Finnish scales (YKI) Step 2: translation of the descriptors into Catalan Step 3: mapping descriptors onto levels (CEF and EOI) ●Experts assign levels to descriptors ●Analysis of sorting exercises ●Collating resulting “surviving” descriptors Step 4: developing new scales ●Drafting ●Validation ●Scale revision ●Validation ●Final version

Descriptors Teachers* Languages ●Reading ●Listening ●Speaking ●Writing ●Grammar ●Vocabulary * Arabic, Basque, Catalan, Dutch, English (>40), French (>20), German (>15), Italian, Japanese, Portuguese, Russian, Spanish Sorting tasks

● Listening: ● Reliability ● B1? Reading N=99 Listening N=90 Speaking N=85 Writing N=86 Vocabulary N=65 Grammar N=72 α Analysis of sorting exercises (N=number of judges/teachers)

Comparing language groups: Reading 1-English; 2-French; 3-German; 4-Mixed

English German French Can understand clearly written, straightforward instructions for a piece of equipment. Language groups: an outlier

● Listening: ● Reliability ● B1? RNDescriptors A1: 1, 12, 22, 24, 30 A2: 13 C2: A2: 2, 23, 28, 36, 39 B1: 16, 25 B2: 11, 32 C1: 6, 19, 27, 29, 33, 38, 40 C2: 7, 8, 17, 20, A1: 15 B1: 3, 21, 34, 37 B2: 4, 9, 18 C1: 10, 35 41C1: 5 51B1: 14 Reading: range

Reading: range - examples n.23. Can find specific information in simple everyday texts. A1-14 A2-82 B1-7 n.14. Can use the context, linguistic and non linguistic, as a means to increase comprehension. A1-9 A2-17 B1-28 B2-13 C1-11 C2-8

Reading: descriptor´s analysis Median Range90 Score R14 Can use the context, both linguistic and non-linguistic, to increase comprehension 35-,15 X R15 Can understand bills, receipts, programmes and lists with basic information 23-1,06 X R16 Can get the gist of lengthy texts on familiar subjects, but unfamiliar topics may cause difficulties 32-,45 R17 Can read virtually all kinds of written language without problems 621,31 R18 Can read complex specialised texts in his/her field 43,58 X R19 Can extract facts and opinions from complex specialised texts 521,20 R20 Can identify the affective and evaluative content of complex texts, even if they include technical vocabulary 621,42 R21 Can identify the main conclusions in clearly signalled argumenative texts 43,14 X R22 Can understand words and familiar names in advertisements, posters and brochures 11-1,61 R23 Can find specific information in simple everyday texts 22-1,06

Drafting the scales ● Using descriptors preferably with range 0-2 ● Following guidelines Appendix A CEF ● Positiveness ● Definiteness ● Clarity ● Brevity ● Independence

● Regular progression from band to band ● Overall coherence and consistency ● Conditions ● Limitations Checking for

●Because the whole is different from its parts. ●To guarantee ●the consecutive order of band descriptors ●the degree of separation between them. ●To establish the link between EOI scales and CEF scales How ? ●Pair comparison exercise: comparing two scales (CEF and EOI), 66 pairs of descriptors per skill. “Read carefully each pair of descriptors and choose the one that describes a higher level of language proficiency”. Validating the scales

How EOI teachers perceive CEF scales

Which is B2 and which is C1? (Spoken interaction) ●Can express him/herself fluently and spontaneously, almost effortlessly. Has a good command of a broad lexical repertoire allowing gaps to be readily overcome with circumlocutions. There is little obvious searching for expressions or avoidance strategies; only a conceptually difficult subject can hinder a natural, smooth flow of language. ●Can use the language fluently, accurately and effectively on a wide range of general, academic, vocational or leisure topics, marking clearly the relationships between ideas. Can communicate spontaneously with good grammatical control without much sign of having to restrict what he/she wants to say, adopting a level of formality appropriate to the circumstances. Can interact with a degree of fluency and spontaneity that makes regular interaction, and sustained relationships with native speakers quite possible without imposing strain on either party. Can highlight the personal significance of events and experiences, account for and sustain views clearly by providing relevant explanations and arguments.

●C1: Can express him/herself fluently and spontaneously, almost effortlessly. Has a good command of a broad lexical repertoire allowing gaps to be readily overcome with circumlocutions. There is little obvious searching for expressions or avoidance strategies; only a conceptually difficult subject can hinder a natural, smooth flow of language. ●B2: Can use the language fluently, accurately and effectively on a wide range of general, academic, vocational or leisure topics, marking clearly the relationships between ideas. Can communicate spontaneously with good grammatical control without much sign of having to restrict what he/she wants to say, adopting a level of formality appropriate to the circumstances. Can interact with a degree of fluency and spontaneity that makes regular interaction, and sustained relationships with native speakers quite possible without imposing strain on either party. Can highlight the personal significance of events and experiences, account for and sustain views clearly by providing relevant explanations and arguments.

How EOI teachers perceive EOI scales

C1 Can understand long complex texts and contemporary literary prose, being able to appreciate subtle differences regarding meaning, register and tone. Nuances conveyed by rare expressions may cause problems in comprehension. C2 Can understand with ease virtually all forms of the written language, although highly idiomatic expressions may still cause some difficulty. C1 Can understand with reasonable ease long and complex texts of different types, including contemporary literary prose, and appreciate subtle differences regarding meaning, register and tone. Highly idiomatic expressions still cause some problems in comprehension. C2 Can understand with ease virtually all forms of the written language, even complex literary texts, identifying the affective and evaluative content and interpreting it critically. Only occasionally may nuances conveyed by some rare or highly idiomatic expressions cause some difficulty. Where is the problem? (EOI Reading)

Linking CEF and EOI scales

CEF - C1 Has a good command of a broad lexical repertoire allowing gaps to be readily overcome with circumlocutions; little obvious searching for expressions or avoidance strategies. Good command of idiomatic expressions and colloquialisms. EOI – C1 Can understand and use a broad lexical repertoire, including synonyms and antonyms of less common words, idiomatic expressions, to express himself/herself correctly and appropriately on a wide range of general, academic, professional and leisure topics. Can understand word play based on multiple meanings and use prefixes and suffixes to form less common words. Less frequent words may cause some effort in comprehension. Comparing EOI and CEF levels (Voc)

Linking CEF and EOI scales

EOI – B1 Can understand and use a sufficient vocabulary to express himself/herself with a good command on topics related to his/her everyday life and to common concrete situations and needs. Lexical gaps may cause hesitation or make circumlocution necessary. Errors may occur when expressing more complex thoughts or when handling new or less common topics or situations. CEF - B1 Has a sufficient vocabulary to express him/herself with some circumlocutions on most topics pertinent to his/her everyday life such as family, hobbies and interests, work, travel and current events. EOI - A2 Can understand and use a basic lexical repertoire, including antonyms and synonyms of most frequent words. Can infer the meaning of some words outside his limited lexical repertoire. Vocabulary errors or shortage of words may cause interruptions and misunderstandings. Comparing EOI and CEF levels (Voc)

Lessons learnt…implications ●The CEF as a catalyst ●Reading is not understanding ●Continued training is necessary ●Appearances need to be checked empirically ●Exact correspondence may not be possible, but is it desirable? ●Linkage does not mean equivalence

Work continues…… The EOI scales as a point of reference for ●curriculum objectives ● reporting learner achievement ● reporting exam levels To meet the need for ●Improving test quality ●Item banking ●Setting standards ●Validating established standards

References EOI scales and a full description of their development can be found at ●Council of Europe (2001) Common European Framework of Reference for Languages: Learning, teaching, assessment. Cambridge. CUP. (available on line at: operation/education/Languages/Language_Policy ) operation/education/Languages/Language_Policy ●Council of Europe (2003) Manual for relating language examinations to the CEF. A preliminary pilot version. Strasbourg. Language Policy Division. DGIV/EDU/LANG ●Kaftandjieva, F. and S. Takala (2002) Council of Europe Scales of Language Proficiency: A Validation Study. In Alderson, J.C.(ed), Common European Framework of Referencefor Languages: Learning, Teaching, Assessment. Case Studies, Strasbourg: Council of Europe Publishing. ●North, B. and Schneider, G. (1998) Scaling descriptors for language proficiency scales. Language Testing 15, 2: