1. Manual for Relating Examinations to the Common European Framework Sauli Takala EALTA Confererence Slovenia, Granjska Gora, May 14-16, 2004

2. Overview Some background Why is the Manual needed? What is the approach adopted? What is/is NOT the Manual for? Procedures proposed in the Manual Current situation Future prospects and challenges

3. Some historical notes The Convention for Cultural Cooperation (CDCC), which provided the mandate for the Council of Europe´s action within the domain of cultural affairs was established 50 years ago Modern languages were included in the agenda about 45 years ago After a number of more ad hoc set of activities, action related to modern languages was organised into medium-term projects Work focused on exploring how learning objectives could be determined in a manner that reflected the social and learner needs The Threshold specification model of learning objectives is a good example of outcomes of this exploration and development work In 1991, at an intergovenmental symposium in Rüschlikon, Switzerland, it was agreed to launch a programme to develop a framework of reference for languages, which would provide a coherent conceptual tool for the various needs of language education. The Portfolio project was also launched. After some ten years of work, inlvolving broad consultation, the Common European Framework of Reference for Languages was adopted at an intergovernmental conference in Strasbourg In recent years both the Council of Europe and European Union have adopted the CEF as the basic tool for their work on language education

4. Current CoE Projects Common European Framwork of Reference of Languages (CEF) European Language Portfolio Manual for Relating Examinations to the CEF Language Policy analyses and reviews Other: CEF-sign language; Romani language curriculum; bank of descriptors; language-specific reference frameworks (German, French...)

5. What is the CEF – some points to consider CEF provides: a ‘vertical dimension’ and outlines an ascending series of common reference levels for describing learner proficiency (6-point scales) a ‘horizontal dimension’ - a set of descriptive categories map out made up of parameters of communicative activity and communicative language competence (not all are scaled)

6. CEF construct of language proficiency: the key to understanding the CEF (p. 9, English version) Language use, embracing language learning, comprises the actions performed by persons who as individuals and as social agents develop a range of competences, both general and in particular communicative language competences. They draw on the competences at their disposal in various contexts under various conditions and under various constraints to engage in language activities involving language processes to produce and/or receive texts in relation to themes in specific domains, activating those strategies which seem most appropriate for carrying out the tasks to be accomplished. The monitoring of these actions by the participants leads to the reinforcement or modification of their competences.

7. C2C2C2C2 Can understand with ease virtually everything heard or read. Can summarise information from different spoken and written sources, reconstructing arguments and accounts in a coherent presentation. Can express him/herself spontaneously, very fluently and precisely, differentiating finer shades of meaning even in more complex situations. C1C1C1C1 Can understand a wide range of demanding, longer texts, and recognise implicit meaning. Can express him/herself fluently and spontaneously without much obvious searching for expressions. Can use language flexibly and effectively for social, academic and professional purposes. Can produce clear, well-structured, detailed text on complex subjects, showing controlled use of organisational patterns, connectors and cohesive devices. B2B2B2B2 Can understand the main ideas of complex text on both concrete and abstract topics, including technical discussions in his/her field of specialisation. Can interact with a degree of fluency and spontaneity that makes regular interaction with native speakers quite possible without train for either party. Can produce clear, detailed text on a wide range of subjects and explain a viewpoint on a topical issue giving the advantages and disadvantages of various options. B1B1B1B1 Can understand the main points of clear standard input on familiar matters regularly encountered in work, school, leisure, etc. Can deal with most situations likely to arise whilst travelling in an area where the language is spoken. Can produce simple connected text on topics which are familiar or of personal interest. Can describe experiences and events, dreams, hopes and ambitions and briefly give reasons and explanations for opinions and plans. A2A2A2A2 Can understand sentences and frequently used expressions related to areas of most immediate relevance (e.g. very basic personal and family information, shopping, local geography, employment). Can communicate in simple and routine tasks requiring a simple and direct exchange of information on familiar and routine matters. Can describe in simple terms aspects of his/her background, immediate environment and matters in areas of immediate need. A1A1A1A1 Can understand and use familiar everyday expressions and very basic phrases aimed at the satisfaction of needs of a concrete type. Can introduce him/herself and others and can ask and answer questions about personal details such as where he/she lives, people he/she knows and things he/she has. Can interact in a simple way provided the other person talks slowly and clearly and is prepared to help.

8. Language learning potential progresses on two dimensions (Source: John de Jong,2004) Quantity Quality

9. Scale hierarchy: quantitative dimension

10. Frequently asked questions: How far do pupils/students progress in L2 in the school system (no PISA in L2; EU Barcelona indicator; European item bank) What is the currency value of national school reports/certificates nationally and internationally? How could it be improved? How can we relate our exams to the CEF? (need of guidance) How do I know that my B1 is your B1? (need of guidance) Need of guidance in relating exams to CEF in a credible manner led to the project of developing a Manual

11. Some history related to the Manual: Invitational Expert Seminar; Helsinki, summer 2002 Report: DG IV/EDU/LANG (2002) 15 Recommended development of a Manual as a collaborative endeavour building on current work in member states. –Describe the examination coverage, administration and analysis procedures; –Align results reported from the examination to the CEF common reference levels; –Provide supporting evidence that reports on the procedures followed to do so.

12. Who done it? Authoring Group/Phase 1: (aka ”Manual workers”) : Brian North, Neus Figueras, Piet Van Avermaet, Sauli Takala, Norman Verhelst Sounding Board of consultants: other Helsinki Seminar participants Ad hoc Project Group (new): Charles Alderson, John de Jong, John Trim... Participants can vary according to topic of discussion

13. Development Timeline July 2002:Helsinki seminar September 2002:Project approval November 2002:Authoring group set up Nov-Jan 2003:Consultation on Outline with Sounding board (consultants) Jan-May 2003:Authoring Group: revision of draft text May 19-20, 2003Meeting of Authoring Group and consultants to discuss version 0.5 (Strasbourg) Sep 2003:English & French drafts (DG IV/EDU/LANG (2003) 5 Sep 2003:Piloting Feedback invited (guide provided) April 15-16, 2004: Seminar on Piloting, Strasbourg; further action discussed

14. Aims of Manual contribute to competence building in the area of linking assessments to the CEF encourage increased transparency on the part of examination providers encourage the development of both formal and informal national and international networks of institutions and experts Thus: Manual is not just a technical guidance tool but a rich resource for thoughtful action

15. What the Manual NOT? Not a guide how to write good tests; presupposes that exam results are reliable and their interpretation is valid Does not demand any particular type of examination format nor exclude automatically some types of examination Being CEF-based, it cannot provide resources that are not in the current CEF-version; however, new descriptive categories can be added in the Specification job (->CEF-revision) Does not prescribe any one method of linking; other approaches are acceptable (cf. Reference Supplement) Does not present a mechanism which would provide accreditation of examinations; this requires a specific rationale of its own Is not a recipe; does nor provide rule-of-thumb shortcuts Does not guarantee success in linking examinations to the CEF; the linking activities must be of high standard

16. Overview of Linking Scheme Chapter 1: Introduction gives a very good overview (“read me first”) Familiarisation: thorough knowledge of the CEF “philosophy” and its resources (Chapters 2 & 3, pp. 13- 28) Specification: aims, CEF audit of coverage –Objectives, coverage, sections, weighting, text & item types, marking criteria, reporting (Chapter 4, pp. 29 – 63) Standardisation: understanding CEF levels –Training with calibrated examples, benchmarking local examples for standardisation training; standard setting (Chapter 5, pp. 65-98) Validation: Collection and analysis of data –Internal (test characteristics) External (relation to calibrated tests and descriptors –Actually ALL four activities are essential elements in building and substantiating an argument for a high quality work with linkage to the CEF –Chapter 6, pp. 99-122

17. What do we claim we are testing? (What is our construct?) We now need to check what out examination really looks like. (check the claim; awareness raising) If we wish to relate our examination to the CEFR, we need to profile our examination drawing on its descriptive scheme and scales. Preliminary Pilot Manual proposes a procedure to profile examinations vis-a-vis the CEFR: SPECIFI- CATION (Ch. 4, 29-63) We need to use the descriptive resources the CEFR provides as much as is possible, and supplement the scheme if it is not sufficient for profiling the exam

18. Specification General description of exam (Form A1): purpose, target population, domains etc (”this is our product”) Test development procedures, marking, reporting results More specific description in terms of communicative activities: reception, interaction, production, mediation More specific description in terms of aspects of language competences in: reception, production, mediation...

19. Specification sample C2 C1 B2 B1 A2 A1 LCSPInter- action RCWr

20. Specification sample C2 C1 B2 B1 A2 A1 PersonalPublicOccupationalEducational

21. How do our exam results relate to CEFR levels? How do I know what level our exam results correspond to? How do I know my B2 corresponds to your B2? We obviously need to standardise our procedures (in Europe)– STANDARDISATION (Ch. 5) We need to set standards, including cut-off scores. We need rigorous rater training: check intrarater consistency, interrater consistency We need to choose scales: CEF scales; validation of local scales We need benchmarks: European, ”local” We will choose to apply one of the several possible methods of standard setting; Manual suggests one approach – Reference Supplement, Section B, contains more information

22. External check of standard setting: verification What it the evidential basis of our linking? Internal validation External validation Chapter 6 in the Manual Reference Supplement

23. Link to CEF (Fellianka Kaftandjieva, Belgrade, 2003) ThroughanotherscaleThroughanothertestDirectStandardSetting

25. Possible outcome: European chart which shows how validated examination results are located on the Framework scales. Not only ”My B1 is your B1” but also ”My own exam´s grade 6 (or ”Good”) corresponds to B2”, grade 7 (”Very good”) is C1.1....

26. MatrGr 9VWO/RCVWO/ LC VWO/ SP Havo/ RC Havo/SP C1.2Laud C1.1Eximia B2.2Magna B2.1Cum l.. B1.2Lubent. B1.1Approb A2.2 Improb A2.1 A1.2 A1.1

27. What the Manual is about? A tool to promote competence to link examinations to CEF; a guide A tool for raising awareness about the particular examination (a tool for reflection) A tool with a certain approach outlined Represents one particular approach; other approaches may be equally valid (need to present rationale and evidence) Is mainly concerned with standard setting (not test development or construction) Requires thoughtful application Presupposes the provision of extensive and adequate empirical evidence to substantiate any claims

28. Contributions welcome: FEEDBACK on reading the Manual PILOTING on the basis of using the Manual CASE STUDY – writing up experience of using the Manual PERFORMANCE SAMPLES – to be collected for speaking + writing; LC and RC items These will contribute to a revised and extended Manual “kit” www.coe.int/lang First feedback Seminar: Strasbourg, April 15-16,2004

29. Final word Many claim linkage to the CEF but few produce evidence, and so far – arguably - no evidence is completely adequate (DIALANG probably the best exception) So: Quis custodiet ipsos custodes? Caveat emptor! When there are good reports of linkage, what happens to them next?