1. |Date 27.04.2010 faculty of arts applied linguistics language centre 1 ICT en Onderwijs in de Humaniora EMBED: Providing Online Teacher Training in the CEFR Nicole Schmidt & Kevin Haines Embed Project University of Groningen

2. |Date 27.04.2010 faculty of arts applied linguistics language centre Overview ›Context: internationalization and growth ›Using the CEFR in practice: maximising the use of available resources ›EMBED Project: design and procedures ›EMBED Project: emerging results ›Conclusions 2

3. |Date 27.04.2010 faculty of arts applied linguistics language centre Context: internationalization and growth in Higher Education in the Netherlands “European Education Ministers have established a goal for 2020 that 20% of graduates should have been ‘mobile’ during their studies compared to the current figure of 12%.” (Bal 2009) “The majority of the Master’s and PhD programmes are taught in English. In addition to the essential degree programmes in Dutch, all faculties have a representative selection of English-taught Bachelor’s degree programmes on offer. University of Groningen: Strategic Plan 2010 – 2015 3

4. |Date 27.04.2010 faculty of arts applied linguistics language centre Context: internationalization and growth in Higher Education in the Netherlands Dutch students generally enter university with at least a B2 level of English (Fasoglio et al., 2007) Non-Dutch students are required to have an IELTS score of 6.5 with no individual element below 6.0. “Passing from B2 to the C level should enable the learner to access higher education, professional fields of employment and the literary culture associated with a language.” Green 2008 (see English Profile Project) 4

5. |Date 27.04.2010 faculty of arts applied linguistics language centre CEFR: advantages in practice ›a single common structure and ‘vocabulary’; ›developed empirically by groups of experts in the language testing field; ›tools for familiarization, also developed by experts through projects such as: ›-CEFTrain:http://www.helsinki.fi/project/ceftrain/http://www.helsinki.fi/project/ceftrain/ -Webcef:www.webcef.euwww.webcef.eu -SLO:www.erk.nlwww.erk.nl ›A Manual (2009) which guides us towards good practice when applying the CEFR to tasks embedded in our curriculum.A Manual (2009) 5

6. |Date 27.04.2010 faculty of arts applied linguistics language centre CEFR: limitations in practice ›CEFR levels are described in general terms, leaving much room for interpretation by individual assessors; ›CEFR levels are very broad i.e. there is a considerable difference between what we may consider a ‘high C1’ and a ‘low C1’; ›limited number of samples relevant to the level of our students (generally B2/C1) and to academic domain(s). 6

7. |Date 27.04.2010 faculty of arts applied linguistics language centre CEFR: limitations in local practice ›Teachers come from a wide range of backgrounds, with a variety of grading or assessment cultures; ›Complex teaching schedules and turnover of teachers result in inconsistencies in CEFR familiarization and standardization within the teaching team. 7

8. |Date 27.04.2010 faculty of arts applied linguistics language centre Aim: EMBED Project: Using the CEFR for EAP Focus on academic writing (essays/reports) Aim = to lower the threshold for staff ‘training’ (consistency/transparency)  Familiarization  Explicit task specification  Local samples (benchmarked)  Standardization (in local context) (A Manual 2009) Spin-off = student awareness of course goals, assessment criteria and learning goals 8

9. |Date 27.04.2010 faculty of arts applied linguistics language centre Aim: Familiarization and Standardization Familiarization (CEFTrain): ›Introductory activities with scales ›Training area (tasks and samples) Standardization (Blackboard): ›Use of procedures described in A Manual ›Delivery through user-friendly interface blended with training workshops 9

10. |Date 27.04.2010 faculty of arts applied linguistics language centre Procedures in validation of samples ›Screening/filtering: an initial bundle of 120 essay samples from the Law course (project group). ›Local panel: 5 experienced English teacher/assessors, using procedures embedded in BlackboardBlackboard ›Consultant: CEFR assessment expert supporting interpretations of local panel ›Trial standardization: four Law essay samples (samples 2, 3, 5 and 6), each with a relatively flat profile ›External panel: assessors at universities from our network of partner universities in Belgium (Leuven), Finland (Helsinki) & Italy (Padua). 10

11. |Date 27.04.2010 faculty of arts applied linguistics language centre Results of sampling ›Procedure ›CEFTrain and Blackboard ›Online teacher training tools + workshops 11 Sample collection  sample validation Data  sample selection

12. |Date 27.04.2010 faculty of arts applied linguistics language centre CEFR Ratings + Justification ›Law external panel C2 coherence comment  “Transparent and professional organisation of text: intro ends with a clear opinion explicitly addressing the question featured in the title of the task.” ›Law external panel B1 accuracy comment  “ Well, well, well, what to say, or more correctly what not to say. Plenty of errors in word form and grammatical, some word order problems. (Are you trying to fool us - is this a native speaker?)” 12

13. |Date 27.04.2010 faculty of arts applied linguistics language centre Biggest challenge: Finding “Flat”samples The first illustrative examples used in the initial stage of standardisations “should show a performance with a relatively ‘flat profile’ across the categories of Table B4 …” (A Manual 2009: 45). 13 Flat Sample B2 Accuracy B2 Range B2Coherence B2 Argument & Description B2

14. |Date 27.04.2010 faculty of arts applied linguistics language centre Does a “flat” sample exist? 14 Figure 1 Assessment results for validation procedure of law samples C2 Coherence: “Transparent and professional organisation of text.” C1 Range: “Some very good expressions at times, but other times makes some rather basic mistakes.” Law external panel comments Panelists (dis)agreement Individual assessor

15. |Date 27.04.2010 faculty of arts applied linguistics language centre Why is the “flat”sample so difficult to locate? ›A variety of CEFR levels represented in a single essay  Multidimensional nature of proficiency  Within-learner inconsistency (“instable system”) “Actually, my appreciation lies between B2 and C1” (Belgian external expert on Sample 3) ›(Dis)agreement between assessors 15

16. |Date 27.04.2010 faculty of arts applied linguistics language centre Degree of (dis)agreement among panelists Law Sample 3: External panel ›C2 Range comment:  “The candidate shows great flexibility in formulating ideas in differing linguistic forms to convey finer shades of meaning precisely, to give emphasis and to eliminate ambiguity...” ›C1 Range comment:  “Slightly limited lexical flexibility is shown in the word choice/ collocational combinations … occasionally repetitive word choice…” 16

17. |Date 27.04.2010 faculty of arts applied linguistics language centre EMBED Project: Results 17 Sample 6RCAArgumentOverall Internal assessors x 5 B2.2 B2.1B2.2B2 Local consultant B2.2 B2.1B2.2B2 External Finland 1 B2 B1B2.2B2 External Finland 2 B2 B1B2.2B2 External Italy B2B1 B2.2B2 External Belgium B2 B1B2.1B1

18. |Date 27.04.2010 faculty of arts applied linguistics language centre EMBED Project: Results 18 Sample 6RCAArgumentOverall Internal assessors x 5 B2.2 B2.1B2.2B2 Local consultant B2.2 B2.1B2.2B2 External Finland 1 B2 B1B2.2B2 External Finland 2 B2 B1B2.2B2 External Italy B2B1 B2.2B2 External Belgium B2 B1B2.1B1

19. |Date 27.04.2010 faculty of arts applied linguistics language centre EMBED Project: Results 19 Sample 6RCAArgumentOverall Internal assessors x 5 B2.2 B2.1B2.2B2 Local consultant B2.2 B2.1B2.2B2 External Finland 1 B2 B1B2.2B2 External Finland 2 B2 B1B2.2B2 External Italy B2B1 B2.2B2 External Belgium B2 B1B2.1B1

20. |Date 27.04.2010 faculty of arts applied linguistics language centre EMBED Project: Results 20 Sample 6RCAArgumentOverall Internal assessors x 5 B2.2 B2.1B2.2B2 Local consultant B2.2 B2.1B2.2B2 External Finland 1 B2 B1B2.2B2 External Finland 2 B2 B1B2.2B2 External Italy B2B1 B2.2B2 External Belgium B2 B1B2.1B1

21. |Date 27.04.2010 faculty of arts applied linguistics language centre EMBED Project: Results 21 Sample 6RCAArgumentOverall Internal assessors x 5 B2.2 B2.1B2.2B2 Local consultant B2.2 B2.1B2.2B2 External Finland 1 B2 B1B2.2B2 External Finland 2 B2 B1B2.2B2 External Italy B2B1 B2.2B2 External Belgium B2 B1B2.1B1

22. |Date 27.04.2010 faculty of arts applied linguistics language centre EMBED Project: Results 22 Sample 6RCAArgumentOverall Internal assessors x 5 B2.2 B2.1B2.2B2 Local consultant B2.2 B2.1B2.2B2 External Finland 1 B2 B1B2.2B2 External Finland 2 B2 B1B2.2B2 External Italy B2B1 B2.2B2 External Belgium B2 B1B2.1B1

23. |Date 27.04.2010 faculty of arts applied linguistics language centre EMBED project: Results

24. |Date 27.04.2010 faculty of arts applied linguistics language centre EMBED project: Results

25. |Date 27.04.2010 faculty of arts applied linguistics language centre EMBED project: Results Words not always used appropriately shows some good phrasing which supports the logical development of the text

26. |Date 27.04.2010 faculty of arts applied linguistics language centre EMBED project: Results Occasional problems with word order Problems with punctuation in combination with conjunctions

27. |Date 27.04.2010 faculty of arts applied linguistics language centre ModelModel for training module 27 Engels schrijven voorbeeld 3 - Europees Referentiekader

28. |Date 27.04.2010 faculty of arts applied linguistics language centre Interactive training website features Strengths of model ›compact grid, all descriptors available ›flexible adjustment of descriptive categories ›adaptable for other languages and skills ›comprehensive expert feedback, linked to user's evaluation in supplementary grid Our alterations ›Use of original CEFR descriptors & task specs per A Manual ›Relevance to our own (longer) samples ›Referenced feedback (direct link from text to expert feedback)

29. |Date 27.04.2010 faculty of arts applied linguistics language centre Challenges -Inconsistency in profiles: non-flat profiles are the norm in the productive work of learners; it is evidence of their learning -Inconsistency in assessors/raters; which is why we need standardization -Inconsistency within texts; this is most evident in longer work at higher language levels

30. |Date 27.04.2010 faculty of arts applied linguistics language centre B1 C1 100wrds 200wrds300wrds400wrds © Haines, 2010

31. |Date 27.04.2010 faculty of arts applied linguistics language centre Remaining challenges: -To define representative extracts within samples (as part of longer text) -To make the right choices in terms of ICT in order to deliver the information clearly to our target group (without raising the threshold) -To create a healthy blend between online and face-to-face training provision (familiarization and standardization)

32. |Date 27.04.2010 faculty of arts applied linguistics Thank you for your attention Are there any questions? With thanks to our colleagues on the Embed Project team: Sake Jager, Petra Jansma, Wander Lowie & Estelle Meima Please feel free to contact us: Kevin Hainesk.b.j.haines@rug.nl Nicole Schmidtn.m.schmidt@rug.nl 32