1. Standardisation of judgements: reading and writing Ülle Türk University of Tartu/Estonian Defence Forces ulle.turk@ut.ee/ylle.tyrk@mil.ee EUROPOS SĄJUNGA

2. Reading

3. 30-31 March 20073 Training with standardised samples for reading Phase 1: Illustration – a first assessment of the level of one text and its corresponding tasks and items –At what CEFR level can a test taker already answer the following item correctly? Phase 2: Controlled practice – different texts with their corresponding tasks and items are assessed under the guidance of the co-ordinator Phase 3: Individual assessment of the remaining items in chunks of 4-6 items The session continues until a spread of results of one and a half levels is achieved. EUROPOS SĄJUNGA

4. 30-31 March 20074 Benchmarking local samples of reading Individual rating followed by (pair) discussion leading to group consensus Discussion of spread across individual ratings and iteration until suitable agreement (spread equal to one and a half levels) is reached Ratings of the CEFR standardised samples should be analysed statistically in order to a)Confirm the relationship to the levels and b)Calculate intra-rater reliability (consistency) and inter- rater reliability EUROPOS SĄJUNGA

5. 30-31 March 20075 Task 4: Items 1–4 Text Characteristics: –Text source: magazine –Authenticity: authentic –Discourse type: mainly argumentative –Domain: public –Topic: other/politics –Nature of content: fairly extensive abstract content –Text length 350 words –Vocabulary: extended vocabulary –Grammar: wide range of complex structures –Text likely to be comprehensible by learner at CEF level: C1 EUROPOS SĄJUNGA

6. 30-31 March 20076 Items 1 & 2 Item 1 –Item type: Selected response/multiple choice –Operations: Infer implicit main idea/gist –Item level estimated: A2/B1 –Key: B –Per cent correct: 90.5% –Discrimination (item/total correlation):.31 Item 2 –Item type: Selected response/multiple choice –Operations: Make explicit inferences about the writer’s mood –Item level estimated: C2 –Key: B –Per cent correct: 36.1 % –Discrimination (item/total correlation):.45 EUROPOS SĄJUNGA

7. 30-31 March 20077 Items 3 & 4 Item 3 –Item type: Selected response/multiple choice –Operations: Make inferences about implicit main idea/gist –Item level estimated: C2 –Key: B –Per cent correct: 43.3 % –Discrimination (item/total correlation):.20 Item 4 –Item type: Selected response/multiple choice –Operations: Make inferences about implicit detail –Item level estimated: B1 –Key: C –Per cent correct: 83.6 % –Discrimination (item/total correlation):.43 EUROPOS SĄJUNGA

8. 30-31 March 20078 Task 5: Items 5–8 Text Characteristics: –Text source: magazine –Authenticity: authentic –Discourse type: mainly expository –Domain: public/personal –Topic: other/contemporary life –Nature of content: mainly abstract content –Text length 418 words –Vocabulary: extended vocabulary –Grammar: mainly complex structures –Text likely to be comprehensible by learner at CEF level: C1 EUROPOS SĄJUNGA

9. 30-31 March 20079 Items 5 & 6 Item 5 –Item type: Selected response/multiple choice –Operations: Infer implicit detail –Item level estimated: B2 –Key: B –Per cent correct: 65,9 % –Discrimination (item/total correlation):.52 Item 6 –Item type: Selected response/multiple choice –Operations: Evaluate implicit opinion –Item level estimated: C2 –Key: C –Per cent correct: 49,2 % –Discrimination (item/total correlation):.32 EUROPOS SĄJUNGA

10. 30-31 March 200710 Items 7 & 8 Item 7 –Item type: Selected response/multiple choice –Operations: Make inferences about implicit main idea/gist –Item level estimated: C1 –Key: C –Per cent correct: 52,1 % –Discrimination (item/total correlation):.36 Item 8 –Item type: Selected response/multiple choice –Operations: Evaluate implicit attitude/conclusion –Item level estimated: B2 –Key: A –Per cent correct: 74,4% –Discrimination (item/total correlation):.24 EUROPOS SĄJUNGA

11. Writing

12. 30-31 March 200712 Training with standardised samples for writing Phase 1: illustration with about three standardised performances – 60 minutes Phase 2: controlled practice with 3–5 standardised performances – 60 minutes Phase 3: individual assessment of 3–5 standardised performances – 60 minutes The session ends when the vast majority of participants agree on the level, with the spread not exceeding one and a half levels. EUROPOS SĄJUNGA

13. 30-31 March 200713 Benchmarking local samples of writing Individual rating and group discussion of high, middle and low performances – 60 minutes Individual rating of about five more performances – 60 minutes EUROPOS SĄJUNGA

14. 30-31 March 200714 Sample task 1 Your English Teacher has asked you to write a story. Your story must have the following title: The most important day of my life Write your story on your answer sheet. EUROPOS SĄJUNGA

15. 30-31 March 200715 Example answer 1 The most important day of my life. During a lifetime there is so many days you could call, the most important day of your life”. It could be the day you chose wich school you are going to, or what you want to work with the rest of your life. Another important day is when you get married, or you chose where to live. But most of all it must be a very important day when you give birth to a child. I think that changes everything you have been doing until then. Than you have to realice that somebody are more important than yourself. EUROPOS SĄJUNGA

16. 30-31 March 200716 Sample task 2 This is an extract from a letter which you receive from the World Opinion Organisation. We are carrying out an international survey on families around the world and the importance of family relationships. Please write us a report for our survey outlining the typical family situation in your own country and suggesting how you think the situation might change in the future. Write your report. Write approximately 250 words. EUROPOS SĄJUNGA

17. 30-31 March 200717 Sample task 3 You have read the extract below as part of a newspaper article on the loss of national and cultural identity. Readers were asked to send in their opinions. You decide to write a letter responding to the points raised and expressing your own views. ‘We are losing our national and cultural identity. Because of recent advances in technology and the easy availability and speed of air travel, different countries are communicating more often and are therefore becoming more and more alike. The same shopping malls and fast food outlets can be found almost everywhere. So can the same types of office blocks, motorways, TV programmes and even lifestyles. How can we maintain the traditions that make each nation unique?’ Write your letter. Do not write any postal addresses. EUROPOS SĄJUNGA

18. 30-31 March 200718 Sample task 3 You recently entered a competition and have just received this letter from the organiser. Read the letter, on which you have made some notes. Then, using all the information in your notes, write a suitable reply. Write a letter of between 120 and 180 words in an appropriate style on the opposite page. Do not write any postal addresses. EUROPOS SĄJUNGA

19. 30-31 March 200719 EUROPOS SĄJUNGA

20. 30-31 March 200720 Sources Manual for relating language examinations to the Common European Framework of Reference for Languages – a preliminary pilot version: http://www.coe.int/T/DG4/Portfolio/?L=E&M=/d ocuments_intro/Manual.html http://www.coe.int/T/DG4/Portfolio/?L=E&M=/d ocuments_intro/Manual.html The CEFR Grid for Writing, v. 2.: The Final Grid for analysing reading and listening items: www.ling.lancs.ac.uk/cefgridwww.ling.lancs.ac.uk/cefgrid EUROPOS SĄJUNGA