Presentation is loading. Please wait.

Presentation is loading. Please wait.

Assessment, Task Design and Related Issues Salzburg, 24th February 2010 ao.Univ.Prof. Mag. Dr. Guenther Sigott Assessment, Task Design and Related Issues.

Similar presentations


Presentation on theme: "Assessment, Task Design and Related Issues Salzburg, 24th February 2010 ao.Univ.Prof. Mag. Dr. Guenther Sigott Assessment, Task Design and Related Issues."— Presentation transcript:

1 Assessment, Task Design and Related Issues Salzburg, 24th February 2010 ao.Univ.Prof. Mag. Dr. Guenther Sigott Assessment, Task Design and Related Issues Salzburg, 24th February 2010 ao.Univ.Prof. Mag. Dr. Guenther Sigott Language Testing Centre Alpen-Adria-Universität Klagenfurt http://www.uni-klu.ac.at/ltc http://www.uni-klu.ac.at/ltc

2 assess v.t. to assess sth. – what? to assess sth. – what? to assess sb. – in terms of what? to assess sb. – in terms of what? Assessment is impossible if we are unclear about the WHAT.

3 teach v.t. to teach sth. – what? to teach sth. – what? to teach sb. sth. – what? to teach sb. sth. – what? Teaching is impossible if we are unclear about the WHAT.

4 Why bother about the WHAT? Important to understand what we want to teach and what pupils are supposed to learn (curricular objectives). Important to understand what we want to test (test constructs).

5 BHS Matura Is it clear what the exam should test? Is it clear what the exam should test in the different school types? Does the exam in fact test what it should test? Does the exam test the same things in all schools of the same type so that the results are comparable? Does the exam test the same things every year so that results are comparable? Does the exam relate to internationally accepted descriptions of educational standards such as the CEFR?

6 Q1: Is it clear what the exam should test? WHAT: curricular objectives / test construct - Models of language competence - Curriculum - CEF HOW: test method

7 Origins of the WHAT

8 L.F. Bachman, Fundamental Considerations in Language Testing, OUP 1990, p. 87.

9 E8 Reading Strategies (extract) Careful Reading Understanding the Text - understanding explicitly stated main idea(s) and / or distinguishing that from supporting details - locating, identifying, understanding and comparing facts, opinions, definitions (this includes search reading strategies) - understanding the logical organisation of the text, e.g. understanding relationships among ideas in a text (problem – solution, cause – effect, temporal sequence, etc.) - understanding cohesive relationships (reference, ellipsis, substitution, conjunction, lexical cohesion) - making propositional inferences (deducing information that is not explicitly stated from information that is explicitly stated) Adapted from Urquhart & Weir 1998

10 CEF Overall Written Production B2 Can write clear, detailed texts on a variety of subjects related to his / her field of interest, synthesising and evaluating information and arguments from a number of sources. C1 Can write clear, well-structured texts of complex subjects, underlining the relevant salient issues, expanding and supporting points of view at some length with subsidiary points, reasons and relevant examples, and rounding off with an appropriate conclusion.

11 HOW? What do we do in order to make our students display their competences?

12 Item 1 You are asking your friend a lot of questions about him and his family. Choose the right answer for each of the questions (there is only one correct answer per question!) and fill in the empty boxes with the correct number.

13 Item01 Test format: Grammar, Matching activity (2 nd grade AHS) You are asking your friend a lot of questions about him and his family. Choose the right answer for each of the questions (there is only one correct answer per question!) and fill in the empty boxes with the correct number. 1 Can Tom drive a car? Yes, they have. 2 Is your brother good at swimming? No, she isn t. 3 Are you listening to the radio? Yes, I have. 4 Have you got a cousin? Yes, it is. 5 Can you come to our barbecue? No, he isn t. 6 Does your sister go to nursery school? No, they can t. 7 Has your aunt got a husband? Yes, we are. 8 Do your parents often go out in the evenings? No, I don t. Yes, we do. No, they haven t. No, we can t. Yes, he can. No, they don t. Yes, she can. Yes, she has. No, she doesn t.

14 Item 1 Tick the answer which is grammatically correct: Is your brother good at swimming? Yes, he can. Yes, you are. Yes, he does. No, he isnt.

15 You will have 10 minutes to do this task. Read the instructions carefully and then write your text on the separate sheet. Your short letter or e-mail should be 40 to 60 words long. Your American aunt has sent you a birthday present. Write a short letter or an e-mail to say thank you. Tell your aunt you really liked your present and why it is useful for you. You can also ask her if you could visit her in your summer vacation. You could start your letter like this: Test Method in Writing and Speaking Resist the temptation: keep method distinct from construct Dear Aunt Lizzy:.................................................. If you rate the resulting text for Grammar, what is construct, what method?

16 Test Method in Writing and Speaking Resist the temptation: keep method distinct from construct Du hast für diese Aufgabe 10 Minuten Zeit. Lies die Anleitung gut durch und schreib Deinen Text auf das beiliegende Blatt. Dein kurzer Brief oder Deine Email soll 40 bis 60 Wörter lang sein. Deine Tante in Amerika hat Dir ein Geburtstagsgeschenk geschickt. Schreib einen kurzen Brief oder eine Email, um dich zu bedanken. Sag Deiner Tante, dass Dir das Geschenk sehr gut gefällt und warum Du es gut gebrauchen kannst. Du kannst sie auch fragen, ob Du sie in den Sommerferien vielleicht besuchen kannst. Du könntest Deinen Brief so beginnen: Dear Aunt Lizzy:.................................................. If you rate the resulting text for Grammar, what is construct, what method?

17 Test Method ctd. If you rate the resulting text for Vocabulary, what is construct, what method? If you rate the resulting text for Cohesion and Coherence, what is construct, what method?

18 If you rate the resulting text for Vocabulary, what is construct, what method? If you rate the resulting text for Cohesion and Coherence, what is construct, what method? Test Method in Writing and Speaking Resist the temptation: keep method distinct from construct Du warst vor ein paar Tagen bei der Geburtstagsparty Deines Freundes/Deiner Freundin. Schreib ihm/ihr ein Email, um ihm/ihr zu sagen, dass Dir die Party gefallen hat. Du warst vor ein paar Tagen bei der Geburtstagsparty Deines Freundes/Deiner Freundin. Schreib ihm/ihr ein Email, um ihm/ihr zu sagen, dass Dir die Party gefallen hat. Sag ihm/ihr, warum Dir die Party gefallen hat. Sag ihm/ihr, warum Dir die Party gefallen hat. Sag ihm/ihr, was Dir am besten gefallen hat. Sag ihm/ihr, was Dir am besten gefallen hat. Frag ihn/sie, wann ihr einander wiedersehen könnt. Frag ihn/sie, wann ihr einander wiedersehen könnt. Mach einen Vorschlag für das kommende Wochenende. Mach einen Vorschlag für das kommende Wochenende.

19 So? To state what the exam should test, we must be able to say what is the construct and what the method. We want to test construct, not method.

20 Q2: Is it clear what the exam should test in the different school types? Once we are clear about the distinction between construct and method, we are in a good position to discuss this question. Which areas of the construct are relevant for all school types, which only for particular school types?

21 Q3: Does the exam in fact test what it should test? Construct + method will yield an exam which we believe tests what we are interested in. It is our obligation to demonstrate that this exam indeed tests what we claim it tests. Are we measuring anything at all or are we just ascribing figures to performances on a random basis? – reliability - Scoring keys must be available. - Rating scales have to be put in place. - Raters need to be trained in using the rating scales. - It should not matter who conducts the exam. - It should not matter who rates / scores it. Are we measuring the things we set out to measure? Accompanying research is needed: - Content validation - Construct validation

22 Q4: Does the exam test the same things in all schools of the same type so that the results are comparable? Use the same exam in all the schools of the same type at the same time. Develop parallel exams which test the same things and which are equally difficult.

23 Q5: Does the exam test the same things every year so that the results are comparable? Use the same exam every year (a theoretical possibility). Develop parallel exams which test the same things and which are equally difficult.

24 Parallel forms are needed. This requires identity of construct plus constancy / comparability of difficulty.Parallel forms are needed. This requires identity of construct plus constancy / comparability of difficulty. This is related to setting cutoff scores for pass / fail decisions.This is related to setting cutoff scores for pass / fail decisions. This leads to the issue of how these cutoff scores relate to the CEF (B2?)This leads to the issue of how these cutoff scores relate to the CEF (B2?) Moreover, there is a need to determine cutoff scores for the individual grades on the national 5- point scale (multiple cutoffs).Moreover, there is a need to determine cutoff scores for the individual grades on the national 5- point scale (multiple cutoffs). This calls for standard-setting procedures to be put in place.This calls for standard-setting procedures to be put in place. Q6: Does the exam relate to internationally used descriptions of educational standards such as the CEF?

25 Standard Setting Problem: item-banking not practical in the Austrian Matura context (item security). Consequence: standard setting required for each parallel form. Needed: descriptions of minimally acceptable competences for Matura (4-3- 2-1). CEF comes in handy but is not sufficient.

26 Bottom Line For all of these things to work, we need items and tasks based on a clearly defined construct with a clearly defined test method.

27 Standard setting: Key Concepts Prerequisites: reliability and validity Prerequisites: reliability and validity Criterion-referenced approach to testing Criterion-referenced approach to testing Performance level labels (e.g. school grades – sehr gut, gut … genügend) Performance level labels (e.g. school grades – sehr gut, gut … genügend) Performance level descriptions (PLDs) (e.g. CEF level descriptors) Performance level descriptions (PLDs) (e.g. CEF level descriptors) Target examinee: e.g. minimally competent examinee at a particular PLD level Target examinee: e.g. minimally competent examinee at a particular PLD level

28 Angoff Procedure (Angoff 1971) Key concept: minimally competent examinee Key concept: minimally competent examinee Dichotomous estimates: Will the minimally competent person answer this item correctly? – Sum of items coded 1 is the cut score. Dichotomous estimates: Will the minimally competent person answer this item correctly? – Sum of items coded 1 is the cut score. Probability-based estimates: State the probability that the minimal competent person will answer this item correctly. Probability-based estimates: State the probability that the minimal competent person will answer this item correctly. Extended Angoff approach: How many score points will the minimally competent candidate achieve on this item? Extended Angoff approach: How many score points will the minimally competent candidate achieve on this item?

29 Item vs Task A cline which is defined by length of response.A cline which is defined by length of response. Tick: item (closed format)Tick: item (closed format) Long open-ended response: taskLong open-ended response: task Receptive skills lend themselves to assessment by closed item formats; productive skills to assessment by open- ended formats.Receptive skills lend themselves to assessment by closed item formats; productive skills to assessment by open- ended formats.

30 Thank you!


Download ppt "Assessment, Task Design and Related Issues Salzburg, 24th February 2010 ao.Univ.Prof. Mag. Dr. Guenther Sigott Assessment, Task Design and Related Issues."

Similar presentations


Ads by Google