Curricular Objectives – Test Constructs – Test Items – Test Tasks … ao.Univ.Prof. Mag. Dr. Guenther Sigott Curricular Objectives – Test Constructs – Test Items – Test Tasks … ao.Univ.Prof. Mag. Dr. Guenther Sigott Language Testing Centre Alpen-Adria-Universität Klagenfurt

Why bother? Important to understand what we want to teach and what pupils are supposed to learn (curricular objectives). Important to understand what we want to test (test constructs).

BHS Matura Is it clear what the exam should test? Is it clear what the exam should test in the different school types? Does the exam in fact test what it should test? Does the exam test the same things in all schools of the same type so that the results are comparable? Does the exam test the same things every year so that results are comparable? Does the exam relate to internationally accepted descriptions of educational standards such as the CEFR?

Q1: Is it clear what the exam should test? WHAT: curricular objectives / test construct - Models of language competence - Curriculum - CEF HOW: test method

Origins of the WHAT

L.F. Bachman, Fundamental Considerations in Language Testing, OUP 1990, p. 87.

E8 Reading Strategies (extract) Careful Reading Understanding the Text - understanding explicitly stated main idea(s) and / or distinguishing that from supporting details - locating, identifying, understanding and comparing facts, opinions, definitions (this includes search reading strategies) - understanding the logical organisation of the text, e.g. understanding relationships among ideas in a text (problem – solution, cause – effect, temporal sequence, etc.) - understanding cohesive relationships (reference, ellipsis, substitution, conjunction, lexical cohesion) - making propositional inferences (deducing information that is not explicitly stated from information that is explicitly stated) Adapted from Urquhart & Weir 1998

CEF Overall Written Production B2 Can write clear, detailed texts on a variety of subjects related to his / her field of interest, synthesising and evaluating information and arguments from a number of sources. C1 Can write clear, well-structured texts of complex subjects, underlining the relevant salient issues, expanding and supporting points of view at some length with subsidiary points, reasons and relevant examples, and rounding off with an appropriate conclusion.

HOW? What do we do in order to make our students display their competences?

Item 1 You are asking your friend a lot of questions about him and his family. Choose the right answer for each of the questions (there is only one correct answer per question!) and fill in the empty boxes with the correct number.

Item01 Test format: Grammar, Matching activity (2 nd grade AHS) You are asking your friend a lot of questions about him and his family. Choose the right answer for each of the questions (there is only one correct answer per question!) and fill in the empty boxes with the correct number. 1 Can Tom drive a car? Yes, they have. 2 Is your brother good at swimming? No, she isn ’ t. 3 Are you listening to the radio? Yes, I have. 4 Have you got a cousin? Yes, it is. 5 Can you come to our barbecue? No, he isn ’ t. 6 Does your sister go to nursery school? No, they can ’ t. 7 Has your aunt got a husband? Yes, we are. 8 Do your parents often go out in the evenings? No, I don ’ t. Yes, we do. No, they haven ’ t. No, we can ’ t. Yes, he can. No, they don ’ t. Yes, she can. Yes, she has. No, she doesn ’ t.

Item 1 Tick the answer which is grammatically correct: Is your brother good at swimming? Yes, he can. Yes, you are. Yes, he does. No, he isn’t.

You will have 10 minutes to do this task. Read the instructions carefully and then write your text on the separate sheet. Your short letter or should be 40 to 60 words long. Your American aunt has sent you a birthday present. Write a short letter or an to say thank you. Tell your aunt you really liked your present and why it is useful for you. You can also ask her if you could visit her in your summer vacation. You could start your letter like this: Test Method in Writing and Speaking Resist the temptation: keep method distinct from construct Dear Aunt Lizzy: If you rate the resulting text for Grammar, what is construct, what method?

Test Method in Writing and Speaking Resist the temptation: keep method distinct from construct Du hast für diese Aufgabe 10 Minuten Zeit. Lies die Anleitung gut durch und schreib Deinen Text auf das beiliegende Blatt. Dein kurzer Brief oder Deine soll 40 bis 60 Wörter lang sein. Deine Tante in Amerika hat Dir ein Geburtstagsgeschenk geschickt. Schreib einen kurzen Brief oder eine , um dich zu bedanken. Sag Deiner Tante, dass Dir das Geschenk sehr gut gefällt und warum Du es gut gebrauchen kannst. Du kannst sie auch fragen, ob Du sie in den Sommerferien vielleicht besuchen kannst. Du könntest Deinen Brief so beginnen: Dear Aunt Lizzy: If you rate the resulting text for Grammar, what is construct, what method?

Test Method ctd. If you rate the resulting text for Vocabulary, what is construct, what method? If you rate the resulting text for Cohesion and Coherence, what is construct, what method?

If you rate the resulting text for Vocabulary, what is construct, what method? If you rate the resulting text for Cohesion and Coherence, what is construct, what method? Test Method in Writing and Speaking Resist the temptation: keep method distinct from construct Du warst vor ein paar Tagen bei der Geburtstagsparty Deines Freundes/Deiner Freundin. Schreib ihm/ihr ein , um ihm/ihr zu sagen, dass Dir die Party gefallen hat. Du warst vor ein paar Tagen bei der Geburtstagsparty Deines Freundes/Deiner Freundin. Schreib ihm/ihr ein , um ihm/ihr zu sagen, dass Dir die Party gefallen hat. Sag ihm/ihr, warum Dir die Party gefallen hat. Sag ihm/ihr, warum Dir die Party gefallen hat. Sag ihm/ihr, was Dir am besten gefallen hat. Sag ihm/ihr, was Dir am besten gefallen hat. Frag ihn/sie, wann ihr einander wiedersehen könnt. Frag ihn/sie, wann ihr einander wiedersehen könnt. Mach einen Vorschlag für das kommende Wochenende. Mach einen Vorschlag für das kommende Wochenende.

So? To state what the exam should test, we must be able to say what is the construct and what the method. We want to test construct, not method.

Q2: Is it clear what the exam should test in the different school types? Once we are clear about the distinction between construct and method, we are in a good position to discuss this question. Which areas of the construct are relevant for all school types, which only for particular school types?

Q3: Does the exam in fact test what it should test? Construct + method will yield an exam which we believe tests what we are interested in. It is our obligation to demonstrate that this exam indeed tests what we claim it tests. Are we measuring anything at all or are we just ascribing figures to performances on a random basis? – reliability - Scoring keys must be available. - Rating scales have to be put in place. - Raters need to be trained in using the rating scales. - It should not matter who conducts the exam. - It should not matter who rates / scores it. Are we measuring the things we set out to measure? Accompanying research is needed: - Content validation - Construct validation

Q4: Does the exam test the same things in all schools of the same type so that the results are comparable? Use the same exam in all the schools of the same type at the same time. Develop parallel exams which test the same things and which are equally difficult.

Q5: Does the exam test the same things every year so that the results are comparable? Use the same exam every year (a theoretical possibility). Develop parallel exams which test the same things and which are equally difficult.

Parallel forms are needed. This requires identity of construct plus constancy / comparability of difficulty. Parallel forms are needed. This requires identity of construct plus constancy / comparability of difficulty. This is related to setting cutoff scores for pass / fail decisions. This is related to setting cutoff scores for pass / fail decisions. This leads to the issue of how these cutoff scores relate to the CEF (B2?) This leads to the issue of how these cutoff scores relate to the CEF (B2?) Moreover, there is a need to determine cutoff scores for the individual grades on the national 5- point scale (multiple cutoffs). Moreover, there is a need to determine cutoff scores for the individual grades on the national 5- point scale (multiple cutoffs). This calls for standard-setting procedures to be put in place. This calls for standard-setting procedures to be put in place. Q6: Does the exam relate to internationally used descriptions of educational standards such as the CEF?

Standard Setting Problem: item-banking not practical in the Austrian Matura context (item security). Consequence: standard setting required for each parallel form. Needed: descriptions of minimally acceptable competences for Matura ( ). CEF comes in handy but is not sufficient.

Bottom Line For all of these things to work, we need items and tasks based on a clearly defined construct with a clearly defined test method.

Item vs Task A cline which is defined by length of response. A cline which is defined by length of response. Tick: item (closed format) Tick: item (closed format) Long open-ended response: task Long open-ended response: task Receptive skills lend themselves to assessment by closed item formats; productive skills to assessment by open-ended formats. Receptive skills lend themselves to assessment by closed item formats; productive skills to assessment by open-ended formats.

Thank you!

Developing Reading Items Read CEF descriptors. Read through the input texts and identify sub-competences in these descriptions for which you then try to write reading comprehension items. Fill in the Item Characterisation Form after you have finished an item.

Developing a Writing Task Develop a writing prompt plus a scale for assessing the student performances originating from the prompt. Prompt - Decide on a topic area. - Decide on a text type and addressee. - Formulate a concrete task. - Decide on required length in number of words. Scale - Decide on the aspects (dimensions) which determine the quality of the performance. - Describe two levels on each dimension: bare pass and excellent. Finally, describe the test construct (The ability to...).