Important Exams and the Concept of Good Practice Dávid Gergely

Interpret focus of lecture Hardly feasible task to cover all important exams Many websites, information booklets, etc. provide the information anyway, for interested fellow professionals New focus about –quality of examinations –what may be considered good professional practice. „Tools for fishing”: a series of criterial questions

What can you do? Study test specifications and other printed info packs, mock exams and any published docs. Study regulations Carry out an observation task, collect information.

Studying the specifications 1 Is the target language sampled widely enough? –The nature of foreign lang. Proficiency –Skills, communicatve language activities, abilities Are all four skills + grammar/vocab tested? Litmus test: Is listening comprehension tested?

Studying the specifications 2 Is the candidate required to produce a large sample of the target language? Are the papers long enough? –Not enough to include 5-6 papers, but there must be enough items etc. in each to give candidates as many chances (fresh new starts, fairness) to show what they know. –In terms of our litmus listening test, how many items based on how many sound tapes (hangzóanyag) is good enough? –20 items based on 3 sound tapes is minimum.

Explanation Not covering enough skills (subskills), activities, abilities and/or doing that through too few items is called the underrepresentation of the test construct – a typical problem with many tests. –The difficulty of inferring underlying proficiency from test proficiency. –The quest for information: Student 1: 1,0,1,0,0,1,0,0,1,1 Student 2: 1,2,1,0,2,0,1,1,0,2

Studying the specifications 3 Static specs of papers, tasks types, etc. Do not speak about what the test provider do to ensure quality Is the testing process specified too? Procedural validity

Model testing process: test material Drafting tasks Moderation Revision of drafts Pretesting draft tasks Revision 2 Administration Post-test quality control Score calculation Standard (level) setting

Model testing process: examiner Training Benchmarking Administration Post-test quality control Score calculation Standard (level) setting

Study regulations, observe practice Is the model testing procedure followed? If it is not followed in every respect, how do they „make up for it”?

Applying the procedure to ensure validity There must be a quality control mechanism that assumes, especially: –Candidate response data (all such data) are collected systematically –Response data are analysed statistically after the pretest, but most importantly the post-test, in order to decide how good the items were, how well the raters marked, whether intended level was maintained, etc. Are response data collected systematically? Are the data analysed statistically?

Illustrations of data problems How is the oral scored? –Two raters added up? Averaged? –Raters agree? Separately given scores recorded? Is it possible to put together, candidate by candidate, scores from the written papers with scores from the oral?

Data? Analysis? Why the hell? The Holy Grail of language testing –Want to find out candidates’ lang. proficiency –Not directly measurable –Must use tools (items, raters, criteria, i.e. test method) to make that proficiency observable –Whatever the method, it will unavoidably influence test performance and results Unknown proficiency+method effect = scored test performance

3232,1,63-74,1,1,1,1,1,1,1,1,0,1,1,1 3233,1,63-74,1,1,1,1,1,1,0,1,1,0,1,1 3234,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3235,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3236,1,63-74,0,0,0,1,0,1,1,1,0,0,0,0 3237,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3238,1,63-74,1,1,0,1,1,1,0,1,0,1,1,1 3239,1,63-74,1,1,1,1,1,1,0,1,1,1,1,1 3240,1,63-74,0,1,1,1,0,1,1,0,1,0,1,0 3241,1,63-74,0,1,1,1,0,0,1,1,0,0,0,0 3242,1,63-74,1,0,1,1,1,1,1,1,1,1,1,1 3243,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3244,1,63-74,0,1,1,1,1,1,0,1,0,1,1,1 3245,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3246,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3247,1,63-74,0,0,1,1,0,0,0,0,0,1,0,0 3248,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3249,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3250,1,63-74,1,0,1,0,1,1,1,1,1,1,1,0 3251,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3252,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3253,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3254,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3255,1,63-74,0,1,0,1,0,1,0,1,1,0,0,0 3256,1,63-74,1,1,1,1,1,1,0,1,1,1,0,0 3257,1,63-74,0,1,0,0,1,1,0,0,1,0,1,1 3258,1,63-74,0,0,0,1,1,0,0,1,0,1,0,0 3259,1,63-74,0,0,0,0,1,0,1,1,0,1,1,0 3260,1,63-74,0,1,1,1,0,1,1,1,1,0,1,0 3261,1,63-74,0,1,1,0,1,0,0,1,1,1,1,0 3262,1,63-74,0,0,0,1,1,0,0,0,1,0,0,0 3263,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3264,1,63-74,1,1,1,1,1,1,0,1,1,1,1,0 3265,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3266,1,63-74,1,1,0,1,1,1,0,1,1,1,1,1 3267,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3268,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3269,1,63-74,0,1,1,1,0,1,1,1,1,1,0,1 3270,1,63-74,1,0,0,1,1,1,1,1,0,0,0,0 3271,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3272,1,63-74,0,1,1,1,1,1,0,1,0,0,0,0 3273,1,63-74,0,0,0,1,0,1,0,1,1,1,1,0 3274,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3275,1,63-74,1,1,1,1,1,1,0,1,1,1,1,1 3276,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3277,1,63-74,0,1,0,1,1,1,0,1,1,1,0,0 3278,1,63-74,1,1,0,0,1,1,0,1,0,0,1,0 3279,1,63-74,1,1,0,1,0,1,1,0,1,0,1,0 3280,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3281,1,63-74,1,1,1,1,1,1,0,1,0,1,0,0 3282,1,63-74,1,1,1,1,1,1,0,1,1,1,1,1 3283,1,63-74,1,1,0,1,1,1,1,1,0,1,1,1 3284,1,63-74,0,0,0,1,0,0,0,1,0,1,0,0 3285,1,63-74,1,1,1,1,1,1,0,1,1,0,0,1 3286,1,63-74,0,1,1,1,0,0,1,0,1,1,1,0 3287,1,63-74,0,0,1,1,1,1,1,0,1,0,1,0 3288,1,63-74,1,0,1,1,1,1,0,1,1,1,1,0 3289,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3290,1,63-74,0,0,0,1,1,1,1,1,0,0,0,0 3291,1,63-74,0,1,0,1,1,0,1,1,1,1,0,0 3292,1,63-74,1,0,1,1,1,0,0,1,1,0,1,0 3293,1,63-74,1,1,1,1,1,0,1,1,0,1,1,0 3294,1,63-74,1,1,0,1,1,0,0,1,0,0,1,0 3295,1,63-74,0,1,1,1,1,1,0,1,1,1,1,0 3296,1,63-74,0,1,1,1,0,1,1,1,1,1,1,1 3297,1,63-74,0,0,0,1,0,1,1,1,0,0,1,0 3298,1,63-74,0,1,0,1,1,1,0,1,0,1,0,0 3299,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3300,1,63-74,1,1,1,1,1,1,1,1,1,1,0,1 3301,1,63-74,1,0,1,1,1,1,1,1,0,1,0,0 3302,1,63-74,1,1,1,0,1,1,1,1,1,1,1,1 3303,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3304,1,63-74,1,0,0,1,1,1,0,1,1,0,1,1 3305,1,63-74,1,0,1,1,1,0,1,1,1,0,0,0 3306,1,63-74,1,1,1,1,1,1,0,1,1,0,0,0 3307,1,63-74,0,0,1,1,0,1,0,1,0,1,0,0 3308,1,63-74,0,1,0,1,0,0,0,1,1,0,0,0 3309,1,63-74,1,1,1,1,1,1,1,1,1,1,0,1 3310,1,63-74,0,1,0,1,1,1,0,1,0,1,1,0 3311,1,63-74,0,0,0,1,0,1,0,1,0,1,0,1 3312,1,63-74,1,1,1,1,1,1,1,1,0,1,1,0 3313,1,63-74,1,1,1,1,1,1,0,1,0,0,1,0 3314,1,63-74,0,0,0,1,0,1,0,1,0,1,0,0 3315,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3316,1,63-74,0,0,0,1,0,0,0,1,0,1,0,0 3317,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3318,1,63-74,1,1,1,1,1,1,0,0,1,1,0,0 3319,1,63-74,0,0,0,1,0,0,0,1,0,0,1,1 3320,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3321,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3322,1,63-74,0,1,0,1,1,1,1,1,0,1,1,1 3323,1,63-74,1,1,1,1,0,1,0,1,1,1,1,1 3324,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3325,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3326,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3327,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3328,1,63-74,0,1,0,0,0,1,1,1,0,1,1,0 3329,1,63-74,1,1,1,1,1,1,0,0,1,0,1,0 3330,1,63-74,0,1,0,1,1,1,1,1,0,0,0,0 3331,1,63-74,0,1,0,0,1,1,0,1,0,1,0,1 3332,1,63-74,1,1,1,1,1,1,0,1,1,0,0,0 3333,1,63-74,0,1,1,1,0,1,1,1,1,1,1,1 3334,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3335,1,63-74,1,1,0,1,1,1,1,1,0,1,1,1 3336,1,63-74,1,1,0,1,1,1,1,1,0,1,1,1 3337,1,63-74,1,0,1,0,1,1,1,1,1,0,1,0 3338,1,63-74,1,1,1,1,1,1,1,1,1,1,0,0 3339,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3340,1,63-74,0,0,0,1,1,0,0,0,1,1,0,0 3341,1,63-74,1,0,1,0,1,1,1,1,1,1,1,1 3342,1,63-74,1,1,1,1,1,1,0,1,1,0,1,0 3343,1,63-74,0,1,1,1,0,1,0,1,0,1,0,1 3344,1,63-74,1,1,0,1,0,1,0,1,1,1,0,1 3345,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3346,1,63-74,1,1,0,1,0,1,1,1,0,1,1,0 3347,1,63-74,0,1,1,1,0,0,0,1,1,1,1,0 3348,1,63-74,0,0,0,0,0,1,0,0,0,1,0,0 3349,1,63-74,1,1,1,1,1,1,0,1,1,1,1,0 3350,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3351,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3352,1,63-74,1,1,1,1,1,1,0,1,0,1,1,1 3353,1,63-74,0,1,1,0,1,1,0,1,1,1,1,0 3354,1,63-74,1,1,1,1,1,1,0,1,0,0,0,0 3355,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3356,1,63-74,0,0,1,1,1,0,0,1,1,0,1,0 3357,1,63-74,1,1,1,0,1,1,0,1,1,1,1,1 3358,1,63-74,1,1,1,1,1,1,0,1,1,1,1,0 3359,1,63-74,1,1,1,1,1,1,0,1,0,1,1,0 3360,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3361,1,63-74,1,1,1,0,1,1,0,1,1,1,1,0 3362,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3363,1,63-74,0,1,0,0,1,1,1,1,1,1,1,1 3364,1,63-74,1,1,1,1,0,1,1,1,1,0,1,1 3365,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3366,1,63-74,1,1,1,1,1,1,0,1,0,0,0,0 3367,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3368,1,63-74,1,0,0,1,1,1,0,1,0,0,0,1 3369,1,63-74,1,1,1,1,1,0,0,0,0,1,0,0 3370,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3371,1,63-74,0,1,1,1,0,1,1,1,1,1,1,0 3372,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3373,1,63-74,0,0,1,1,1,1,0,1,1,0,1,0 3374,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3375,1,63-74,1,1,1,1,1,1,0,1,1,1,1,1 3376,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3377,1,63-74,1,1,1,1,1,1,1,0,1,1,1,0 3378,1,63-74,0,0,1,1,0,1,1,1,0,0,0,0 3379,1,63-74,1,1,1,1,1,1,0,1,1,1,1,0 3380,1,63-74,1,1,1,1,1,1,1,1,0,1,1,0 3381,1,63-74,1,0,1,0,1,1,0,1,1,0,1,0 3382,1,63-74,0,1,0,1,1,1,1,1,1,1,1,0 3383,1,63-74,1,1,0,1,1,1,0,1,1,1,1,1 3384,1,63-74,1,1,1,1,1,1,1,1,1,1,0,1 3385,1,63-74,1,1,1,1,1,1,1,1,0,1,0,1 3386,1,63-74,1,0,0,1,1,0,1,0,0,0,0,0 3387,1,63-74,1,0,1,1,0,1,0,1,0,0,0,1 3388,1,63-74,1,1,1,1,1,1,0,1,1,1,1,0 3389,1,63-74,0,1,1,1,0,1,1,1,1,1,1,0 3390,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3391,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3392,1,63-74,0,1,1,0,1,0,0,1,0,1,0,1 3393,1,63-74,0,1,0,1,1,1,1,1,0,1,0,0 3394,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3395,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3396,1,63-74,0,0,0,1,1,1,0,1,0,0,1,1 3397,1,63-74,0,0,1,1,1,1,0,1,0,0,0,0 3398,1,63-74,0,1,1,1,0,1,1,1,1,1,1,0 3399,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3400,1,63-74,1,1,1,1,1,1,0,1,1,1,1,1 3401,1,63-74,0,1,1,1,0,1,0,1,1,1,0,1 3402,1,63-74,0,0,1,1,1,0,0,1,0,0,0,0 3403,1,63-74,0,0,1,1,0,0,0,1,1,0,1,0 3404,1,63-74,0,1,1,1,0,0,0,0,0,0,1,0 3405,1,63-74,1,1,1,1,1,1,1,0,1,1,1,1 3406,1,63-74,0,0,0,1,0,0,0,1,0,0,0,0 3407,1,63-74,1,1,1,1,1,1,0,1,1,1,0,1 3408,1,63-74,0,0,0,0,0,1,0,0,0,0,1,0 3409,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3410,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3411,1,63-74,0,0,1,0,1,0,1,0,1,0,1,0 3412,1,63-74,0,1,0,1,1,1,0,1,1,1,1,0 3413,1,63-74,1,0,0,1,1,1,0,1,0,1,1,0 3414,1,63-74,1,1,1,0,1,1,1,1,0,1,1,0 3415,1,63-74,1,1,1,1,1,1,0,1,1,1,1,1 3416,1,63-74,0,1,1,1,1,1,0,1,1,1,0,0 3417,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3418,1,63-74,1,1,1,1,1,1,1,1,0,1,1,1 3419,1,63-74,1,0,0,1,0,1,0,1,0,1,0,0 3420,1,63-74,0,1,1,1,0,1,1,1,0,0,1,1 3421,1,63-74,0,1,1,1,1,1,0,1,0,0,1,1 3422,1,63-74,1,1,1,1,0,1,0,1,1,1,0,0 3423,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3424,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3425,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3426,1,63-74,1,1,1,1,1,1,0,1,0,1,1,1 3427,1,63-74,0,0,1,1,0,1,0,0,1,1,1,0 3428,1,63-74,1,1,1,1,1,1,0,0,0,1,0,0 3429,1,63-74,0,0,0,0,0,0,0,0,1,0,0,0 3430,1,63-74,1,1,1,1,1,1,0,1,0,1,1,1 3431,1,63-74,0,0,1,1,0,0,0,0,1,1,0,1 3432,1,63-74,0,0,0,0,1,0,0,0,0,0,1,0 3433,1,63-74,0,1,0,1,0,1,1,1,0,0,1,1 3434,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3435,1,63-74,0,1,0,1,1,1,0,1,0,1,1,0 3436,1,63-74,0,0,0,1,0,0,0,0,0,0,1,0

Study regulations, observe practice Has the exam gone through accreditation procedure by the Ministry? Many exams have, some have not (list –The Hungarian accreditation system not perfect, but at least –a body of experts to review exams regularly and on demand –documentation must be submitted Some achievements

Study regulations, observe practice How soon are results announced? If announced very soon, it is unlikely to be a good test, as it is impossible to do the quality control work that needs to be done.

Study regulations, observe practice Are most complaints handled favourably for the candidates? If quality control procedures are weak, such practice sends a wrong message: –„Kaparj kurta neked is jut”, making the candidate responsible for finding problematic items, raters, etc. –Implicit in law –Unfair. Quality control should not depend on candidates’ awareness of their rights.

Study regulations, observe practice Are pass rates very high? Bear national educational context in mind: –falling numbers in the age-group where candidates might come from (14-22) –falling foreign language education standards should predict falling pass rates too if standards (CEFR levels) are maintained. –The horrible case of a prestigious German exam 10 years ago.

Summary What is good testing practice? –Use this checklist of questions –Read up on language testing and add more questions What you need –a keen eye –a sane mind.

Required reading McNamara, T. (2000). Language Testing. Oxford Introductions to Language Study. Series Editor: H.G Widdowson.Oxford: OUP. –The required chapters are –The Testing Cycle (Chapter 3) pp ; Validity: testing the test (Chapter 5) pp ; –Measurement (Chapter 6) pp