Presentation on theme: "How to be a Good Examiner Assessment Inset Day - 6 January 2011 Stephen May."— Presentation transcript:
How to be a Good Examiner Assessment Inset Day - 6 January 2011 Stephen May
Papers and Books? A lot on teaching in HE Less on assessment, and very methods-oriented Some on role of external examiners (including Code of Practice) Very little on the subject/discipline (internal) examiner! [Most available lists of relevant qualities relate to public examination roles.]
Possible to extract from assessment and staff development literature the key principles which need to be understood, and qualities which are required.
Key Elements Knowledge of subject Knowledge of students Knowledge of course, and learning outcome expectations Assessment knowledge and skills Good written communication skills (and verbal, if using oral examinations) Good organisational / team working abilities The right attitude!
Knowledge of Subject Important for question-setting (and also marking if any subjectivity involved) Less important, for marking stage, the more objective the test Ramsden (2003) – rules for better assessments –Focus on validity before reliability! (Avoid testing trifling aspects because they are easier to measure.) –Use multiple choice and other objective tests cautiously So subject knowledge will always be important in all aspects of professional examinations, but.... not nearly enough of itself!
Knowledge of Students and Course Expectations Understanding by students of the assessment game Knowledge of learning outcomes, and the implications for assessment –Knowledge –Reasoning –Decision-making –Technical skills –Communication skills –Professional attributes?
Assessment Knowledge and Skills Assessment Purposes (Code of Practice, Section 6) Evaluating student knowledge, understanding, abilities or skills Providing a mark or grade.... may also be used to make progress decisions Enabling the public (including employers).... to know.... fitness to practise Promoting student learning.... feedback
Assessment Knowledge and Skills Assessment Methods Subjective v. objective Holistic v. fragmented Higher order v. factual information and rote learning Strengths and weaknesses of different formats Remembering: Links to learning outcomes Hierarchical skills development across course
Assessment Knowledge and Skills Assessment Criteria / Marking Schemes Objective methods (e.g. multiple choice): straightforward, increasingly do not need a marker Subjective methods (e.g. long answer) –Knowledge –Structure of response –Clarity of expression –Level of engagement with question
Level of Engagement with Question Blooms taxonomy (1956) Lower to higher order skills Biggs SOLO taxonomy (1997, 1999) Ability to structure understand, integrate and reconceptualise Hatton and Smith (1995) Evidence-based, but self-critical review of knowledge and skills Problem is interpretation of separate components, and weighting for aggregation
Assessment Knowledge and Skills Assessment Criteria / Marking Schemes – Problems (Knight 2001) Difficult to develop statements of learning outcomes –What is critical thinking? –Your good communication = my bombast Precision produces wordy and complex documents and looser statements reduce clarity, so need a balance Must not confuse criterion-referenced judgements with criterion-determined ones All criteria subject to social processes by which meanings are constructed (both examiners and students)
Assessment Knowledge and Skills Assessment Criteria / Marking Schemes – Problems Long answers require clear individual understanding of the weighting process, and common understanding across examiner teams Hawk / dove effect difficult to avoid Key = comparable ranking of candidates Occasional random markers exposed by public examining boards (very difficult to train, unless have not understood criteria)
Assessment Knowledge and Skills Assessment Standards – Establishing Borderlines Norm referencing – based on set percentages on each occasion obtaining certain grades Criterion referencing – based on linkage to specific, pre-agreed levels of achievement
Assessment Knowledge and Skills Assessment Standards – Establishing Borderlines Criterion referencing For professional course, external expectations (day one skills) For other courses, collective academic expectations Traditional Oxbridge undergraduate standard: –Reasonable account of the examiners current knowledge 2(ii) –Account intelligently discussing knowledge the examiner once knew 2(i) –Stimulating debate of concepts and issues the examiner did not know, but can recognise as legitimate 1st But now embedded in progressive marking schemes (such as 17-point scheme) based on collective agreement
Assessment Knowledge and Skills Assessment Standards – Establishing Borderlines Criterion referencing –In long answer-type assessments, examiners have worked towards 50%, based on perceived degree of difficulty of question, memory of previous groups of similar students and perceived requirement to pass –Worked from point of view of examination, but often involved a norm referencing component at individual examiner level –Disastrous approach for objective examinations – must be standard set unless all questions are tested for level of difficulty in advance and mixed in appropriate ratios!
Good Written and Communication Skills Question-Writing Traditional essays Less work in advance, major workload associated with marking Writing (relatively) straightforward Standard setting during/after marking Objective questions (MCQ/EMQ) All work in advance (including standard setting) Much more time and care required for question writing; often team-based Standard setting level to defensibly adjust after the event
Good Written and Communication Skills Question Writing Need to avoid ambiguity Need to be succinct (appropriate structuring for MCQ stems and distractors) Need to anticipate all legitimate interpretations Need to avoid the If Ms/Mr/Dr/Professor X, they will be expecting.... answer if the question does not uniquely demand that answer
Good Written and Communication Skills Question Reading Its important to be able to pick up what the candidate is saying. Distinguish between what they actually did mean (which then deserves credit) and what you think they may have said / meant (which deserves no credit)
Good Organisational Ability Getting questions written on time (the reason why most people do not want to chair examination boards) –For internal moderation –For external scrutiny –For final examination papers Getting the marking done on time Telling people well in advance if you have a legitimate reason for not being available! The examination process (like all team activities) is only as good as its weakest link!
The Right Attitude You dont want a smart Alec: selecting examiners to assess doctoral dissertations (Kiley 2009) Experienced supervisors ensure they know the personality traits of potential examiners Avoidance of the bad and the mad Need a balance of empathy and understanding, with standards and integrity Keen to improve: contribute to updating assessment methods and learning to use these effectively
A Compulsory Examination for All Assessors! Answer all questions. Consultation with others (including students) is RECOMMENDED. 1.How well does your approach to assessment align with your learning outcomes? 2.Justify and criticise your choice of assessment methods (with reference to relevant research). 3.Describe, justify and criticise your assessment criteria and methods of grading/marking 4.Describe, justify and criticise your techniques for overcoming: Variations in standards – on a single and different occasions Variations in assessors 5.How do you ensure that your standards are similar to those in comparable assessments? 6.What values underlie your approach to assessment, and how are they manifest in your practice? (Modified from Brown 2001)
References: Biggs, J. (1997) Enhancing teaching through constructive alignment. Higher Education 32, 347-364. Biggs, J. (1999) Teaching for Quality Learning at University. Open University Press, Buckingham. Bloom, B.S. (1956) Taxonomy of Educational Objectives, the Classification of Educational Goals. Handbook 1: Cognitive Domain. McKay, New York. Brown, G. (2001) Assessment: Guide for Lecturers. LTSN Generic centre Assessment Series No. 3, LTSN, York. Hatton, N. and Smith, D. (1995) Reflection in teacher education: towards definition and implementation. Teaching and Teacher Education 11(1), 33-49. Kiley, M. (2009) You Dont Want a Smart Alec: Selecting Examiners to Assess Doctoral Dissertations. Studies in Higher Education 34, 889-903. Knight, P. (2001) A Briefing on Key Concepts: Formative and Summative, Criterion and Norm-Referenced Assessment. LTSN Generic Centre Assessment Series No. 7, LTSN, York. QAA (2006) Code of Practice for the Assurance of Academic Quality and Standards in Higher Education. Section 6: Assessment of Students. QAA, Gloucester. Ramsden, P. (2003) Learning to Teach in Higher Education, 2nd Edition. Routledge Falmer, London.