1. Regulating for quality in assessment AAIA Conference 2010 Isabel Nisbet 15 September 2010

2. Summary  Ofqual  Responsibilities for assessments  Reliability  Standards  Our approach in action - GCSEs and A levels - Checking that standards are maintained  Innovation and E-assessment and on-demand testing

3. Overview Ofqual’s role in relation to qualifications and [National Curriculum] assessments:  encourage and support the development and implementation of [qualifications] [regulated assessments] that give a reliable indication of achievement, and that indicate a consistent level of attainment (including over time) between comparable [qualifications] [assessments]  promote public confidence in [regulated qualifications] [assessment arrangements]

4. Ofqual’s Reliability Programme “..As the regulator of qualifications in England I believe that it is essential for us to understand better the reliability of assessments in our national systems… Ofqual will undertake an in-depth programme of work – call it a health check – on the reliability of tests, examinations and teacher assessments in this country.” Kathleen Tattersall, May 2008

5. A perfect assessment? "We have really got to begin to engage with the public in a dialogue about the nature of the examination system. "There's a broad expectation that assessment should be absolutely perfect and accurate, that a mark of 50 is a mark of 50, regardless of who marks, the time at which it is marked and so on. "Would a student have received the same result if she happened to have taken a different version of the examination, on a different day, with a different examiner marking her work?” Kathleen Tattersall, 2008

6. Reliability  Reliability is an indicator of the quality of an assessment  Reliability work in England has generally been - Isolated - Partial - Under-theorised - Under-reported - Misunderstood  Ofqual’s reliability programme will help to improve the situation

7. To gather evidence for Ofqual to develop regulatory policy on reliability of results from national tests, examinations and qualifications Aims

8. Strand 1: Generating evidence of reliability Strand 2: Interpreting and communicating evidence of reliability Strand 3: Developing reliability policy Strand 3a: Exploring public understanding of reliability Strand 3b: Developing Ofqual policy on reliability Programme structure

9. Work in progress  AQA: The reliabilities of GCSE and GCE components  Cambridge Assessment: The reliabilities of GCSE and GCE components and qualifications  City & Guilds: The reliabilities of vocational qualifications:  Assessment Europe: Quantifying and interpreting component reliability using G-theory analysis  Assessment Europe: Literature review of reliability studies of teacher assessments  TAG Report: Summary, further study and advice  Ofqual: Quantitative investigation of public perceptions using online questionnaire

10. Reliability – lessons  Concept of “error” misunderstood - Variability doesn’t mean “mistakes”  Public concerns but understanding about use of judgement  Issues around implications for progression and accountability of small differences in marks  Limits to reliability of other methods of assessment too - Pre-university selection tests - Interviews/vivas

11. Standards – some provocative remarks  Tired old arguments about standards over time  Important new arguments about whether qualifications (academic, professional or vocational) are FIT FOR PURPOSE  Time to challenge: –Unevidenced prejudices » against some qualifications » against some awarding organisations –Urban myths –National inferiority complexes –Unattainable ideals about reliability

12. Maintaining standards in practice – GCSEs and A levels  Ofqual sets conditions awarding organisations must meet to be recognised – governance; expertise; quality assurance  Ofqual recognises awarding organisations that show they can meet the conditions  Ofqual sets criteria for specific groups of qualifications (eg “GCSE history”)  Ofqual accredits qualifications that meet these criteria  Ofqual makes sure that different versions of the same qualification are comparable  Ofqual issues codes of practice - principles, processes and practices to be followed by organisations developing and delivering GCSEs and A levels  Ofqual makes sure that action is taken in real time to ensure that each set of awards is consistent and fair and that standards are maintained over time

13. Checking standards in all regulated qualifications  Monitoring the questions set and the marking of examiners  Checking that awarding organisations carry out their roles correctly, through - scrutiny programme - code of practice monitoring programme - audits of awarding organisations  Comparability studies (reviewing qualification standards over time, across organisations and between subjects)  Planned international comparison study  ALL qualifications, selected by risk

14. Innovation  Ofqual aims to facilitate innovation – e-assessment? On- demand?  Need for valid assessments of 21 st Century learning  What might drive innovation? - Efficiency - Demand - Quality - Changes in teaching and learning - Technological change  Ofqual has published principles to govern E-assessment to support innovation while protecting standards and integrity

15. Regulatory principles for e-assessment  Validity and reliability  Security  Data integrity  Access to e-assessment  Business continuity and disaster recovery  Use of e-portfolios

16. The extent of e-marking across Awarding Bodies: 2007  36% of GCSE and A level scripts electronically marked (wide variation across awarding bodies)  Awarding Body trials found: - Reliable - Timely - Scanning and supporting technologies did not fail - Accurate assembly of results and archiving - (Senior) Examiner satisfaction -No extra admin burden on centres  No link to concerns over quality of marking

17. The big step  From E-marking to E-testing

18. Over to you  How we can help you - Providing essential information on assessments - Providing information on what we require from those involved in delivering assessments  How you can help us - Give us your view on what assessments should deliver - Help us to make assessments as fair and accurate as possible  Contact us: www.ofqual.gov.uk

19. Summary  Ofqual  Responsibilities for assessments  Reliability  Standards  Our approach in action - GCSEs and A levels - Checking that standards are maintained  Innovation and E-assessment and on-demand testing

20. Thank you www.ofqual.gov.uk isabel.nisbet@ofqual.gov.uk