1. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 1 The Importance of being Valid Alastair Pollitt & Ayesha Ahmed Cambridge Exam Research alastair@camexam.co.uk ayesha@camexam.co.uk

2. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 2 Defining Validity There is no true meaning of Validity Definitions of Validity are opinions about what the word should mean How do we decide what Validity should mean? We need an inclusive concept of Validity that is useful to everyone involved in the process of assessment, and not too far from what they normally mean by Valid. A useful definition of Validity

3. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 3 The real world meanings of valid 1offer 102 2ticket 32 3licence 25 4certificate 25 5passport 19 6law 15 7voucher 14............... 18reason6 Relative RankConcordancefrequency Source: Collins Cobuild Corpora (British + American)

4. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 4 What we mean by Validity We have worked for many years with professionals who: Write questions; Review questions; Write mark schemes; Mark tests; Design tests; etc The key issue for them, and us, is what the students minds are/were doing during the test – in other words, the Evidence that gives us about achievement.

5. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 5 What we mean by Validity This is the essence of Validity – the interaction between a test question and the students mind. If this goes wrong then Validity is impossible. But how do we get it right? By considering further the concept of Evidence.

6. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 6 Evidence It is not enough to be Logically Valid. A widely used definition of assessment: The process of seeking and interpreting evidence for use by … (Assessment Reform Group, 2002: quoted by Stobart, 2009)

7. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 7 Importance Statement A negotiated consensual statement of what is really important in the study of that subject during that phase of education. Available to teachers, students, parents, examiners, employers, selectors – to everyone with an interest in the subject.

8. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 8 The Importance of Design & Technology Design and technology prepares pupils to participate in tomorrow's rapidly changing technologies. They learn to think and intervene creatively to improve quality of life. The subject calls for pupils to become autonomous and creative problem solvers, as individuals and members of a team. They must look for needs, wants and opportunities and respond to them by developing a range of ideas and making products and systems. They combine practical skills with an understanding of aesthetics, social and environmental issues, function and industrial practices. As they do so, they reflect on and evaluate present and past design and technology, its uses and effects. Through design and technology, all pupils can become discriminating and informed users of products, and become innovators.

9. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 9 The Importance of the Importance Statement To ensure that teachers emphasise what is important To ensure that test designers, question and mark scheme writers test what is important To ensure that markers give credit for what is important To ensure that test results are interpreted in terms of what is important The source of Validity

10. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 10 A model of Validity Validity is about measuring and interpreting what is important in education. It is not merely about measuring and interpreting what is measured. Our model describes where Validity comes from, and how it is maximised.

11. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 11 The Bucket Brigade model Suppose:Y is a house on fire. Suppose:X is a well full of water. How do you put out the fire?

12. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 12

13. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 13 A typical assessment professional

14. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 14 The Entropy law of validity Version 1 During the assessment process, validity always decreases Version 2 Validity can only be lost, it can never be recovered.

15. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 15 How to maintain validity - 1 Writing the Importance Statement

16. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 16 How to maintain validity - 2 Specifying the test

17. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 17 How to maintain validity - 3 Writing the questions and mark schemes

18. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 18 How to maintain validity - 4 The students

19. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 19 How to maintain validity - 5 Marking

20. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 20 How to maintain validity - 6 Grading, etc

21. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 21 How to maintain validity - 7 Interpreting results

22. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 22 How to maintain validity 1Writing the Importance Statement 2Specifying the test 3Writing the questions and mark schemes 4Communicating with students 5Marking 6Grading 7Interpreting

23. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 23 Summary The meaning of test results depends crucially on a proper development of the test, so that it elicits evidence of what is important. Without meaning, Validity is an empty concept. The source of Validity must be a clear and shared concept of Importance.

24. Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 24 8 The importance of being Valid E Q i M Idea of a Question Evaluation Elicitation IS A Importance Statement Teaching T Assessment Evidence