Effective assessment in science and maths Daisy Christodoulou, Director of Education, No More Marking

Quality model and difficulty model

Quality model of assessment The most obvious [method of assessment] is to ask the students to do something and see how well they do it. By referring to a scaled set of criteria or descriptors, the examiner judges which number on the scale best indicates how well each student has performed. Competitive ice dancing uses a pure quality model. The skater is expected to go out and perform in a way that impresses the judges as much as possible. Ahmed, Ayesha, and Alastair Pollitt. "The support model for interactive assessment." Assessment in Education: Principles, Policy & Practice 17.2 (2010): 133-167, p.134 The most obvious [method of assessment] is to ask the students to do something and see how well they do it. By referring to a scaled set of criteria or descriptors, the examiner judges which number on the scale best indicates how well each student has performed. We call this the quality model of measurement because central to it is the process of judging the quality of a performance Competitive ice dancing uses a pure quality model. The skater is expected to go out and perform in a way that impresses the judges as much as possible. In practice, the method is limited to the sorts of activities in which we can observe students operating in reasonably standard circumstances, such as speaking in a foreign language, playing the piano, or painting in oils. It is not so appropriate for measuring understanding of theories in subjects such as mathematics, science or geography.” Ahmed, Ayesha, and Alastair Pollitt. "The support model for interactive assessment." Assessment in Education: Principles, Policy & Practice 17.2 (2010): 133-167, p.134

Difficulty model of assessment “The usual approach consists of setting a series of tasks that vary in difficulty, often starting with easy ones and moving on to harder ones. What we ‘see’ is which tasks each student can do successfully. We call this the difficulty model of measurement because central to it is the mapping of students’ ability to the difficulties of a graded set of tasks (it has also been called the Counting strategy)” A high jump competition is a clear example of the difficulty model, consisting of a series of tasks of ever increasing difficulty which continues until everybody has failed.

Quality model and difficulty model – equal technical challenges, but in different ways “The technical concern in the quality model is the reliability of the judges, whereas in the difficulty model it is the internal consistency of the questions.” Quality model –a lot of work to do after the assessment! Difficulty model –a lot of work to do before!

What’s appropriate for maths and science? Traditionally, we use the difficulty model for maths and science. This is totally appropriate and can make assessment very easy for the teacher / marker. Advantages: as long as you use externally-set tests, you can quickly and easily get nationally standardised data on your pupil’s performance. Disadvantage 1: tests you’ve set yourselves might look just like a GCSE paper – but they aren’t! You can’t use these to get nationally standardised results. Disadvantage 2: you can’t use more open questions.

Why can’t I set my own questions and get nationally standardised results? ‘Can compare two fractions to identify which is larger’ Which is bigger: 3/7 or 5/7? 90% get this right Which is bigger: 3/4 or 4/5? 75% get this right Which is bigger: 5/7 or 5/9? 15% get this right Qtd in Wiliam, Principled Assessment Design, SSAT 2014

What if I want to use quality model styles of question in maths and science? People often want to use quality model questions because they think they are more valid. The problem is that – traditionally - quality model questions are hard to mark reliably. If an assessment is not reliable, it cannot be valid either!! Reliability is a necessary condition of validity. The risk is that you introduce an exciting and original real- world question designed to shed new light on a pupil’s thinking – but because it is so hard to mark it consistently, it doesn’t actually shed any new light! You don’t know if the results from it are down to a pupil’s original thinking – or marking error!

Other problems with the way quality model questions are traditionally used in maths and science People attempt to address the unreliability problem by producing very specific mark schemes – the problem with this is that it then ends up stereotyping the response from the pupil (this was a real problem with the old maths & science coursework) Further risk with maths and science is that you end up rewarding literacy, not maths and science, as open questions tend to involve extended writing

Maths and science GCSEs include more ‘quality model’ style questions – and it’s not always a good thing…

Can Comparative Judgement provide a solution?

Normally, we ask: does this essay meet the criteria? This is how traditional marking of writing works. Normally, we ask: does this essay meet the criteria?

Instead, we should ask, is this essay better than this essay? This is how comparative judgement works. Instead, we should ask, is this essay better than this essay?

What rules do you know in maths that are always true? Good questions for Comparative Judgement Not so good What rules do you know in maths that are always true? How do you add and subtract fractions? How do you solve equations? Add ¾ and ½ A block of cheese costs £2.66 per kg. How much does 4.5kg cost?

Where does the human body get energy from? What is a chemical bond? Science Good questions for Comparative Judgement Not so good Where does the human body get energy from? What is a chemical bond? Why are some elements more reactive than others? Write the formula for a molecule of chlorine gas. Give one purpose of chlorine in industry.

Quality model and difficulty model – equal technical challenges, but in different ways “The technical concern in the quality model is the reliability of the judges, whereas in the difficulty model it is the internal consistency of the questions.” Quality model –a lot of work to do after the assessment! Comparative judgement solves this problem by getting teachers / markers to judge the pupil work Difficulty model –a lot of work to do before! Comparative judgement solves this problem by getting teachers / markers / experts to judge the difficulty of the questions in advance of them being taken by pupils

No More Marking Twitter: @daisychristo Email: support@nomoremarking.com daisy@nomoremarking.com Blog: https://blog.nomoremarking.com/