1. Using Comparative Judgement to collaborate on writing assessment
Daisy Christodoulou, Director of Education, No More Marking

2. Objectives By the end of this session, participants will understand
1) how comparative judgement works
2) how primaries and secondaries have been using it this year to collaborate on writing assessments.

3. What is Comparative Judgement?
Comparative judgement is based on teachers’ holistic judgements of the quality of one piece of writing compared to another. Despite its apparent subjectivity, comparative judgement is based on a longstanding psychological principle, which means that resulting assessments are reliable, valid and robust. It allows teachers to reward quality writing, rather than writing that conforms to a tick box.

4. Comparative not absolute Reduces reliance on prose descriptors
How is Comparative Judgement different from the traditional marking of writing?
Comparative not absolute
Reduces reliance on prose descriptors
Instead of making an absolute judgement about one piece of work in isolation, comparative judgement involves making a series of comparisons about pairs of pieces of writing.
Instead of making a judgement using a set of prose descriptors or criteria, comparative judgement involves teachers making holistic judgements based more on their instinct of what good writing is.

5. Who invented it? Louis Thurstone – 1927
Instead of making an absolute judgement about one piece of work in isolation, comparative judgement involves making a series of comparisons about pairs of pieces of writing.
Instead of making a judgement using a set of prose descriptors or criteria, comparative judgement involves teachers making holistic judgements based more on their instinct of what good writing is.

6. Normally, we ask: does this essay meet the criteria?
This is how traditional marking of writing works.
Normally, we ask: does this essay meet the criteria?

7. Instead, we should ask, is this essay better than this essay?
This is how comparative judgement works.
Instead, we should ask, is this essay better than this essay?

8. Why use comparative judgement?
Efficiency – it’s quick!
Reliability – it’s accurate!
Validity – it can be used immediately to give helpful and specific feedback, and has longer-term potential to change & improve the way you teach writing.
is an owner of this task

9. So how does this work in practice? Try the colours game…

11. Validity – are we measuring the right thing
Validity – are we measuring the right thing? Which is EXS and which is WTS?

12. Validity – are we measuring the right thing
Validity – are we measuring the right thing? Which is EXS and which is WTS?

13. Using comparative judgement to collaborate with other schools

14. What is a national judging window?
In a national judging window, you don’t just judge your own pupils’ work.
80% of the judgements you do involve comparing your pupils vs. your pupils.
The other 20% involve comparing pupils from other schools with pupils from other schools.
This means the final results are nationally standardised.

15. What do national results look like?
In November 2017, 121 schools and 6,723 pupils took part in the Sharing Standards year 3 judging window. 1,393 teachers made 113,200 judgements on their work.
In November 2017, 5530 pupils from 37 schools took part in Progress to GCSE English, judging year 10 mock questions. The pupils’ work was judged by 367 teachers who completed a total of 83,046 judgements.

16. How did pupils in the pilot perform
How did pupils in the pilot perform? & How did pupils at my school do compared to the other pupils in the pilot?
The box and whisker plot on the left represents just your pupils. The box and whisker plot on the right represents all 6,724 pupils who took part in the pilot. This gives you a way to compare how well your pupils did with all of those in the pilot.
For each box and whisker plot:
The horizontal line in the middle of the box represents the median score of the pupils.
The middle 50 per cent of pupils are in the box, with half of them above the horizontal line and half below.
The vertical lines above and below the box represent the top and bottom 25 per cent of pupils respectively.
The dashed green line shows you the EXS threshold. Pupils who scored above this line got EXS or above.
The dashed blue line shows you the GDS threshold. Pupils who scored above this line got GDS.

17. How did my school do compared to the other schools in the pilot?

18. Year 3 90th percentile
As well as the individual results, you can see national results on the No More Marking website here.
The above screenshot on the left shows you a selection of ten portfolios that are available to read. The screenshot on the right is the first page of the best portfolio.

19. Year 3 50th percentile
As well as the individual results, you can see national results on the No More Marking website here.
The above screenshot on the left shows you a selection of ten portfolios that are available to read. The screenshot on the right is the first page of the best portfolio.

20. Year 3 30th percentile
As well as the individual results, you can see national results on the No More Marking website here.
The above screenshot on the left shows you a selection of ten portfolios that are available to read. The screenshot on the right is the first page of the best portfolio.

21. What’s the quality of work you can expect?
As well as the individual results, you can see national results on the No More Marking website here.
The above screenshot on the left shows you a selection of ten portfolios that are available to read. The screenshot on the right is the first page of the best portfolio.

22. No More Marking Twitter: @daisychristo
Blog: