1. Effect Size Robert Coe @ProfCoe ResearchED 2013, Dulwich College, 7 Sept 2013

2. 2 The case for using effect size I.What is Effect Size? II.The case for using effect size (5 reasons) III.Problems in using effect size (7 problems) IV.Recommendations (13 recommendations) V.Further reading

3. © 2003 Robert Coe, University of Durham 3 I What is Effect Size?

4. Effect Size is the difference between the two groups, relative to the standard deviation Effect Size = Mean of experimental group – Mean of control group Standard deviation

6. © 2003 Robert Coe, University of Durham 6 II The case for using effect size measures

7. © 2003 Robert Coe, University of Durham 7 1. Effect size allows differences in uncalibrated measures to be interpreted How would you compare/interpret o 5 marks difference on a home-made test o A gain of 8 points on a standardised test o Improvement of 1 sub-level at KS2 o An effect of half a grade at GCSE Comparing each to the spread of values in the population gives it some meaning

8. © 2003 Robert Coe, University of Durham 8 2. Effect size allows the accumulation of knowledge Meta-analysis can combine results from different studies: o What is the average effect? o What factors mediate the effect? o Which interventions have the biggest effects? Small studies are worth doing because they can be added to the pool

9. Impact vs cost Cost per pupil Effect Size (months gain) £0 0 8 £1000 Meta-cognitive Peer tutoring Early Years 1-1 tuition Homework (Secondary) Mentoring Summer schools After school Aspirations Performance pay Teaching assistants Smaller classes Ability grouping Promising May be worth it Not worth it Feedback Phonics Homework (Primary) Collaborative Small gp tuition Parental involvement Individualised learning ICT Behaviour Social www.educationendowmentfoundation.org.uk/toolkit

10. © 2003 Robert Coe, University of Durham 10 3. Effect size emphasises amounts, not just statistical significance The dichotomous “significant/not” decision is almost never appropriate The size of a difference is almost always important “Significance” has many meanings, but is inevitably related to the size of the difference

11. © 2003 Robert Coe, University of Durham 11 Don’t ignore amounts

12. © 2003 Robert Coe, University of Durham 12 4. Effect size draws attention to the margin of error Statistical power is important, but often overlooked Much apparent disagreement is actually just sampling error

13. © 2003 Robert Coe, University of Durham 13 5. Effect size may help reduce reporting bias The “file-drawer” problem is alive and well Within-study reporting bias can also be a problem

14. © 2003 Robert Coe, University of Durham 14 III Problems in using effect size measures

15. Problems with effect size 1.Alternative effect size measures 2.Which standard deviation? 3.Measurement reliability 4.Non-normal distributions 5.Interpreting effects: small, medium, large 6.Incommensurability: outcomes, treatments, populations 7.Is it really an ‘effect’? © 2013 Robert Coe, University of Durham 15

16. © 2003 Robert Coe, University of Durham 16 V Further reading …

17. © 2013 Robert Coe, University of Durham 17 Coe, R. (2002) It's the effect size, stupid: what effect size is and why it is important. Paper presented at the Annual Conference of the British Educational Research Association, University of Exeter, England, 12-14 September 2002. www.cem.org/attachments/ebe/ESguide.pdf Coe, R.J. (2012) ‘Effect Size’ in J. Arthur, M. Waring, R. Coe, and L.V. Hedges (Ed.s) (2012) Research Methods and Methodologies in Education. London: Sage. An introduction, guide and tool for calculating effect sizes: www.cem.org/evidence-based-education/effect-size-resources 1 2 3

18. © 2013 Robert Coe, University of Durham 18 http://www.nfer.ac.uk/nfer/publications/SEF01/SEF01.pdf Coe, R. (2004) ‘Issues arising from the use of effect sizes in analysing and reporting research’ in I. Schagen and K. Elliot (Eds) But what does it mean? The use of effect sizes in educational research. Slough, UK: National Foundation for Educational Research. 4