1. Using meta-analyses in your literature review BERA Doctoral Workshop 3rd September 2008 Professor Steven Higgins Durham University s.e.higgins@durham.ac.uk

2. Acknowledgements This presentation is an outcome of the work of the ESRC-funded Researcher Development Initiative: “Training in the Quantitative synthesis of Intervention Research Findings in Education and Social Sciences” which ran from 2008- 2011. The training was designed by Steve Higgins and Rob Coe (Durham University), Carole Torgerson (Birmingham University) and Mark Newman and James Thomas, Institute of Education, London University. The team acknowledges the support of Mark Lipsey, David Wilson and Herb Marsh in preparation of some of the materials, particularly Lipsey and Wilson’s (2001) “Practical Meta-analysis” and David Wilson’s slides at: http://mason.gmu.edu/~dwilsonb/ma.html (accessed 9/3/11). http://mason.gmu.edu/~dwilsonb/ma.html The materials are offered to the wider academic and educational community community under a Creative Commons licence: Creative Commons Attribution- NonCommercial-ShareAlike 3.0 Unported LicenseCreative Commons Attribution- NonCommercial-ShareAlike 3.0 Unported License You should only use the materials for educational, not-for-profit use and you should acknowledge the source in any use.

3. Aims To support understanding of meta-analysis of intervention research findings in education; To extend understanding of reviewing quantitative research literature; To describe the techniques and principles of meta-analysis involved to support understanding of its benefits and limitations; To provide references and examples to support further work.

4. ESRC Researcher Development Initiative Quantitative synthesis of intervention research findings in education –Collaboration between Durham University York University Institute of Education, London

5. Why review? Ask the person next to you what the purpose of the literature review is in their thesis See how many different purposes you can think of Join another pair and identify which are the 3 you think are the most important

6. Why review? Summarise existing knowledge What we know, and how we know it For what purpose? –Expectation –Scenery –State of the art (summary) –Positioning (conceptual) –Progressing knowledge (logic)

7. The PhD literature review Narrative summary of the area Grand tour of the concepts and terminology Synthesis of empirical findings Background to the study

8. A systematic review is usually more comprehensive; is normally less biased, being the work of more than one reviewer; is transparent and replicable (Andrews, 2005)

9. Examples of systematic reviews EPPI Centre –UK based - wide range of educational topics The Campbell Collaboration –5 education reviews Best Evidence Encyclopedia –John’s Hopkins’ - aimed at practice

10. Systematic reviewing Key question Search protocol Inclusion/exclusion criteria Coding and Mapping In-depth review (sub-question) Techniques for systematic synthesis

11. Systematic reviews Research and policy Specific reviews to answer particular questions –What works? - impact and effectiveness research with a tendency to focus on quantitative and experimental designs

12. Literature reviewing - conceptual relations Systematic review Meta-analysis Narrative review

13. Meta-analysis Synthesis of quantitative data –Cumulative –Comparative –Correlational “Surveys” educational research (Lipsey and Wilson, 2001)

14. Origins 1952: Hans J. Eysenck concluded that there were no favorable effects of psychotherapy, starting a raging debate which 25 years of evaluation research and hundreds of studies failed to resolve 1978: To proved Eysenck wrong, Gene V. Glass statistically aggregated the findings of 375 psychotherapy outcome studies Glass (and colleague Smith) concluded that psychotherapy did indeed work - “the typical therapy trial raised the treatment group to a level about two-thirds of a standard deviation on average above untreated controls; the average person received therapy finished the experiment in a position that exceeded the 75th percentile in the control group on whatever outcome measure happened to be taken” (Glass, 2000). Glass called the method “meta-analysis” ( adapted from Lipsey & Wilson, 2001)

15. Historical background Underpinning ideas can be identified earlier: –K. Pearson (1904) Averaged correlations for typhoid mortality after inoculation across 5 samples –R. A. Fisher (1944) “When a number of quite independent tests of significance have been made … although few or none can be claimed individually as significant, yet the aggregate gives an impression that the probabilities are on the whole lower than would often have been obtained by chance” (p. 99). Source of the idea of cumulating probability values –W. G. Cochran (1953) Discusses a method of averaging means across independent studies Set out much of the statistical foundation for meta-analysis (e.g., Inverse variance weighting and homogeneity testing) ( adapted from Lipsey & Wilson, 2001)

16. Significance versus effect size Traditional test is of statistical ‘significance’ The difference is unlikely to have occurred by chance –However it may not be: Large Important, or even Educationally ‘significant’

17. The rationale for using effect sizes Traditional reviews focus on statistical significance testing –Highly dependent on sample size –Null finding does not carry the same “weight” as a significant finding Meta-analysis focuses on the direction and magnitude of the effects across studies –From “Is there a difference?” to “How big is the difference?” –Direction and magnitude represented by “effect size”

18. Comparison of impact Same AND different measures Significance vs effect size –Does it work? vs How well does it work? Effect size

19. Standardised way of looking at gain scores Different methods for calculation Experimental group mean - Control mean/ Standard deviation ‘Effect size’

20. What is “effect size”? Standardised way of looking at difference –Different methods for calculation Odds Ratio Correlational (Pearson’s r) Standardised mean difference –Difference between control and intervention group as proportion of the dispersion of scores

21. Calculating effect size Control group gain minus experimental group gain divided by the standard deviation of the groups

22. Effect size and impact From: Marzano, R. J. (1998) A Theory-Based Meta-Analysis of Research on Instruction. Aurora, Colorado, Mid-continent Regional Educational Laboratory. Available at: http://www.mcrel.org:80/topics/products/83/ (accessed 2/9/08).

23. Relative effects - average is about 0.37 - 0.4 (Sipe and Curlette, 1997; Hattie, Biggs and Purdie, 1996) Doing something different makes a difference Visualising the difference Interpreting effect sizes

24. 0.1 = percentile gain of 6 points ie a class ranked 50th in a league table of 100 schools would move from 50th to about 44th place 0.5 = percentile gain of 20 points ie move from 50th to 30th place 1.0 = percentile gain of 34 points ie move from 50th to 16th place How much is the impact?

25. 0.2 “small” = difference in height between 15-16 year olds 0.5 “medium” = difference in height between 14 and 18 year olds 0.8 “large” = difference in height between 13 and 18 year olds Other interpretations Cohen 1969

26. Meta-analysis Key question Search protocol Inclusion/exclusion criteria Coding Statistical exploration of findings –Mean –Distribution –Sources of variance

28. Some findings from meta- analysis Pearson et al. 2005 20 research articles, 89 effects ‘related to digital tools and learning environments to enhance literacy acquisition’. Weighted effect size of 0.489 indicating technology can have a positive impact on reading comprehension Bernard et al. 2004 Distance education and classroom instruction - 232 studies, 688 effects - wide range of effects (‘heterogeneity’); asynchronous DE more effective than synchronous

29. More findings Hattie and Timperley, 2007 ‘The Power of Feedback’, synthesis of other meta-analyses on feedback to provide a conceptual review 196 studies, 6972 effects - average effect of feedback on learning 0.79

30. Formative assessment CASE (Cognitive Acceleration Through Science Education) Individualised instruction ICT Homework Direct instruction Rank (or guess) some effect sizes…

31. 1. 04 CASE (Cognitive Acceleration Through Science Education) (Boys science GCSE - Adey & Shayer, 1991) 0.6 Direct instruction (Sipe & Curlette, 1997) 0.43 Homework (Hattie, 1999) 0.32 Formative assessment (KMOFAP) 0.31 ICT (Hattie, 1999) 0.1 Individualised instruction (Hattie, 1999) Rank order of effect sizes

32. ‘Super-syntheses’ Syntheses of meta-analyses Relative effects of different interventions Assumes variation evens out across studies with a large enough dataset (Marzano/Hattie) or attempts to control for the variation statistically (Sipe & Curlette)

33. Synthesis of study skills interventions Meta-analysis of 51 studies of study skills interventions. Categorised the inverventions using the SOLO model (Biggs & Collis, 1982), classified studies into four hierarchical levels of structural complexity and as either ‘near’ or ‘far’ transfer. The results support situated cognition, and that training for other than simple mnemonic tasks should be in context, use tasks within the same domain as the target content, and promote a high degree of learner activity and metacognitive awareness. (average effect 0.4) Hattie Biggs and Purdie, 1996

34. Sipe and Curlette, 1997 “A metasynthesis of factors relating to educational achievement” - testing Walberg’s ‘educational productivity’ model - synthesis of 103 meta- analyses

35. ‘Theory driven’ Self system - metacognition - cognition/ knowledge Self - 0.74 Metacogntive 0.72 Cognitive 0.55 Marzano, 1998

36. Discussion Work with a colleague to put the statements in order of how comparable you think the research findings are Join another pair (or pairs) and decide how comfortable would you be with comparing the findings

37. Issues and challenges in meta-analysis Conceptual –Reductionist - the answer is 42 –Comparability - apples and oranges –Atheoretical - ‘flat-earth’ Technical –Heterogeneity –Publication bias –Methodological quality

38. Reductionist or ‘flat earth’ critique The “flat earth” criticism is based on Lee Cronbach’s assertion that a meta-analysis looks at the “big picture” and provides only a crude average. According to Cronbach, “… some of our colleagues are beginning to sound like a Flat Earth Society. They tell us that the world is essentially simple: most social phenomena are adequately described by linear relations; one- parameter scaling can discover coherent variables independent of culture and population; and inconsistencies among studies of the same kind will vanish if we but amalgamate a sufficient number of studies…The Flat Earth folk seek to bury any complex hypothesis with an empirical bulldozer…” (Cronbach, 1982, in Glass, 2000).

39. Comparability Apples and oranges –Same test –Different measures of the same construct –Different measures of different constructs –What question are you trying to answer? –How strong is the evidence for this? “Of course it mixes apples and oranges; in the study of fruit, nothing else is sensible; comparing apples and oranges is the only endeavor worthy of true scientists; comparing apples to apples is trivial” (Glass, 2000).

40. Empirical not theoretical? What is your starting point? Conceptual/ theoretical critique –Marzano –Hattie –Sipe and Curlette

41. Technical issues Interventions Publication bias Methodological quality Sample size Homogeneity/ heterogeneity

42. Interventions “Super-realisation bias” (Cronbach & al. 1980) –Small-scale interventions tend to get larger effects –Enthusiasm, attention to detail, quality of personal relationships

43. Publication bias Statistically significant (positive) findings Smaller studies need larger effect size to reach significance Larger effects –‘Funnel plot’ sometimes used to explore this Scatterplot of the effects from individual studies (horizontal axis) against a study size (vertical axis)

45. Methodological quality Traditional reviews privilege methodological rigour –Low quality studies higher effect sizes (Hattie Biggs & Purdie, 1996) –No difference (Marzano, 1998) –High quality studies, higher effect sizes (Lipsey & Wilson, 1993) Depends on your definition of quality

47. Sample size “Median effect sizes for studies with sample sizes less than 250 were two to three times as large as those of larger studies.” (Slavin & Smith, 2008)

48. Heterogeneity Variation in effect sizes Investigate to find clusters (moderator variables) Assumption that the effect will be consistent

49. Questions and reactions With a colleague see if you can identify a question arising from the presentation so far What is your reaction to the technique How useful is it –Generally –To your own work?

50. Strengths of Meta-Analysis Uses explicit rules to synthesise research findings Can find relationships across studies which may not emerge in qualitative reviews Does not (usually) exclude studies for methodological quality to the same degree as traditional methods Statistical data used to determine whether relationships between constructs need clarifying Can cope with large numbers of studies which would overwhelm traditional methods of review

51. Summary “Replicable and defensible” method for synthesizing findings across studies (Lipsey & Wilson, 2001) Identifies gaps in the literature, providing a sound basis for further research Indicates the need for replication in education Facilitates identification of patterns in the accumulating results of individual evaluations Provides a frame for theoretical critique

52. Other approaches to synthesis Narrative Quantitative (meta-analysis) Best-evidence synthesis (Slavin) Realist (Pawson) Meta-ethnography (Noblitt & Hare) Thematic synthesis (Thomas & Harden) Grounded theory

53. Suggestions Be explicit about your rationale Be systematic (or at least methodical) Be transparent Describe Analyse (content and methodology) Synthesise

54. A (narrative) metaphor… Literature review as rhetoric An act of persuasion Introduce your study…

55. Some useful websites EPPI, Institute of Education, London http://eppi.ioe.ac.uk/ The Campbell Collaboration http://www.campbellcollaboration.org/ Best Evidence Encyclopedia, Johns Hopkins http://www.bestevidence.org/ Best Evidence Synthesis (BES), NZ http://www.educationcounts.govt.nz/themes/BES Institute for Effective Education (York) http://www.york.ac.uk/iee/research/#reviews

56. Further training ESRC RDI in quantitative synthesis –One day training sessions Introduction (for interpretation) Methods Training (for application) Issues Seminars (methodological issues) –Durham, London, Edinburgh, Bristol, Belfast, York s.e.higgins@durham.ac.uk

57. References Bernard, R.M., Abrami, P.C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., Wallet, P.A., Fiset, M.,& Huang, B. (2004) How Does Distance Education Compare with Classroom Instruction? A Meta-Analysis of the Empirical Literature Review of Educational Research, 74. 3, (Autumn, 2004), pp. 379-439. Chambers, E.A. (2004). An introduction to meta-analysis with articles from the Journal of Educational Research (1992- 2002). Journal of Educational Research, 98, pp 35-44. Cronbach, L. J., Ambron, S. R., Dornbusch, S. M., Hess, R.O., Hornik, R. C., Phillips, D. C., Walker, D. F., & Weiner, S. S. (1980). Toward reform of program evaluation: Aims, methods, and institutional arrangements. San Francisco, Ca.: Jossey-Bass. Glass, G.V. (2000). Meta-analysis at 25. Available at: http://glass.ed.asu.edu/gene/papers/meta25.html (accessed 9/9/08)http://glass.ed.asu.edu/gene/papers/meta25.html Hattie, J. A. (1992). Measuring the effects of schooling. Journal of Education, 36, pp 5-13 Hattie, J., Biggs, J. and Purdie, N. (1996) Effects of Learning Skills Interventions on Student Learning: A Meta-analysis Review of Educational Research 66.2 pp 99-136. Hattie, J.A. (1987) Identifying the salient facets of a model of student learning: a synthesis of meta-analyses International Journal of Educational Research, 11 pp 187- 212. Hattie, J. & Timperley, H. (2007) The Power of Feedback Review of Educational Research 77. 1, pp. 81–112. Lipsey, Mark W., and Wilson, David B. (2001). Practical Meta-Analysis. Applied Social Research Methods Series (Vol. 49). Thousand Oaks, CA: SAGE Publications. Marzano, R. J. (1998) A Theory-Based Meta-Analysis of Research on Instruction. Aurora, Colorado, Mid-continent Regional Educational Laboratory. Available at: http://www.mcrel.org:80/topics/products/83/ (accessed 2/9/08).http://www.mcrel.org:80/topics/products/83/ Pearson, D.P., Ferdig, R.E., Blomeyer, R.L. & Moran, J. (2005) The Effects of Technology on Reading Performance in the Middle-School Grades: A Meta-Analysis With Recommendations for Policy Naperville, Il: University of Illinois/North Central Regional Educational Laboratory. Sipe, T. & Curlette, W.L. (1997) A Meta-Synthesis Of Factors Related To Educational Achievement: A Methodological Approach To Summarizing And Synthesizing Meta-Analyses International Journal of Educational Research 25. 7. pp. 583-698. Slavin, R.E. and Smith, D. (2008) Effects of Sample Size on Effect Size in Systematic Reviews in Education Paper presented at the annual meetings of the Society for Research on Effective Education, Crystal City, Virginia, March 3-4, 2008.

58. Acknowledgements This presentation is an outcome of the work of the ESRC-funded Researcher Development Initiative: “Training in the Quantitative synthesis of Intervention Research Findings in Education and Social Sciences” which ran from 2008- 2011. The training was designed by Steve Higgins and Rob Coe (Durham University), Carole Torgerson (Birmingham University) and Mark Newman and James Thomas, Institute of Education, London University. The team acknowledges the support of Mark Lipsey, David Wilson and Herb Marsh in preparation of some of the materials, particularly Lipsey and Wilson’s (2001) “Practical Meta-analysis” and David Wilson’s slides at: http://mason.gmu.edu/~dwilsonb/ma.html (accessed 9/3/11). http://mason.gmu.edu/~dwilsonb/ma.html The materials are offered to the wider academic and educational community community under a Creative Commons licence: Creative Commons Attribution- NonCommercial-ShareAlike 3.0 Unported LicenseCreative Commons Attribution- NonCommercial-ShareAlike 3.0 Unported License You should only use the materials for educational, not-for-profit use and you should acknowledge the source in any use.