1. Systematic Reviews: The Potential of Meta-analysis ESRC Research Methods Festival Oxford 5 th July, 2012 Professor Steven Higgins Durham University s.e.higgins@durham.ac.uk

2. What is meta-analysis? A way of combining the results of quantitative research To accumulate evidence from smaller studies To compare results of similar studies - consistency To investigate patterns of association in the findings of different studies – explaining variation ‘Surveys’ research studies

3. Key points Understanding ‘effect-size’ as a common measure Why do we need meta-analysis? What are its limitations? What is its potential?

4. What is an “effect size”? Standardised way of looking at difference Different methods for calculation Binary (Risk difference, Odds ratio, Risk ratio) Continuous  Correlational (Pearson’s r)  Standardised mean difference (d, g, Δ)  Difference between control and intervention group as proportion of the dispersion of scores  Intervention group score – control group score / standard deviation of scores

5. Examples of Effect Sizes: ES = 0.2 “Equivalent to the difference in heights between 15 and 16 year old girls” 58% of control group below mean of experimental group Probability you could guess which group a person was in = 0.54 Change in the proportion above a given threshold: from 50% to 58% or from 75% to 81%

6. “Equivalent to the difference in heights between 13 and 18 year old girls” 79% of control group below mean of experimental group Probability you could guess which group a person was in = 0.66 ES = 0.8 Change in the proportion above a given threshold: from 50% to 79% or from 75% to 93%

7. The rationale for using effect sizes Traditional quantitative reviews focus on statistical significance testing Highly dependent on sample size Null finding does not carry the same “weight” as a significant finding Meta-analysis focuses on the direction and magnitude of the effects across studies From “Is there a difference?” to “How big is the difference?” and “How consistent is the difference?” Direction and magnitude represented by “effect size”

8. Meta-analysis Synthesis of quantitative data Cumulative Comparative Correlational “Surveys” educational research (Lipsey and Wilson, 2001)

9. Forest plots Effective way of presenting results Studies, effect sizes, confidence intervals Provides an overview of consistency of effects Summarises an overall effect (with confidence interval) Useful visual model of a meta-analysis

10. Anatomy of a forest plot… Studies N of study Line of no effect C.I Study effect size Pooled effect size Study effect size (with C.I.) Weighting of study in meta- analysis

11. Issues and challenges in meta-analysis Conceptual Reductionist - the answer is.42 Comparability - apples and oranges Atheoretical - ‘flat-earth’ Technical Heterogeneity Publication bias Methodological quality

12. Some recent findings from meta- analysis in education Klauer & Phye 2008 74 studies, 3,600 children - training in inductive reasoning improves academic performance (0.69) more than intelligence test performance (0.52). Gersten et al. 2009 Maths interventions for low attainers - 42 studies ES ranging from 0.21-1.56. Teaching heuristics and explicit instruction particularly beneficial. Domino 2010 31 studies / 5288 - pupils those who used manipulatives during mathematics instruction had higher mathematics achievement than students who were taught by traditional teaching methods - effect size 0.50 (CI 0.34 to 0.65)

13. Methodological heterogeneity Study design Sample characteristics Assessment (measures, timing)

15. Educational heterogeneity ‘Clinical’or ‘pedagogical’ heterogeneity Systematic variation in response to the intervention Teacher level effects Pupil level effects

16. Statistical Due to chance Unexplainable

17. Statistical methods to identify heterogeneity Presence Q statistic (Cooper & Hedges, 1994) Significance level (p-value)  2  2 Extent I 2 (Higgins & Thompson, 2002) If it exceeds 50%, it may be advisable not to combine the studies All have low power with a small number of studies (Huedo- Medina et al. 2006)

18. Exploring heterogeneity In a meta-analysis, exploring heterogeneity of effect can be as or even more important than reporting averages Exploring to what extent the variation can be explained by factors in the coding of studies (age, gender, duration of intervention etc) through regression Forming sub-groups with greater homogeneity Identifying the extent of the variation through further analysis

19. Coding for exploration Factors which may relate to variation The intervention E.g. duration, intensity, design, implementation The sample E.g. age, gender, ethnicity, particular needs The research E.g. design (RCT, quasi-experimental), quality, tests/outcomes, comparison group

20. Pooling the results In a meta-analysis, the effects found across studies are combined or ‘pooled’ to produce a weighted average effect of all the studies-the summary effect. Each study is weighted according to some measure of its importance. In most meta-analyses, this is achieved by giving a weight to each study in inverse proportion to the variance of its effect.

21. Fixed effect model The difference between the studies is due to chance Observed study effect = Fixed effect + error

22. Fixed effect model Each study is seen as being a sample from a distribution of studies, all estimating the same overall effect, but differing due to random error

23. Random effects model Assumes there are two component of variation 1. Due to differences within the studies (e.g. different design, different populations, variations in the intervention, different implementation, etc.) 2. Due to sampling error

24. Random effects model Each study is seen as representing the mean of a distribution of studies There is still a resultant overall effect size

25. “Random effects” model assumes a different underlying effect for each study. This model gives relatively more weight to smaller studies and wider confidence intervals than fixed effect models. The use of this model is recommended if there is heterogeneity between study results. Also recommended as it provides a more conservative estimate for the pooled effect. Which model?

26. Exploring heterogeneity Conceptual: are the studies sufficiently similar in terms of the intervention or treatment? Statistical: greater variation than would be predicted

27. Sensitivity analysis Provides feedback about whether assumptions and decisions made during the meta-analysis have had a major effect on the results Repeats the analysis using different assumptions (as a quality check to make sure results are consistent) e.g. Effect of including and excluding low quality studies Excluding and including outliers Undertaking fixed effect and and random-effects analyses

28. Meta-regression Examines the impact of moderator variables on pooled effect size using regression-based techniques. Estimates the extent to which covariates (e.g. age, intervention length) can explain between study heterogeneity If covariate is not associated with heterogeneity then it will not be significant in the regression

29. Interpreting review findings The standardised mean difference represents the amount of a standard deviation that the two groups differ by Can therefore be converted back to a more ‘user- friendly’ metric. For example fruit and vegetable consumption was found to have increased by a standardised mean difference of 0.65 If, on baseline fruit and vegetable consumption was measured as being 2.4 portions per day with a standard deviation of 0.9, we can say that the intervention increased consumption by 0.585 portions, or from 2.4 to nearly 3 portions per day

30. Cumulative meta-analysis Meta-analysis can have powerful applications e.g. detecting changes in paradigms Nykänen H & Koricheva J (2004) Damage-induced changes in woody plants and their effects on insect herbivore performance: a meta- analysis. Oikos, 104, 247-268. They measured the responses of woody plants to natural or simulated damage. “Cumulative meta-analyses revealed dramatic temporal changes in the magnitude and direction of the plant and herbivore responses reported during the last two decades.” Not a change in plant behaviour: a change in human understanding of them (and thus a change in measurement practices).

31. Comparative meta-analysis Theory testing Practical value Ability grouping Slavin 1990 b (secondary low attainers)-0.06 Lou et al 1996 (on low attainers)-0.12 Kulik & Kulik 1982 (secondary - all)0.10 Kulik & Kulik 1984 (elementary - all)0.07 Meta-cognition and self-regulation strategies Abrami et al. 20080.34 Haller et al. 19880.71 Klauer & Phye 20080.69 Higgins et al. 20040.62 Chiu 19980.67 Dignath et al. 20080.62

32. Summary Meta-analysis is only as good as the systematic review in which it is located Systematic bias in search strategy can lead to invalid results Sensitivity analyses are essential in order to explore the robustness of the findings Heterogeneity must be examined A statistical method for combining the quantitative results of primary studies Cumulative Comparative Meta-analysis overcomes a lack of statistical power in small primary studies Offers a more precise estimate of effect Offers a way to explore systematic variation Can settle controversies from apparently conflicting studies or generate new hypotheses

33. References, further readings and information Books and articles Borenstein, M., Hedges, L.V., Higgins, J.P.T. & Rothstein, H.R. (2009) Introduction to Meta Analysis (Statistics in Practice) Oxford: Wiley Blackwell. Chambers, E.A. (2004). An introduction to meta-analysis with articles from the Journal of Educational Research (1992-2002). Journal of Educational Research, 98, pp 35-44. Cooper, H.M. (1982) Scientific Guidelines for Conducting Integrative Research Reviews Review Of Educational Research 52; 291. *Cooper, H.M. (2009) Research Synthesis and meta-analysis: a step-by-step approach London: SAGE Publications (4th Edition). Cronbach, L. J., Ambron, S. R., Dornbusch, S. M., Hess, R.O., Hornik, R. C., Phillips, D. C., Walker, D. F., & Weiner, S. S. (1980). Toward reform of program evaluation: Aims, methods, and institutional arrangements. San Francisco, Ca.: Jossey-Bass. Glass, G.V. (2000). Meta-analysis at 25. Available at: http://glass.ed.asu.edu/gene/papers/meta25.html (accessed 9/9/08)http://glass.ed.asu.edu/gene/papers/meta25.html Lipsey, Mark W., and Wilson, David B. (2001). Practical Meta-Analysis. Applied Social Research Methods Series (Vol. 49). Thousand Oaks, CA: SAGE Publications. Torgerson, C. (2003) Systematic Reviews and Meta-Analysis (Continuum Research Methods) London: Continuum Press. Websites What is an effect size?, by Rob Coe: http://www.cemcentre.org/evidence-based-education/effect-size-resourceshttp://www.cemcentre.org/evidence-based-education/effect-size-resources The meta-analysis of research studies: http://echo.edres.org:8080/meta/http://echo.edres.org:8080/meta/ The Meta-Analysis Unit, University of Murcia: http://www.um.es/metaanalysis/http://www.um.es/metaanalysis/ The PsychWiki: Meta-analysis: http://www.psychwiki.com/wiki/Meta-analysishttp://www.psychwiki.com/wiki/Meta-analysis Meta-Analysis in Educational Research: http://www.dur.ac.uk/education/meta-ed/http://www.dur.ac.uk/education/meta-ed/

34. Acknowledgements This presentation is an outcome of the work of the ESRC-funded Researcher Development Initiative: “Training in the Quantitative synthesis of Intervention Research Findings in Education and Social Sciences” which ran from 2008-2011. The training was designed by Steve Higgins and Rob Coe (Durham University), Carole Torgerson (Birmingham University) and Mark Newman and James Thomas (Institute of Education, London University). The team acknowledges the support of Mark Lipsey, David Wilson and Herb Marsh in preparation of some of the materials, particularly Lipsey and Wilson’s (2001) “Practical Meta- analysis” and David Wilson’s slides at: http://mason.gmu.edu/~dwilsonb/ma.html (accessed 9/3/11).http://mason.gmu.edu/~dwilsonb/ma.html The materials are offered to the wider academic and educational community community under a Creative Commons licence: Creative Commons Attribution-NonCommercial- ShareAlike 3.0 Unported LicenseCreative Commons Attribution-NonCommercial- ShareAlike 3.0 Unported License You should only use the materials for educational, not-for-profit use and you should acknowledge the source in any use.