1. Meta-analysis, systematic reviews and research syntheses
© LOUIS COHEN, LAWRENCE MANION AND KEITH MORRISON
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

2. STRUCTURE OF THE CHAPTER
Meta-analysis
Systematic reviews
Methodologically inclusive research syntheses
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

3. META-ANALYSIS The analysis of other analyses.
Uses data from randomized controlled trials.
Aggregates and combines the results of comparable studies into a coherent account to discover main effects.
Often uses statistical processes.
Looks at effect size, not only statistical significance.
Combines the results of small-scale studies.
Uses transparent means to draw conclusions.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

4. META-ANALYSIS AND RESEARCH SYNTHESIS
Meta-analysis involves rigorous statistical integration of findings reported across a number of primary research studies.
Meta-analysts employ explicit protocols to enhance consistency
Meta-analyses facilitate comparisons across quantitative studies by bringing them on a common metric (‘effect size’), but they are unsuitable for synthesizing qualitative research.
Research syntheses are often used for qualitative research.
‘Research synthesis’ can draw together studies in a range of styles into a single review, seeking aims to produce new knowledge by making explicit connections and relations that were previously invisible between individual reports.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

5. WORKING WITH META-ANALYSIS
Formulating a problem
Searching for relevant literature
Extracting relevant information from selected studies
Integrating findings across studies
Presenting the findings in a report
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

6. AN EIGHT-STEP PROCESS FOR META-ANALYSIS1
Identify the variables for focus (dependent and independent) 2
Identify all the studies which feature the variables in which the researcher is interested and which meet selection criteria 3
Code each study for those characteristics that might be predictors of outcomes and effect sizes (e.g. age, gender, ethnicity) 4
Estimate the effect sizes (i.e. a common metric) for each pair of variables (dependent and independent variable), weighting the effect-size by the sample size
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

7. AN EIGHT-STEP PROCESS FOR META-ANALYSIS5
Calculate the mean and the standard deviation of effect-sizes across the studies, i.e. the variance across the studies 6
Determine the effects of sampling errors, measurement errors and range of restriction 7
If a large proportion of the variance is attributable to the issues in Step 6, then the average effect-size may be an accurate estimate of relationships between variables 8
If a large proportion of the variance is not attributable to the issues in Step 6, review those characteristics of interest which correlate with the study effects
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

8. EFFECT SIZE Effect size is calculated as:
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

9. CONCERNS ABOUT META-ANALYSIS
Lacks comprehensiveness, as they make selective use of subsets of studies.
Misrepresents research findings.
Relies too heavily on significance tests.
Understates effect sizes as a way of supporting hypotheses.
Fails to recognize that sampling error can play a part in creating variations in findings.
Overlooks differing/conflicting research findings.
Published research is favoured over unpublished research.
Interaction effects overlooked in favour of main effects.
Unclear how it differentiated between ‘good’ and ‘bad’ research, e.g. between rigorous and poorly constructed research.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

10. CONCERNS ABOUT META-ANALYSIS
Difficult to draw logical conclusions from studies that use different interventions, measurements, definitions of variables, and participants.
Results from poorly designed studies take their place alongside results from higher-quality studies.
Multiple results from a single study are used, making the overall meta-analysis appear more reliable than it is, since the results are not independent.
Dangerous consequences because its apparent objectivity and precision may disguise procedural invalidity in the studies.
Risks trying to synthesize studies which are insufficiently similar to each other to permit this.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

11. CONCERNS ABOUT META-ANALYSIS
Fails to examine critically the evidence, methods and conclusions of previous reviews.
Overlooks the extent to which findings from research are mediated by sample characteristics.
Overlooks the importance of intervening variables in research.
Unreplicable because procedures for integrating the research findings not made explicit.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

12. RESEARCH SYNTHESES AND SYSTEMATIC REVIEWS
Include studies that are not solely randomized controlled trials.
Apply explicit protocols and rigorous criteria for searching for and selecting relevant primary, usually empirical, studies, to remove bias; standards for acceptable methodological rigour; relevance to the topic in question; scope of the studies included; adoption of a consistent and clearly stated approach to combining information from across different studies; drawing careful, relevant conclusions and recommendations.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

13. RESEARCH SYNTHESES AND SYSTEMATIC REVIEWS
Combine qualitative and quantitative studies.
Follow a standard set of stages.
Are accountable, able to be replicated and updated.
Must be relevant and useful to users.
Must answer specific research questions.
Are evidence-based.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

14. RESEARCH SYNTHESES AND SYSTEMATIC REVIEWS
Clear, transparent criteria for:
inclusion and exclusion of studies
standards for acceptable methodological rigour
relevance to the topic in question
scope of the studies included
team approaches to reviewing in order to reduce bias
adoption of a consistent and clearly stated approach to combining information from across different studies
careful, relevant conclusions and recommendations
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

15. SEQUENCE FOR CONDUCTING SYSTEMATIC REVIEWS1
Formulate the problem and kinds of relevant research evidence to answer the research question or hypothesis 2
Examine the research designs in the studies used, the treatment of the main effects, the kind of research, the conceptual relevance of the studies, i.e. screen the studies. 3
Search the literature 4
Gather information from studies (including developing and using a coding guide, identifying predictor and outcome variables, research designs used, sampling, context, statistics used, effect sizes)
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

16. SEQUENCE FOR CONDUCTING SYSTEMATIC REVIEWS5
Evaluate the quality of studies (decide which studies to include and exclude, identify problems in the research studies used, evaluate the suitability of the research design for the research synthesis, evaluate the quality and rigour of the research) 6
Analyze and integrate the outcomes of studies 7
Interpret the evidence 8
Present the results
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

17. METHODOLOGICALLY INCLUSIVE RESEARCH SYNTHESES
Suri’s (2014) three general guiding principles for a quality research synthesis
Informed subjectivity and reflexivity
Purposefully informed selective inclusivity
Audience-appropriate transparency
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

18. SURI’S (2014) SIX PHASES FOR CONDUCTING A RESEARCH SYNTHESIS
Identify an appropriate epistemological orientation 2
Identify an appropriate purpose 3
Search for relevant literature 4
Evaluate, interpret and distil evidence from selected reports 5
Construct connected understandings 6
Communicate with an audience
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors