1. The Campbell Collaborationwww.campbellcollaboration.org C2 Training: May 9 – 10, 2011 Introduction to Systematic Reviews

2. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Systematic Review Methods Systematic reviews are a form of research – secondary observations – in which studies are the unit of analysis Follow basic steps in the research process Aim to minimize bias and error – But SRs are not immune to bias and error (not a panacea)

3. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org 3 Stages of a research synthesis (Cooper, 1982) The seminal article outlining five stages of a research review is by Harris Cooper: Cooper, H.M. (1982). Scientific guidelines for conducting integrative research reviews. Review of Educational Research, 52, 291-302. Cooper, H. M. (2009). Research synthesis and meta- analysis: A step-by-step approach. Thousand Oaks, CA: Sage.

4. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org 4 Stages of a research synthesis (Cooper, 1982) Problem formulation – Clarifying your questions and writing a protocol – Set explicit inclusion/exclusion criteria Data collection – Literature search – Information-gathering from studies Data evaluation – Criteria for including and excluding studies – Assessing study quality

5. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org 5 Stages of a research synthesis (Cooper, 1982) Data analysis and interpretation – Integrating the effects from collected studies – Interpreting analysis results Report preparation – Narrative, statistics, graphs, tables

6. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org What’s required? A team with – Substantive expertise – Methodological expertise – Statistical expertise – Information retrieval expertise Time and money – SRs are labor intensive – $50-$150K depending on scope, complexity, and number of studies

7. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Protocols for SRs A detailed protocol (plan) for the SR should be developed and made available to readers (Higgins & Green, 2008; Moher et al., 2009) – Protocols increase transparency, limit ad hoc decisions The review process is iterative and plans may change during the process – The final report should document and explain changes made (deviations from the protocol)

8. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org 8 Stages of a research synthesis Most of the work involved in conducting a review is not spent in statistical analysis. The scientific contribution of the final product is dependent on all stages of the review and not just the statistical analysis stage.

9. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org 9 An example of the stages of a review We will illustrate each of the stages by considering the C2 systematic review on Multisystemic Therapy (MST): Littell, J.H., Popa, M., & Forsythe, B. (2005). Multisystemic Therapy for social, emotional, and behavioral problems in youth aged 10 – 17. (Campbell version)

10. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org 10 Stage 1 Problem Formulation

11. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Problem formulations Conceptual issues arise in attempts to combine results of studies that vary (to some degree) in their methods, treatments, samples, outcome measures – Apples, oranges, and other fruits – Parallel problems in studies of individuals (no 2 people are identical and there is no “average” person) Given variation in primary research – What should be included in a synthesis? – How should we synthesize results?

12. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org SRs vary in scope Specific, narrow questions – Useful for testing effects of specific treatments Broad, global questions – Useful for generating new knowledge Identify common elements of effective programs (Lipsey, 2008) build better intervention theories to guide program development and evaluation design (Lipsey, 1997)

13. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Scope of Systematic Reviews Not limited to questions about effects – Can address trends, epidemiology, accuracy of diagnostic and prognostic tests Not limited to randomized controlled trials or quantitative data – Qualitative synthesis (e.g., meta-ethnography, narrative analysis of qualitative research reports) – Mixed/multiple methods synthesis (e.g., Thomas, Harden, et al. on progams to combat childhood obesity)

14. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org What kinds of research questions can be asked in a systematic review? Questions about intervention effects: – What are the effects of x intervention on y outcomes for z populations/problem? – Variations on this theme (e.g., differences in effects of interventions x 1 vs x 2 )

15. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Kinds of research questions (continued) Questions about associations – How does x 1 relate to x 2 for population z? (direction and strength of correlation) – Variations on this theme (e.g., differences in relation of x 1 and x 2 between populations z 1 and z 2 ) Diagnostic/Prognostic questions – Which test (A vs. B) is a better predictor of y? – Which test (A vs. B) is a better predictor of y for z 1 vs. z 2 populations?

16. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org 16 Problem formulation To assess the impacts of MST on out-of-home living arrangements, crime and delinquency, and other behavioral and psychosocial outcomes for youth and families.

17. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org 17 Problem formulation What research evidence will be relevant to the problem or hypothesis of interest in the synthesis? – Studies where youth (age 10-17) with social, emotional, and/or behavioral problems were randomised to licensed MST programs or other conditions (usual services or alternative treatments).

18. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org 18 Problem formulation Define the (a) variables and (b) relationships of interest so that relevant and irrelevant studies can be distinguished. Multisystemic Therapy (MST) is a multi-faceted, short-term, home- and community-based intervention for families of youth with severe psychosocial and behavioral problems. Based on social ecological and family systems theories, and on research on the causes and correlates ofserious antisocial behavior in youth (Henggeler 1998, Henggeler 2002a), MST is designed to address complex psychosocial problems and provide alternatives to out-of-home placement of children and youth.

19. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Problem formulation (continued) Measures of behavioral, psychosocial, and family outcomes were examined. – Behavioral outcomes included antisocial behavior (as measured by arrest or conviction of a criminal offense), drug use (self-reports and drug tests), and school attendance. – Psychosocial outcomes included measures of psychiatric symptoms (on standardized scales),school performance (teacher reports), peer relations (self-reports and parent or teacher reports),and self esteem. – To assess the impacts of MST on out-of-home living arrangements, crime and delinquency, and other behavioral and psychosocial outcomes for youth and families.

20. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Step two: Establish study inclusion and exclusion criteria The PICOS framework: – P opulation/Participants (problems/conditions) – I nterventions (if applicable) – C omparison group (e.g., absolute vs. relative effects, counterfactual conditions) – O utcomes (primary and secondary outcomes, acceptable outcome measures) – S tudy Design (should be fit for purpose) Geographic area, time, language, other criteria

21. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Participants: Who is included in the sample of subjects? Why 10 - 17 only? Exclude special populations? Exclude studies with only males or females? Include families of these children? What is the rationale for the choice?

22. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Interventions: How do we define MST? Comparisons: What is the comparison? What kinds of programs are considered MST? What types of treatments are the “right” comparisons to MST?

23. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org What are the Outcomes? Research question states: – Placement in out-of-home living arrangements – Crime and delinquency – Behavioral outcomes – Psychosocial outcomes

24. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Study Designs: What types of study designs are relevant for our review? If the research question is the assessment of the impacts of MST on out-of-home living arrangements, crime and delinquency, and other behavioral and social outcomes for youth and families, what designs are most appropriate?

25. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org What studies are relevant? Will we attempt to generalize to all studies done? Will we look only at “high quality” studies? Will we look at both published and unpublished? Will we set a time period for the studies? Will we look at only randomized controlled trials or will we include quasi-experimental studies? Why?

26. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Study eligibility criteria Usually focus on inclusion criteria, with few exclusion criteria Studies should not be excluded based on: – Sample size (or statistical power considerations) – Publication status

27. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org What kinds of studies should be included? Garbage in, garbage out Synthesis of invalid studies produces invalid conclusions What constitutes credible evidence? – Depends on the question Need to set clear standards – Based on methodological principles – Relevant for topic and context

28. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Small group exercise: Problem formulation Form a working group of 2 – 5 people and develop a preliminary research question for a comprehensive review Elect a recorder for the group who will report back to the whole Each group will share the research question, and any challenges that they will face in studying this question See the first exercise on problem formulation