Paul Montgomery Joanne Yaffe David Albright Completing a Campbell Systematic Review Society for Social Work and Research Washington, DC January 15, 2012

Mission: The Campbell Collaboration (C2) helps people make well- informed decisions by preparing, maintaining and disseminating systematic reviews in education, crime and justice, and social welfare. Who are we? The Campbell Collaboration is an international research network that produces systematic reviews of the effects of social interventions. Campbell is based on voluntary cooperation among researchers of a variety of backgrounds.

What is a systematic review? The purpose of a systematic review is to sum up the best available research on a specific question. This is done by synthesizing the results of several studies. A systematic review uses transparent procedures to find, evaluate and synthesize the results of relevant research. Procedures are explicitly defined in advance, in order to ensure that the exercise is transparent and can be replicated. This practice is also designed to minimize bias. Studies included in a review are screened for quality, so that the findings of a large number of studies can be combined. Peer review is a key part of the process; qualified independent researchers control the author's methods and results.

Evidence-based practice demands high quality evidence on which to base practice and policy decisions. It is impossible for most social work practitioners to read all relevant research on a given topic. Even if they could, many of the available studies are of low quality. Studies accessible in social work journals are biased in favor of positive findings. “Through critical exploration, evaluation, and synthesis the systematic review separates the insignificant, unsound or redundant deadwood from the salient and critical studies that are worthy of reflection.”

Additional Rationale for Systematic Reviews: Decision Making Practitioners need SRs to keep up with the literature in their field Policy-makers need them to make sensible decisions Researchers need them to identify gaps and establish protocols for future research.

Additional Rationale for Systematic Reviews: Efficiency A review is usually less costly and time-consuming than embarking on a new study (although an SR may conclude that new studies are called for). Reviews can also suggest or even establish that further research in a given area is not called for or even unethical. Continuously updated reviews can shorten the time between research discoveries and ‘real world’ implementation of effective diagnostic or treatment strategies.

Additional Rationale for Systematic Reviews: Generalizability Diversity of multiple reviewed studies provides an interpretive context not available in any one study…. studies addressing similar questions often use different eligibility criteria for participants, different definitions of the ‘problem’, different variations of an intervention, different study designs, etc.

Additional Rationale for Systematic Reviews: Examining Consistency of Relationships Are effects in the same direction, and of the same general magnitudes, given the variation in study protocols? Does consistency exist between studies of the same intervention or different dosages/intensities/classes of the same intervention?

Additional Rationale for Systematic Reviews: Examining Inconsistency of Relationships If an intervention strategy is effective in one setting and not in another, or among some people and not among others, this is important! Furthermore, whether findings from a single study stand alone for any reason (uniqueness of study population, study quality or outcome measure) should be explored…

Additional Rationale for Systematic Reviews: Power Quantitative SRs have the advantage of increased power. The advantage of increasing power is particularly relevant to conditions of relatively low event rates or when small effects are being assessed.

Additional Rationale for Systematic Reviews: Precision A measure of the likelihood of random errors in the results of a study, meta-analysis or measurement. Confidence intervals around the estimate of effect from each study are a measure of precision. The weight given to the results of each study in a meta- analysis (typically the inverse of the variance of the estimate of effect) is a measure of precision (i.e. the degree to which a study influences the overall estimate of effect in a meta-analysis is determined by the precision of its estimate of effect).

Additional Rationale for Systematic Reviews: Accuracy …or at least an improved reflection of reality…. Traditional (non-systematic) reviews can be criticized as biased and haphazard… subject to idiosyncrasies of the reviewer… SRs and meta-analyses using scientific principles which aim to reduce random and systematic errors are superior.

A Campbell systematic review must have: Clear inclusion/ exclusion criteria An explicit search strategy Systematic coding and analysis of included studies Meta-analysis (where possible)

How do Campbell systematic reviews differ from other systematic reviews? Campbell reviews must include a systematic search for unpublished reports (to avoid publication bias). Campbell reviews are usually international in scope. A protocol (project plan) for the review is developed in advance and undergoes peer review. Study inclusion and coding decisions are accomplished by at least two reviewers who work independently and compare results. Campbell reviews undergo peer review and editorial review.

The Campbell Library of Systematic Reviews Campbell reviews are published in the Campbell Library of Systematic Reviews, and are freely accessible online. Campbell systematic reviews can be updated as further relevant information emerges.

Completing a Campbell Review

Major Steps for a Campbell Systematic Review Title Registration Protocol Development Protocol Publication Review Development Review Publication Periodic Updates

Campbell Systematic Review Select a topic and formulate a focused question Register the title Prepare the protocol (Campbell Editorial Board & external peer review) Electronic publication of protocol in The Campbell Library Perform a comprehensive literature search Prepare the Review (Editorial Board & external peer review) Electronic publication of completed Review in the Campbell Library Respond to comments/criticism Keep the review up-to-date

Title Registration Reserve the topic Do your homework! Components – Title: Intervention for Problem in Population E.g.: Does Eye Movement Desensitization and Reprocessing (EMDR) for Posttraumatic Stress Disorder (PTSD) in Combat Veterans – Background and objective of the review – Define the population – Define the intervention(s) – Outcome(s) – Methodology – Review team – Roles and responsibilities – Potential conflicts of interest – Support – Funding – Preliminary timeframe

Protocol Roadmap for Review Components – Background – Objective of the review – Methods – Acknowledgements – References – Appendices – Contribution of authors – Declaration of Interest – Sources of Support

Background Description of the condition Description of the intervention How the intervention might work Why it is important to do this review

Objective of the Review Examples: The primary objective is to complete a SR of experimental and quasi-experimental studies of EMDR for combat veterans with PTSD The secondary objective is to synthesize the results of these studies to asses the effect of EMDR on reducing PTSD in combat veterans

Methods Criteria for considering studies for the review Types of studies (e.g., experimental, quasi-experimental) Types of participants (e.g., military combat veterans) Types of interventions (i.e., EMD/EMDR) Types of outcomes (e.g., level of PTSD symptoms via structured diagnostic interview or self-report questionnaire)

Methods Search methods for identification of studies Electronic searches (e.g., EMBASE, MEDLINE, PILOTS) Search terms Searching other resources (e.g., correspondence, grey literature, hand- searching)

Methods Data Collection and analysis Selection of studies (stage 1 and 2) Data extraction and management (stage 3) Assessment of risk of bias within included studies (e.g., sequence generation, incomplete outcome data) Measures of treatment effect (how we will handle dichotomous and continuous data) Unit of analysis issues Dealing with missing data and incomplete data Assessment of heterogeneity (e.g., Q-statistic, I² statistic) Assessment of publication bias

Methods Data Synthesis Subgroup analysis, moderator analysis and investigation of heterogeneity Sensitivity analysis

Appendices Examples: Screening: Stage 1 Screening: Stage 2 Extraction

Dr Paul Montgomery University of Oxford Dr Jane Dennis, School for Policy Studies University of Bristol

1. Did the review address a clearly focussed question? 2. Were the right sort of studies selected? 3. Was the search strategy explicit and comprehensive? 4. Did the reviewers assess the quality of the identified studies?

Are non-drug CBT interventions effective in resolving the sleep problems of people aged over 60?

1. Did the review address a clearly focussed question? 2. Were the right sorts of studies selected? 3. Was the search strategy explicit and comprehensive? 4. Did the reviewers assess the quality of the identified studies?

Types of studies: Randomised controlled trials Types of participants Over 60s diagnosed with sleep problems via standardised measure screened to exclude dementia, depression and sleep apnoea or secondary insomnia (sleep disturbance caused by a psychiatric or medical disorder)

Sleep onset latency (SOL) Wake after sleep onset (WASO) Total wake time (TWT) Sleep duration (total) Early morning wakening Sleep efficiency Self-report of sleep satisfaction Scales related to sleep, eg the Pittsburgh Sleep Quality Index (PSQI); Stanford Sleepiness Scale Quality of life Outcomes were divided, where possible, into immediate post-treatment, medium term (3-12 months) and long-term (more than 12 months).

1. Did the review address a clearly focussed question? 2. Were the right sort of studies selected? 3. Was the search strategy explicit and comprehensive? 4. Did the reviewers assess the quality of the identified studies?

Potential hazards: Publication Bias Tower of Babel Bias Uncritical use of electronic databases

“the tendency of investigators, reviewers and editors to differentially submit or accept manuscripts for publication on the direction or strength of the study findings.” Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A et al. Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA 1993; 269:

Prospective Registries of all trials in progress Retrospective Trial amnesty Negotiation Funnel plot

Controversial Unpublished data may not be a full or representative sample (Cook 1993) Publication is no guarantee of scientific quality (Oxman 1991) Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A et al. Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA 1993; 269: Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchison BG, Milner RA et al. Agreement among reviewers of review articles. J.Clin.Epidemiol. 1991;44:91-98.

Studies that find a treatment effect are more likely to be published in English-language journals. Opposing studies may be published in non-English-language journals. Gregoire G, Derderan F, Le Lorier J. Selecting the language of the publications included in a meta-analysis: is there a Tower of Babel Bias? J.Clin.Epidemiol. 1995;48:

Electronic databases Reference checking Hand-searching Personal communications

sleep problems the older adult psychological, physical, phototherapeutic, treatments

1. Did the review address a clearly focussed question? 2. Were the right sort of studies selected? 3. Was the search strategy explicit and comprehensive? 4. Did the reviewers assess the quality of the identified studies?

Should be routine Dependent on sort of studies included Use of rating scales with fixed cut-offs potentially misleading

1. Were the results similar from study to study? 2. What is the overall result of the review? 3. How precise are the results?

1. Were the results similar from study to study? 2. What is the overall result of the review? 3. How precise are the results?

Sources of Heterogeneity: Differences in study participants Different comparisons Different duration of follow-up Different outcome measures Differences in methodological quality

Look at plots of results Formal tests of homogeneity.

1. Were the results similar from study to study? 2. What is the overall result of the review? 3. How precise are the results?

 Estimates from individual trials vary more than can be explained by the play of chance alone  N.B. Meta-analysis should NOT overlook important material differences in subgroup response

does it makes sense to average effects from the studies?

 Qualitative v. quantitative  Qualitative – reconsider pooling  Fixed v. random effects

short term improvements in sleep quality, sleep efficiency, and wake time and to long term improvement in self reported sleep duration

Yes

Pallesen, Nordhus and Kvale (1998) Non-pharmacological interventions for insomnia in older adults….”psychological interventions included in this meta-analysis produce significant and long-lasting changes in the sleep of older insomniacs.” However, they included non-randomised trials that included people without a diagnosis of a sleep problem and trials where more than 30% of participants dropped out

However this review includes non-randomised trials with participants as young as 17 as well as people who were not insomniacs

Title Registration Protocol Development Protocol Publication Review Development Review Publication Periodic Updates

Resources Cochrane Handbook for Systematic Reviews of Interventions Cochrane Open Learning Materials net.org/openlearning/index.htm net.org/openlearning/index.htm EQUATOR Network (Enhancing the Quality and Transparency of Health Research) Centre for Reviews and Dissemination (CRD), University of York, UK David Wilson's page on "Meta-Analysis Stuff“ All can be found via: