Systematic Reviews and Meta-Analysis Methodologies for a new era summer school School of Applied Social Studies, University College Cork 20 June 2011 Dr Paul Montgomery

Aims 1) Discuss the advantages and main features of systematic reviews 2) Introduce basic principles of meta- analysis Course feedback

The Problem Millions of articles published in thousands of journals each year Practitioners and researchers are busy Subjective summaries may misrepresent research

Reviews Systematic Reviews l Aim to answer specific questions, reduce uncertainty, identify outstanding questions l Common methods include narrative synthesis, meta-analysis (meta-regression) Traditional ‘journalistic’ reviews l Aim to persuade, draw attention to a topic, synthesise information, etc. l Narrative synthesis most common

Systematic Review: “the application of scientific strategies that limit bias to the systematic assembly, critical appraisal, and synthesis of all relevant studies on a specific topic." Cook DJ, Sakett DL, Spitzer WO. Methodological guidelines for systematic reviews of randomized contro trials in health care from the Potsdam Consultation on Meta-Analysis. J. Clin. Epidemiol. 1995;48:167-71

Systematic Reviews Clear Question l Define the population, problem, intervention, alternative interventions, and outcomes Replicable Method l Search strategy l Inclusion criteria l Analytical strategy Transparent Process

Advantages Explicit methods limit bias in identifying and rejecting studies Information can be understood quickly Reduced delay between discoveries and implementation Results can be formally compared Heterogeneity can be identified and new hypotheses generated Quantitative reviews increase precision

Producers Cochrane Campbell EPPI DARE NICE Interested practitioners/ academics

Cochrane Review Process Register titles and check for overlap Protocols developed and peer reviewed Searches performed widely on all main databases, grey literature searches, personal contacts Abstracts reviewed by two authors Data collected and trial quality assessed Data synthesis and analysis Write-up Reviewed by Cochrane/ Campbell editors, then peer reviewed

Systematic reviews Key components: 1. Ask a good question 2. Identify studies 3. Extract data 4. Synthesise data 5. Interpreting the results

Who Should Review? “Experts, who have been steeped in a subject for years and know what the answer ‘ought’ to be, are less able to produce an objective review of the literature in their subject than non-experts. This would be of little consequence if experts' opinions could be relied on to be congruent with the results of independent systematic reviews, but they cannot.” (Trisha Greenhalgh)

PICO Mad-libs For P ____________ does I ____________ compared to C ____________ improve/reduce O ____________ ?

Highly Sensitive Search Electronic Searches Databases/Indexes Additional Electronic Searches Hand Searches Personal Contacts

Electronic databases: BA Medline PsycInfo Medline covers 23% of the core 505 ‘psychiatric’ journals, plus most of the major biomedical journals. Biological Abstracts covers 48%, plus lots of life sciences stuff. Embase covers 67% plus lots of European journals that Medline misses. Embase PsycInfo covers 73% and has a psychological focus Searching PsycLit and Embase will cover 92% of the core 505 ‘psychiatric’ journals.

Electronic Searches Sensitivity vs. Specificity Even if 2 terms and 3 databases return almost all literature on a subject, the goal of a systematic review is to find everything.

Electronic Searches Specific Authors Reverse Citation Agencies / Non-Profits Funding Bodies Academic Groups / Research Centers Google

Additional Searches Previous Reviews Bibliographies of Related Articles Hand Search Journals (that aren’t indexed) Conference Reports (many are electronically published)

Personal Communication Call or Authors Attend Conferences Write to: Agencies / Non-Profits Providers / Manufacturers / Distributors Funding Bodies Academic Groups / Research Centers

Questions to Ask Which programs will be studied? Compared to what? What study designs are acceptable? What must a study measure? How must it be measured? Must researchers be blind at allocation, during the trial, etc? How will dropouts be handled? What about missing data?

Inclusion and Exclusion Types of studies Types of studies Types of participants Types of participants Types of comparisons Types of comparisons Specify: Specify: Types of outcomes Multiplicity (time, comparisons, measures, statistics)

Transparency Be clear about all definitions, searches, inclusion and exclusion criteria, etc. Report ongoing trials List excluded studies, particularly if: l The trials contain valuable information l Exclusion was a close call l You discovered something about a trial

Evaluating a Review Even if a review is ‘systematic’ it may not be well-conducted. How do we tell the difference? Even if a review is ‘systematic’ it may not be well-conducted. How do we tell the difference?

Validity 1.Did the review address a clearly focussed question? 2.Were the right sort of studies selected? 3.Was the search strategy explicit and comprehensive? 4.Did the reviewers assess the quality of the identified studies?

Importance: 1.Were the results similar from study to study? 2.What is the overall result of the review? 3.How precise are the results?

Potential Sources of Bias Describe aspects of study design that might have influenced the magnitude or direction of results Use of rating scales with fixed cut-offs potentially misleading Consider external validity

Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta- analysis. JAMA 1999; 282:

Tower of Babel Studies that find a treatment effect are more likely to be published in English- language journals. Opposing studies may be published in non-English-language journals. Gregoire G, Derderan F, Le Lorier J. Selecting the language of the publications included in a meta-analysis: is there a Tower of Babel Bias? J.Clin.Epidemiol. 1995;48:

Publication Bias “the tendency of investigators, reviewers and editors to differentially submit or accept manuscripts for publication on the direction or strength of the study findings.” Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A et al. Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA 1993; 269:

Unpublished data Controversial Unpublished data may not be a full or representative sample (Cook 1993) Publication is no guarantee of scientific quality (Oxman 1991) Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A et al. Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA 1993; 269: Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchison BG, Milner RA et al. Agreement among reviewers of review articles. J.Clin.Epidemiol. 1991;44:91-98.

Meta-analyses: “A systematic review that employs statistical methods to combine and summarise the results of several studies.” Cook DJ, Sakett DL, Spitzer WO. Methodological guidelines for systematic reviews of randomized contro trials in health care from the Potsdam Consultation on Meta-Analysis. J. Clin. Epidemiol. 1995;48:167-71

Summarising trials Reviews Systematic reviews Meta-analyses

Mathematically combine the results of different studies For dichotomous or continuous outcomes From analytical (treatment) or observational (aetiology, diagnosis, prognosis) studies ‘Weighted’ by study size (usually 1/se 2 ) and/or quality

Benefits of meta-analysis: 1.To increase statistical power for primary end points and for subgroups. 2.To improve estimates of effect size. 3.To resolve uncertainty when reports disagree 4.To answer questions not posed at the start of individual trials. Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC. Meta-analyses of randomized controlled trials. N.Engl.J.Med. 1987;316:

Outcome Measures Continuous / Dichotomous (/ Ordinal) Objective / Subjective

Meta-analysis Some outcomes are measured on scales – e.g. depression or continuously e.g. sleep minutes Continuous outcomes can be calculated using the scale on which they were measured (WMD) If changes in depression are measured on different scales it is still possible to combined them but on a standardised scale

Meta-analysis Alternatively we might be interested in binary data - two mutually exclusive states Dead/alive; hospitalised/not hospitalised These data will be measured in a different way to continuous (scale) data Reported as ‘event rates’

Meta-analysis Central Tendency: l Mean (Cohen’s d, Hedges’s g) l Odds Ratio / Relative Risk / Rate Ratio Variance (Confidence Interval) Clinical Significance (NNT/NNH) Heterogeneity (I 2, Q, Chi 2 )

Dichotomous Outcomes Odds are calculated by dividing the number of events by non-events (ie clients experiencing the event divided by clients not experiencing an event) Risk/Rate is more widely reported in reviews as it tends to be easier to communicate

Weighting Some studies contribute more weight to the ‘average’ result than do others The more precise the effect estimate, the more weight is given Wide variation is sometimes associated with small studies

Weighting Clinical trials are rarely conducted according to identical protocols Severity of the problem, intensity of the intervention, duration, setting of trial, age may account for differences in response

Apples and oranges? Sources of Heterogeneity: l Study participants l Comparisons l Intervention design l Delivery l Duration of follow-up l Outcome measures l Methods

Heterogeneity  Estimates from individual trials vary more than can be explained by the play of chance alone  N.B. Meta-analysis should NOT overlook important material differences in subgroup response

Heterogeneity – approaches  Qualitative v. quantitative  Qualitative – reconsider pooling  Does it makes sense to average effects from the studies?  Fixed v. random effects

Subgroup Analysis If together there is excessive variation, when analysed separately there is a uniform response to treatment in each subgroup Hypothesis generating

Sensitivity analysis: Sensitivity analyses investigate how the conclusions of a review change when one or more of the decisions or assumptions are altered.

Testing for heterogeneity Look at plots of results Formal tests of homogeneity l I 2 l Q l Chi 2 Assess qualitative differences in study design or implementation.

Anxiety (Self-Rated Symptoms) at Post-Treatment

Institutionalisation (RR<1 favours home visits)

Mortality: Trim and Fill (missing studies shown, 1 trimmed)

Anxiety (Self-Rated Symptoms) at Post-Treatment

Mortality by age: Meta-regression

Anxiety (Self-Rated Symptoms) at Post-Treatment: Number of contacts with researchers and clinicians Point Estimate SEP QdfP Slope0.03<0.01<0.001Model12.931<0.001 Intercept <0.01Residual Total

Limitations Junk-In, Junk-Out The results of large trials sometimes differ Chance Events: Aggregation and Disaggregation

Conclusion Systematic reviews seek to reduce bias and improve the reliability and accuracy of the conclusions. Meta-analysis is a powerful research tool, but it should be conducted only in the context of a systematic review, and it has important limitations.