1. EPI-214: Lecture 1 Designing a Systematic Review (Meta-analysis)
Dejana Braithwaite
Assistant Professor
UCSF Department of Epidemiology and Biostatistics

2. 8 steps of a systematic review
1 Formulate research question
Lecture 1
2 Develop review protocol
3 Initiate search strategy
4 Apply inclusion /exclusion criteria
5 Quality appraisal
6 Data abstraction
7 Analysis
Lectures 1, 2 & 3
8 Interpret findings
Bent et al. 2004

3. What’s a Systematic Review?
“A review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review.”
Cochrane Collaboration

4. …and meta-analysis?
Statistical combination of >= 2 studies to produce single estimate of effect of exposure
NOTE: meta-analysis is simply the statistical combination of findings; it should not be considered as a type of review

5. Rapid growth Pubmed search up to 3/29/2010:
Systematic review >1.5 million hits
Meta-analysis >40,000 hits
up to 1/1/1990:
Systematic review ,994 hits
Meta-analysis

6. The Cochrane Collaboration International systematic review initiative
Archie Cochrane’s vision led to the opening of the first Cochrane centre (in Oxford, UK) in 1992 and the founding of the Cochrane Collaboration in 1993
The Cochrane Collaboration is a group of over 27,000 volunteers in more than 90 countries who review the effects of health care interventions tested in biomedical randomized controlled trials.[3] A few more recent reviews have also studied the results of non-randomized, observational studies. The results of these systematic reviews are published as "Cochrane Reviews" in the Cochrane Library.
Source:

7. Meta-analysis news coverage
Sattar N, et al. Statins and risk of incident diabetes: a collaborative meta-analysis of randomised statin trials. Lancet Feb 27;375(9716): Epub 2010 Feb 16.

9. Systematic Review in context
Traditional review
Systematic
Review
Meta-analysis
Author
An individual
A team
Search strategy
Individual
Based on a protocol
Summary Conclusion
Author’s judgement
Can be qualitative
Summary statistical techniques

10. Narrative review Uses informal, unsystematic and subjective methods
Searching quality and synthesis not described
Disadvantage
may have preconceived biases and may overestimate value of some studies

11. Systematic review
Driven by evidence-based medicine movement and Cochrane collaboration
Advantages:
Reduces bias
Replicable
Resolves controversy between conflicting findings
Provides reliable basis for decision making

12. Use of meta-analysis as a prelude to clinical trials
Define pre-trial expected effect sizes
sample size estimation
Determine effect estimates in key subgroups (e.g. based on gender, race/ethnicity or age)
Identify sources of heterogeneity in prior studies
Addressing these sources in design phase of new trial
Determine effect estimates in key subgroups (e.g. based on gender, race/ethnicity or age)
Identify (through meta-regression) the sources of heterogeneity in prior studies

13. Use of meta-analysis in study designs that are not clinical trials
Observational studies
(e.g. case control, cohorts, cross-sectional prevalence studies, etc.)
Studies evaluating diagnostic tests (sensitivity, specificity, predictive value)
“IPD” = individual patient data studies
Qualitative studies (meta-ethnography)
Although meta-analysis is most widely known in biomedical settings for its application to clinical trials, the technique can also be used to synthesize/analyze other study designs:

14. Meta-analyses
IPD*
Systematic reviews
*IPD= individual participant data

15. Resources required for systematic reviewing
Can be time consuming
Team science (to reduce bias)
Bibliographic software (e.g. Endnote)
Statistical software (if appropriate)

16. Formulate research question
Step 1
Formulate research question

17. FINER criteria for research question
Feasible
Interesting
Novel
Ethical
Relevant
Usually the best if systematic reviews and meta-analyses are conducted by teams that include investigators with both methodologic and content expertise.
Hulley S, et al Designing Clinical Research

18. Components of a research question (PICOT)
Patient:
Disease or condition
Stage, severity
Demographic characteristics (age, gender, etc.)
Intervention (or “Exposure”):
Type of intervention or exposure
Dose, duration, timing, route, etc.
Comparison:
Absence of risk or treatment
Placebo or alternative therapy
Outcome:
Risk or protective
Dichotomous or continuous
Type: mortality, morbidity, quality of life, etc.

19. PICOT
Type of Study:
RCTs
Cohort
Case-control
Cross-sectional
All

20. Formulation of an etiology question
Exposure
Outcome
Is smoking a risk factor for lung cancer?
Patient
Exposure
Are people who smoke regularly at a greater risk of developing lung cancer as compared to those who do not smoke?
+ cohort & case-control studies
Outcome
Comparison

21. Formulation of a diagnosis question
Test (intervention)
Outcome
Is MRI a good screening test for breast cancer?
Test (intervention)
Outcome
Is MRI a more sensitive and specific test in diagnosing breast cancer as compared to mammography among
high risk women?
Comparison

22. Develop review protocol
Step 2
Develop review protocol

23. Protocol Background Objectives Pre-determined selection criteria
Planned search strategy
Planned data abstraction
Proposed method of synthesis of findings
Establishment of an advisory group
Background
give rationale for systematic review as well as importance of research question

24. Initiate search strategy
Step 3
Initiate search strategy

25. Where to locate studies
Pubmed
CINAHL
Web of Science
EMBASE
PsychINFO

26. Additional sources to identify studies for systematic reviews
Reference lists of retrieved articles
Manual searching of relevant publications
Experts in the field
Corresponding or first authors of published
studies identified for the systematic review

27. Publication bias Search bias
Issues to consider
Publication bias
Search bias

28. Pubmed citation example
Title: Interaction between 5-HTTLPR genotype, stressful life events and depression
Search terms:
Life stress
Life event
Depression
Depress
Serotonin transporter
5-HTTLPR
Interaction
Moderation
Risch et al. JAMA 2010

29. Apply inclusion /exclusion criteria
Step 4
Apply inclusion /exclusion criteria

30. Inclusion/exclusion criteria
P - Population
I - Intervention
C - Comparison (if necessary)
O - Outcome
T - Type of study (if necessary)
Subject headings OR Textwords
To find studies using all of the PICO elements:
P and I and C and O (and T)
Reviewers must determine which studies are relevant and suitable for inclusion in the systematic review. As noted before, it is
crucial that the criteria for study selection be determined ahead of time to the extent possible, so investigators cannot, for example, decide to exclude certain types of studies that do not show a positive result. The specific inclusion criteria will depend on the study question.

31. Exclusion criteria Keep log of excluded studies
Note reasons for exclusion
Have eligibility checked by more than one reviewer
Develop strategy to resolve disagreements

32. Search strategy example
Risch et al. JAMA 2010

34. Step 5
Quality appraisal

35. Principles of quality appraisal
Quantitative studies
Internal Validity
allocation bias, confounding, attrition, statistical
analysis, intervention integrity, withdrawals and
dropouts
External Validity (generalizability or applicability)

36. Are there quality appraisal tools?
Consider assessment by more than 1 reviewer

37. QUORUM for trials
Moher et al. Improving the quality of reports of meta-analyses of randomized controlled trials: The QUORUM statement. Lancet 1999;354:

38. MOOSE for observational designs
Stroup et al. Meta-analysis of observational studies in epidemiology. JAMA 2000;283:

39. Qualitative research
Checklists available to assess the quality of qualitative research. E.g. CASP appraisal tool for qualitative research (

40. Allocate to intervention and control groups
Recruit participants
SELECTION BIAS
ALLOCATION BIAS
Allocate to intervention and control groups
Intervention group
Control group
CONFOUNDING
INTEGRITY OF INTERVENTION
Implement intervention
Implement intervention
INTENTION-TO-TREAT
Follow-up participants
Follow-up participants
WITHDRAWALS/ DROP OUTS
Always assess selection and allocation bias, blinding and handling of patient attrition
Measure outcomes
BLINDING OUTCOME ASSESSORS
Measure outcomes
DATA COLLECTION METHODS
Analyze outcomes
Analyze outcomes
STATISTICAL ANALYSIS

41. Step 6
Data abstraction

42. Data abstraction Design and pilot data abstraction form
Consider >1 reviewer
Consider blinding of observers to authors, institutions and journals

43. Data abstraction elements
publication details
study design
population details (n, characteristics)
intervention details
setting
outcomes and findings

44. Summary of study characteristics - example
Risch et al. JAMA 2009

45. Data abstraction examples

46. Step 7
Analysis

47. Synthesizing the Evidence
NARRATIVE SYNTHESIS
primary studies explained qualitatively and summarized
META-ANALYSIS
findings summarized and then combined statistically
Before deciding which approach to take, it is important
to tabulate the findings from the studies and examine
heterogeneity (i.e. diversity of studies).

48. Is there heterogeneity?No
Yes
Meta-analysis
Narrative synthesis
Deal with heterogeneity?
(e.g. subgroup analysis)

51. Guidelines for narrative synthesis
Not available yet

52. Step 8
Interpret findings

53. Interpretation of results
Strength of the evidence
Explanations of effectiveness
Applicability
Trade-offs between benefits and harms
Implications for practice
These issues should be included in the discussion and recommendations section of a systematic review:

54. Strength of the evidence
Describe the overall strength of the evidence
The quality of the evidence
The size and consistency of the results (particularly important in population based studies where a small effect at the community level may have much significance than the effect of comparable size at the individual level)
Describe biases and limitations of the review process (e.g. difficulties in locating articles, inability to translate non-English articles)

55. 8 steps of a systematic review Bent et al. 2004
1 Formulate research question
Lecture 1
2 Develop review protocol
3 Initiate search strategy
4 Apply inclusion /exclusion criteria
5 Quality appraisal
6 Data abstraction
7 Analysis
Lectures 1,2 & 3
8 Interpret findings

56. Section 1 Assignment Developing a Protocol for a Systematic Review
Homework:
Write a 1-2 page protocol to address the clinical problem described in the assignment.