1. Understanding systematic reviews and meta-analyses in neonatal-perinatal medicine
Roger F. Soll, MD
H. Wallace Professor of Neonatology
Larner College of Medicine, University of Vermont
Coordinating Editor, Cochrane Neonatal
President, Vermont Oxford Network 1

2. Editorial Team Roger F. Soll Coordinating Editor Colleen Ovelman
Managing Editor
Caitlin O’Connell
Assistant
Managing Editor 2

3. Editorial Team Bill McGuire Hull York Medical School
Co-coordinating Editor
Cochrane Neonatal 3

4. Editorial Team
New and Improved!
The “Co-Co” 3

5. Editorial Team Michael Bracken Yale University Jeffrey Horbar
University of Vermont
Prakeshkumar Shah
University of Toronto
Gautham Suresh
Baylor University 3

6. Associate Editors 4

7. Guest Discussants Danielle Ehret, MD, MPH Deirdre O'Reilly, MD, MPH
Assistant Professor, University of Vermont
Director, Global Health, Vermont Oxford Network
Deirdre O'Reilly, MD, MPH
Assistant Professor, University of Vermont
Director, NPM Fellowship, University of Vermont

8. Sponsors
Section on Neonatal-Perinatal Medicine

9. Understanding systematic reviews
and meta-analyses in neonatal-perinatal medicine
Roger F. Soll, M.D. is the President
of the Vermont Oxford Network
and the Coordinating Editor
of Cochrane Neonatal
No other relevant financial issues to disclose

10. Understanding systematic reviews
and meta-analyses in neonatal-perinatal medicine
To develop an understanding of the strengths and weaknesses of evidence provided by systematic reviews and meta-analyses to inform our practice of neonatal-perinatal medicine.

11. What evidence should I use to inform my practice?
The Evidence Hierarchy
What evidence should I use to inform my practice?

12. Understanding systematic reviews and meta-analysis
Akobeng AK
Archives of Disease in Childhood 2005;90:

13. “Narrative Reviews” vs. “Systematic Reviews”
Review articles in the medical literature have traditionally been in the form of “narrative reviews” where experts in a particular field provide what is supposed to be a “summary of evidence” in that field.
Narrative reviews, although still very common in the medical field, have been criticized because of the high risk of bias, and “systematic reviews” are preferred.
Systematic reviews apply scientific strategies in ways that limit bias to the assembly, a critical appraisal, and synthesis of relevant studies that address a specific clinical question.

14. The problem with traditional “Narrative Reviews”
While traditional review articles or narrative reviews can be useful when conducted properly, there is evidence that they are usually of poor quality.
Authors of narrative reviews often use informal, subjective methods to collect and interpret studies and tend to be selective in citing reports that reinforce their preconceived ideas or promote their own views on a topic.
They are also rarely explicit about how they selected, assessed, and analyzed the primary studies, thereby not allowing readers to assess potential bias in the review process.
Narrative reviews are therefore often biased, and the recommendations made may be inappropriate.

15. Limit: Newborn: birth to 1 month; Randomized controlled trial (RCT);
Search: “Neonate”
Limit: Newborn: birth to 1 month;
Randomized controlled trial (RCT);
Past 10 years
5207 RCT identified

16. Systematic Overview “a study of studies”
Applies specific research strategies to identify,
appraise, and synthesize data
from all relevant clinical studies
“a study of studies”

17. Meta-analysis Quantitative systematic reviews include meta-analyses:
statistical methods to combine the results
of similar randomized controlled trials
to produce a typical estimate of the effect size

18. Meta-analysis
Meta-analysis demands the same methodological quality expected in a randomized controlled trial:
prospectively designed protocol
comprehensive and explicit search strategy
strict criteria for inclusion of studies
standard definitions of outcomes
standard statistical techniques

19. Meta-analysis What’s the use of meta-analysis?
increase statistical power
increase precision of estimate
explore differences between study results
create structure for incorporating new evidence

20. Meta-analysis is a two-stage process
Meta-Analysis: Methods
Meta-analysis is a two-stage process
1. The first stage involves the calculation of a measure of treatment effect with its 95% confidence intervals (CI) for each individual study.
The statistics that are usually used to measure treatment effect include odds ratios (OR), relative risks (RR), and risk differences.

21. Meta-Analysis Methods: Dichotomous Outcomes
Intervention event rate (IER) = rate at which an event occurs in the experimental group
Control event rate (CER) = rate at which an event occurs in the control group
Risk ratio (Relative risk) = the ratio of the risk in the experimental group to the risk in the control group
Risk difference (absolute risk difference) = the difference between the risks in the experimental and control groups
Number needed to treat = number of persons who must be treated for one person to benefit = 1/absolute value of the risk difference

22. Meta-Analysis Methods: Dichotomous Outcomes
95% Confidence Interval (CI)
The 95% confidence intervals would contain the true underlying effect in 95% of the occasions if the study was repeated again and again (and again).

23. Clinically relevant measures of treatment effect
Relative Risk and Relative Risk Reduction measure the strength of the association not the clinical importance
Need to consider frequency of outcome in population
Large RR: evidence of a causal relationship
Large ARD: evidence of actual effect of treatment

24. Meta-Analysis Methods
2. In the second stage of meta-analysis, an overall treatment effect is calculated as a weighted average of the individual summary statistics.
In meta-analysis, data from the individual studies are not simply combined as if they were from a single study.
Greater weights are given to the results from studies that provide more information, because they are likely to be closer to the “true effect” we are trying to estimate.
The weights are often the inverse of the variance (the square of the standard error) of the treatment effect, which relates closely to sample size. The typical graph for displaying the results of a meta-analysis is called a “forest plot”.
We will look at “forest plots” later in the discussion today.

25. Appraising a systematic review with or without meta-analysis
Although systematic reviews occupy the highest position in the hierarchy of evidence for articles on effectiveness of interventions, it should not be assumed that a study is valid merely because it is stated to be a systematic review.
Just as in RCTs, the main issues to consider when appraising a systematic review can be condensed into three important areas:
The validity of the trial methodology.
The magnitude and precision of the treatment effect.
The applicability of the results to your patient or population.

26. Have you ever meta-analysis you didn’t like?
Steven Goodman MD, MHS, PhD
Annals of Internal Medicine 1991; 114:

27. Only fair agreement between large clinical trials and meta-analyses
Problems with meta-analysis
Only fair agreement between large clinical trials and meta-analyses
LeLorier 1997

28. META-ANALYSIS OF MULTIPLE SMALL STUDIES COMPARED TO SINGLE LARGE STUDY
ASPIRIN FOR PREVENTION OF PRE-ECLAMPSIA
Odds Ratio
Decreased
Risk
Increased
CHARACTERISTIC
(95% CI)
0.2
0.5
1.0
2.0
4.0
META-ANALYSIS
0.30 (0.20, 0.42)
SINGLE LARGE RCT
0.82 (0.72, 1.05)
0.2
0.5
1.0
2.0
4.0
Odds Ratio and 95% CI

29. Problems with meta-analysis
Methodological flaws in meta-analyses
Publication bias
The tendency for investigators to preferentially submit studies with positive results, and the tendency for editors to choose positive studies for publication
Heterogeneity
Concerning variation in the direction or the degrees of results between individual studies

30. PROPORTION OF CONTROLLED TRIALS PUBLISHED IN GERMAN AND ENGLISH
Language Bias
PROPORTION OF CONTROLLED TRIALS PUBLISHED IN GERMAN AND ENGLISH
EGGER 1997

31. Heterogeneity
Inevitably, studies brought together in a systematic review will differ.
Any kind of variability among studies in a systematic review may be termed heterogeneity.
Clinical: Variability in the participants, interventions and outcomes studied
Methodological: Variability in study design and risk of bias

32. “You cannot make a silk purse from a sow’s ear” RCTs
Systematic Reviews

33. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement.
Moher D, Liberati A, Tetzlaff J, Altman DG,
The PRISMA Group.
BMJ 2009;339:b2535, doi: /bmj.b2535

34. Checklist of Items to Include When Reporting
a Systematic Review or Meta-Analysis
Section/topic #
Checklist item
Reported on page #
TITLE
Title 1
Identify the report as a systematic review, meta-analysis, or both.
ABSTRACT
Structured summary 2
Provide a structured summary including, as applicable: background; objectives; data sources; study eligibility criteria, participants, and interventions; study appraisal and synthesis methods; results; limitations; conclusions and implications of key findings; systematic review registration number.
INTRODUCTION
Rationale 3
Describe the rationale for the review in the context of what is already known.
Objectives 4
Provide an explicit statement of questions being addressed with reference to participants, interventions, comparisons, outcomes, and study design (PICOS).
METHODS
Protocol and registration 5
Indicate if a review protocol exists, if and where it can be accessed (e.g., Web address), and, if available, provide registration information including registration number.
Eligibility criteria 6
Specify study characteristics (e.g., PICOS, length of follow-up) and report characteristics (e.g., years considered, language, publication status) used as criteria for eligibility, giving rationale.
Information sources 7
Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched.
Search 8
Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated.
Study selection 9
State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if applicable, included in the meta- analysis).
Data collection process 10
Describe method of data extraction from reports (e.g., piloted forms, independently, in duplicate) and any processes for obtaining and confirming data from investigators.
Data items 11
List and define all variables for which data were sought (e.g., PICOS, funding sources) and any assumptions and simplifications made.
Risk of bias in individual studies 12
Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level), and how this information is to be used in any data synthesis.
Summary measures 13
State the principal summary measures (e.g., risk ratio, difference in means).
Synthesis of results 14
Describe the methods of handling data and combining results of studies, if done, including measures of consistency (e.g., I2) for each meta- analysis.

35. Checklist of Items to Include When Reporting
a Systematic Review or Meta-Analysis
Section/topic #
Checklist item
Reported on page #
Risk of bias across studies 15
Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies).
Additional analyses 16
Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified.
RESULTS
Study selection 17
Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram.
Study characteristics 18
For each study, present characteristics for which data were extracted (e.g., study size, PICOS, follow-up period) and provide the citations.
Risk of bias within studies 19
Present data on risk of bias of each study and, if available, any outcome level assessment (see item 12).
Results of individual studies 20
For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each intervention group (b) effect estimates and confidence intervals, ideally with a forest plot.
Synthesis of results 21
Present results of each meta-analysis done, including confidence intervals and measures of consistency. 22
Present results of any assessment of risk of bias across studies (see Item 15).
Additional analysis 23
Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression [see Item 16]).
DISCUSSION
Summary of evidence 24
Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups (e.g., healthcare providers, users, and policy makers).
Limitations 25
Discuss limitations at study and outcome level (e.g., risk of bias), and at review-level (e.g., incomplete retrieval of identified research, reporting bias).
Conclusions 26
Provide a general interpretation of the results in the context of other evidence, and implications for future research.
FUNDING
Funding 27
Describe sources of funding for the systematic review and other support (e.g., supply of data); role of funders for the systematic review.

36. Cooling for newborns with hypoxic ischaemic encephalopathy
Jacobs SE, Berg M, Hunt R, Tarnow-Mordi WO, Inder TE, Davis PG
Cochrane Database Syst Rev Jan 31;(1):CD doi: / CD pub3. .

37. Cooling for newborns with hypoxic ischaemic encephalopathy
Objectives: To determine the effect of therapeutic hypothermia on death and long-term neurodevelopmental disability, and to ascertain clinically important side effects in newborn infants with HIE.
Jacobs and colleagues. Cochrane Database Syst Rev Jan 31;(1):CD doi: / CD pub3.

38. PICOT Question

39. Cooling for newborns with hypoxic ischaemic encephalopathy
PICOT Question
In newborn infants with HIE, does therapeutic hypothermia compared to conventional management decrease the risk of death or long-term neurodevelopmental disability at 18 to 24 months of age.

40. Searching databases Bank 14813 items Cast iron bank 805 items
Mechanical bank items
Book of knowledge bank 14 items

41. Search strategy
We use the standard search strategy of Cochrane Neonatal to search Cochrane Central Register of Controlled Trials (CENTRAL) in the Cochrane Library, OVID Medline; MEDLINE via PubMed for the previous year; and CINAHL.
We also search clinical trials databases and the reference lists of retrieved articles for randomized controlled trials and quasi-randomized trials.

42. Searching bibliographic databases
Therapeutic hypothermia items
Newborn items
Clinical trials items
Randomized controlled trials 127 items

43. Data synthesis  
If we identify multiple studies that we consider to be sufficiently similar, we will perform meta-analysis using Review Manager 5 (Review Manager 2014).
For categorical outcomes, we will calculate the typical estimates of RR and RD, each with its 95% CI; for continuous outcomes, we will calculate the mean difference or the standardized mean difference, each with its 95% CI.
We will use a fixed-effect model to combine data where it is reasonable to assume that studies were estimating the same underlying treatment effect.

44. The forest plot
The plot shows, at a glance, information from the individual studies that went into the meta-analysis, and an estimate of the overall results.
It also allows a visual assessment of the amount of variation between the results of the studies (heterogeneity).

45. Death or Major Disability in Survivors: Relative Risk

46. Death or Major Disability in Survivors: Risk Difference

47. Death or Major Disability in Survivors: Relative Risk
Annotated Forest Plot
Death or Major Disability in Survivors: Relative Risk
Confidence interval
Point estimate
Individual study
Typical estimate
Line of identity
I squared

48. Heterogeneity
Chi2 assesses whether observed differences in results are compatible with chance alone.
I2 quantifying inconsistency across studies
A rough guide to interpretation is as follows:
 0% to 40%: might not be important;
30% to 60%: may represent moderate heterogeneity
50% to 90%: may represent substantial heterogeneity
75% to 100%: considerable heterogeneity

49. Page. Research designs in sports physical therapy
Page. Research designs in sports physical therapy. International journal of sports physical therapy. Oct 2012;7(5):482‐492.

50. Quality of evidence
We use the GRADE approach to assess the quality of evidence for clinically relevant outcomes.
We consider evidence from randomized controlled trials as high quality, downgrading the evidence one level for serious (or two levels for very serious) limitations based upon the following: design (risk of bias), consistency across studies, directness of the evidence, precision of estimates, and presence of publication bias.

52. Applicability/Generalizability of results to YOUR patients
Health care professionals should always make judgements about whether the results of a study are applicable to their own patient or group of patients.
These include similarity of study population to your population, benefit vs harm, patients' preferences, availability, and costs.

53. Meta-analysis is useful
In guiding evidence-based practice
To formulate research questions
To create a context in which to view new evidence

54. Guest Discussants Danielle Ehret, MD, MPH Deirdre O'Reilly, MD, MPH
Assistant Professor, University of Vermont
Director, Global Health, Vermont Oxford Network
Deirdre O'Reilly, MD, MPH
Assistant Professor, University of Vermont
Director, NPM Fellowship, University of Vermont

56. Get your facts first,
then you can distort them as you please
Mark Twain