Presentation is loading. Please wait.

Presentation is loading. Please wait.

Roger F. Soll, MD H. Wallace Professor of Neonatology

Similar presentations


Presentation on theme: "Roger F. Soll, MD H. Wallace Professor of Neonatology"— Presentation transcript:

1 Understanding systematic reviews and meta-analyses in neonatal-perinatal medicine
Roger F. Soll, MD H. Wallace Professor of Neonatology Larner College of Medicine, University of Vermont Coordinating Editor, Cochrane Neonatal President, Vermont Oxford Network 1

2 Editorial Team Roger F. Soll Coordinating Editor Colleen Ovelman
Managing Editor Caitlin O’Connell Assistant Managing Editor 2

3 Editorial Team Bill McGuire Hull York Medical School
Co-coordinating Editor Cochrane Neonatal 3

4 Editorial Team New and Improved! The “Co-Co” 3

5 Editorial Team Michael Bracken Yale University Jeffrey Horbar
University of Vermont Prakeshkumar Shah University of Toronto Gautham Suresh Baylor University 3

6 Associate Editors 4

7 Guest Discussants Danielle Ehret, MD, MPH Deirdre O'Reilly, MD, MPH
Assistant Professor, University of Vermont Director, Global Health, Vermont Oxford Network Deirdre O'Reilly, MD, MPH Assistant Professor, University of Vermont Director, NPM Fellowship, University of Vermont

8 Sponsors Section on Neonatal-Perinatal Medicine

9 Understanding systematic reviews
and meta-analyses in neonatal-perinatal medicine Roger F. Soll, M.D. is the President of the Vermont Oxford Network and the Coordinating Editor of Cochrane Neonatal No other relevant financial issues to disclose

10 Understanding systematic reviews
and meta-analyses in neonatal-perinatal medicine To develop an understanding of the strengths and weaknesses of evidence provided by systematic reviews and meta-analyses to inform our practice of neonatal-perinatal medicine.

11 What evidence should I use to inform my practice?
The Evidence Hierarchy What evidence should I use to inform my practice?

12 Understanding systematic reviews and meta-analysis
Akobeng AK Archives of Disease in Childhood 2005;90:

13 “Narrative Reviews” vs. “Systematic Reviews”
Review articles in the medical literature have traditionally been in the form of “narrative reviews” where experts in a particular field provide what is supposed to be a “summary of evidence” in that field. Narrative reviews, although still very common in the medical field, have been criticized because of the high risk of bias, and “systematic reviews” are preferred. Systematic reviews apply scientific strategies in ways that limit bias to the assembly, a critical appraisal, and synthesis of relevant studies that address a specific clinical question.

14 The problem with traditional “Narrative Reviews”
While traditional review articles or narrative reviews can be useful when conducted properly, there is evidence that they are usually of poor quality. Authors of narrative reviews often use informal, subjective methods to collect and interpret studies and tend to be selective in citing reports that reinforce their preconceived ideas or promote their own views on a topic. They are also rarely explicit about how they selected, assessed, and analyzed the primary studies, thereby not allowing readers to assess potential bias in the review process. Narrative reviews are therefore often biased, and the recommendations made may be inappropriate.

15 Limit: Newborn: birth to 1 month; Randomized controlled trial (RCT);
Search: “Neonate” Limit: Newborn: birth to 1 month; Randomized controlled trial (RCT); Past 10 years 5207 RCT identified

16 Systematic Overview “a study of studies”
Applies specific research strategies to identify, appraise, and synthesize data from all relevant clinical studies “a study of studies”

17 Meta-analysis Quantitative systematic reviews include meta-analyses:
statistical methods to combine the results of similar randomized controlled trials to produce a typical estimate of the effect size

18 Meta-analysis Meta-analysis demands the same methodological quality expected in a randomized controlled trial: prospectively designed protocol comprehensive and explicit search strategy strict criteria for inclusion of studies standard definitions of outcomes standard statistical techniques

19 Meta-analysis What’s the use of meta-analysis?
increase statistical power increase precision of estimate explore differences between study results create structure for incorporating new evidence

20 Meta-analysis is a two-stage process
Meta-Analysis: Methods Meta-analysis is a two-stage process 1. The first stage involves the calculation of a measure of treatment effect with its 95% confidence intervals (CI) for each individual study. The statistics that are usually used to measure treatment effect include odds ratios (OR), relative risks (RR), and risk differences.

21 Meta-Analysis Methods: Dichotomous Outcomes
Intervention event rate (IER) = rate at which an event occurs in the experimental group Control event rate (CER) = rate at which an event occurs in the control group Risk ratio (Relative risk) = the ratio of the risk in the experimental group to the risk in the control group Risk difference (absolute risk difference) = the difference between the risks in the experimental and control groups Number needed to treat = number of persons who must be treated for one person to benefit = 1/absolute value of the risk difference

22 Meta-Analysis Methods: Dichotomous Outcomes
95% Confidence Interval (CI) The 95% confidence intervals would contain the true underlying effect in 95% of the occasions if the study was repeated again and again (and again).

23 Clinically relevant measures of treatment effect
Relative Risk and Relative Risk Reduction measure the strength of the association not the clinical importance Need to consider frequency of outcome in population Large RR: evidence of a causal relationship Large ARD: evidence of actual effect of treatment

24 Meta-Analysis Methods
2. In the second stage of meta-analysis, an overall treatment effect is calculated as a weighted average of the individual summary statistics. In meta-analysis, data from the individual studies are not simply combined as if they were from a single study. Greater weights are given to the results from studies that provide more information, because they are likely to be closer to the “true effect” we are trying to estimate. The weights are often the inverse of the variance (the square of the standard error) of the treatment effect, which relates closely to sample size. The typical graph for displaying the results of a meta-analysis is called a “forest plot”. We will look at “forest plots” later in the discussion today.

25 Appraising a systematic review with or without meta-analysis
Although systematic reviews occupy the highest position in the hierarchy of evidence for articles on effectiveness of interventions, it should not be assumed that a study is valid merely because it is stated to be a systematic review. Just as in RCTs, the main issues to consider when appraising a systematic review can be condensed into three important areas: The validity of the trial methodology. The magnitude and precision of the treatment effect. The applicability of the results to your patient or population.

26 Have you ever meta-analysis you didn’t like?
Steven Goodman MD, MHS, PhD Annals of Internal Medicine 1991; 114:

27 Only fair agreement between large clinical trials and meta-analyses
Problems with meta-analysis Only fair agreement between large clinical trials and meta-analyses LeLorier 1997

28 META-ANALYSIS OF MULTIPLE SMALL STUDIES COMPARED TO SINGLE LARGE STUDY
ASPIRIN FOR PREVENTION OF PRE-ECLAMPSIA Odds Ratio Decreased Risk Increased CHARACTERISTIC (95% CI) 0.2 0.5 1.0 2.0 4.0 META-ANALYSIS 0.30 (0.20, 0.42) SINGLE LARGE RCT 0.82 (0.72, 1.05) 0.2 0.5 1.0 2.0 4.0 Odds Ratio and 95% CI

29 Problems with meta-analysis
Methodological flaws in meta-analyses Publication bias The tendency for investigators to preferentially submit studies with positive results, and the tendency for editors to choose positive studies for publication Heterogeneity Concerning variation in the direction or the degrees of results between individual studies

30 PROPORTION OF CONTROLLED TRIALS PUBLISHED IN GERMAN AND ENGLISH
Language Bias PROPORTION OF CONTROLLED TRIALS PUBLISHED IN GERMAN AND ENGLISH EGGER 1997

31 Heterogeneity Inevitably, studies brought together in a systematic review will differ. Any kind of variability among studies in a systematic review may be termed heterogeneity. Clinical: Variability in the participants, interventions and outcomes studied Methodological: Variability in study design and risk of bias

32 “You cannot make a silk purse from a sow’s ear” RCTs
Systematic Reviews

33 Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement.
Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. BMJ 2009;339:b2535, doi: /bmj.b2535

34 Checklist of Items to Include When Reporting
a Systematic Review or Meta-Analysis Section/topic # Checklist item Reported on page # TITLE Title 1 Identify the report as a systematic review, meta-analysis, or both. ABSTRACT Structured summary 2 Provide a structured summary including, as applicable: background; objectives; data sources; study eligibility criteria, participants, and interventions; study appraisal and synthesis methods; results; limitations; conclusions and implications of key findings; systematic review registration number. INTRODUCTION Rationale 3 Describe the rationale for the review in the context of what is already known. Objectives 4 Provide an explicit statement of questions being addressed with reference to participants, interventions, comparisons, outcomes, and study design (PICOS). METHODS Protocol and registration 5 Indicate if a review protocol exists, if and where it can be accessed (e.g., Web address), and, if available, provide registration information including registration number. Eligibility criteria 6 Specify study characteristics (e.g., PICOS, length of follow-up) and report characteristics (e.g., years considered, language, publication status) used as criteria for eligibility, giving rationale. Information sources 7 Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched. Search 8 Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated. Study selection 9 State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if applicable, included in the meta- analysis). Data collection process 10 Describe method of data extraction from reports (e.g., piloted forms, independently, in duplicate) and any processes for obtaining and confirming data from investigators. Data items 11 List and define all variables for which data were sought (e.g., PICOS, funding sources) and any assumptions and simplifications made. Risk of bias in individual studies 12 Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level), and how this information is to be used in any data synthesis. Summary measures 13 State the principal summary measures (e.g., risk ratio, difference in means). Synthesis of results 14 Describe the methods of handling data and combining results of studies, if done, including measures of consistency (e.g., I2) for each meta- analysis.

35 Checklist of Items to Include When Reporting
a Systematic Review or Meta-Analysis Section/topic # Checklist item Reported on page # Risk of bias across studies 15 Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies). Additional analyses 16 Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified. RESULTS Study selection 17 Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram. Study characteristics 18 For each study, present characteristics for which data were extracted (e.g., study size, PICOS, follow-up period) and provide the citations. Risk of bias within studies 19 Present data on risk of bias of each study and, if available, any outcome level assessment (see item 12). Results of individual studies 20 For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each intervention group (b) effect estimates and confidence intervals, ideally with a forest plot. Synthesis of results 21 Present results of each meta-analysis done, including confidence intervals and measures of consistency. 22 Present results of any assessment of risk of bias across studies (see Item 15). Additional analysis 23 Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression [see Item 16]). DISCUSSION Summary of evidence 24 Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups (e.g., healthcare providers, users, and policy makers). Limitations 25 Discuss limitations at study and outcome level (e.g., risk of bias), and at review-level (e.g., incomplete retrieval of identified research, reporting bias). Conclusions 26 Provide a general interpretation of the results in the context of other evidence, and implications for future research. FUNDING Funding 27 Describe sources of funding for the systematic review and other support (e.g., supply of data); role of funders for the systematic review.

36 Cooling for newborns with hypoxic ischaemic encephalopathy
Jacobs SE, Berg M, Hunt R, Tarnow-Mordi WO, Inder TE, Davis PG Cochrane Database Syst Rev Jan 31;(1):CD doi: / CD pub3. .

37 Cooling for newborns with hypoxic ischaemic encephalopathy
Objectives: To determine the effect of therapeutic hypothermia on death and long-term neurodevelopmental disability, and to ascertain clinically important side effects in newborn infants with HIE. Jacobs and colleagues. Cochrane Database Syst Rev Jan 31;(1):CD doi: / CD pub3.

38 PICOT Question

39 Cooling for newborns with hypoxic ischaemic encephalopathy
PICOT Question In newborn infants with HIE, does therapeutic hypothermia compared to conventional management decrease the risk of death or long-term neurodevelopmental disability at 18 to 24 months of age.

40 Searching databases Bank 14813 items Cast iron bank 805 items
Mechanical bank items Book of knowledge bank 14 items

41 Search strategy We use the standard search strategy of Cochrane Neonatal to search Cochrane Central Register of Controlled Trials (CENTRAL) in the Cochrane Library, OVID Medline; MEDLINE via PubMed for the previous year; and CINAHL. We also search clinical trials databases and the reference lists of retrieved articles for randomized controlled trials and quasi-randomized trials.

42 Searching bibliographic databases
Therapeutic hypothermia items Newborn items Clinical trials items Randomized controlled trials 127 items

43 Data synthesis   If we identify multiple studies that we consider to be sufficiently similar, we will perform meta-analysis using Review Manager 5 (Review Manager 2014). For categorical outcomes, we will calculate the typical estimates of RR and RD, each with its 95% CI; for continuous outcomes, we will calculate the mean difference or the standardized mean difference, each with its 95% CI. We will use a fixed-effect model to combine data where it is reasonable to assume that studies were estimating the same underlying treatment effect.

44 The forest plot The plot shows, at a glance, information from the individual studies that went into the meta-analysis, and an estimate of the overall results. It also allows a visual assessment of the amount of variation between the results of the studies (heterogeneity).

45 Death or Major Disability in Survivors: Relative Risk

46 Death or Major Disability in Survivors: Risk Difference

47 Death or Major Disability in Survivors: Relative Risk
Annotated Forest Plot Death or Major Disability in Survivors: Relative Risk Confidence interval Point estimate Individual study Typical estimate Line of identity I squared

48 Heterogeneity Chi2 assesses whether observed differences in results are compatible with chance alone. I2 quantifying inconsistency across studies A rough guide to interpretation is as follows:  0% to 40%: might not be important; 30% to 60%: may represent moderate heterogeneity 50% to 90%: may represent substantial heterogeneity 75% to 100%: considerable heterogeneity

49 Page. Research designs in sports physical therapy
Page. Research designs in sports physical therapy. International journal of sports physical therapy. Oct 2012;7(5):482‐492.

50 Quality of evidence We use the GRADE approach to assess the quality of evidence for clinically relevant outcomes. We consider evidence from randomized controlled trials as high quality, downgrading the evidence one level for serious (or two levels for very serious) limitations based upon the following: design (risk of bias), consistency across studies, directness of the evidence, precision of estimates, and presence of publication bias.

51

52 Applicability/Generalizability of results to YOUR patients
Health care professionals should always make judgements about whether the results of a study are applicable to their own patient or group of patients. These include similarity of study population to your population, benefit vs harm, patients' preferences, availability, and costs.

53 Meta-analysis is useful
In guiding evidence-based practice To formulate research questions To create a context in which to view new evidence

54 Guest Discussants Danielle Ehret, MD, MPH Deirdre O'Reilly, MD, MPH
Assistant Professor, University of Vermont Director, Global Health, Vermont Oxford Network Deirdre O'Reilly, MD, MPH Assistant Professor, University of Vermont Director, NPM Fellowship, University of Vermont

55

56 Get your facts first, then you can distort them as you please Mark Twain


Download ppt "Roger F. Soll, MD H. Wallace Professor of Neonatology"

Similar presentations


Ads by Google