Levels of evidence and Interpretation of a systematic review

THE HIERARCHY OF EVIDENCE 1a. Systematic review with homogeneity of RCTs 1b. RCT with narrow confidence interval 2. Cohort studies with > 80% follow up 3. Case control study 4. Case series and poor quality cohort/ case control 5. Expert opinion There are many systems such as Canadian Task force / GRADE which classify the levels of evidence. GRADE One such system goes as follows: Level 1a of evidence is constituted by systematic reviews free of heterogeneity either in direction or degree of effect. Level 1b includes good quality randomised trials where the confidence intervals are narrow. Level 2 evidence comprises sytematic review of cohort studies or individual cohort studies where follow up rates are above 80% . Level 3 evidence arises from systematic review of case control studies or individual case control studies, Level 4 – case series and level 5 from expert opinion without explicit critical appraisal. Cohort studies are said to be of poor quality if comparison groups are not clearly defined, exposures and outcomes are not measured in the same manner, confounders are not appropriately identified and adjusted for or if follow up is insufficient. Evidence based Medicine: Sharon Strauss 4th edition

SYTEMATIC REVIEW AND META ANALYSIS Meta analysis- A statistical technique used to combine the results of several studies addressing the same question into a single summary measure Systematic reviews do not have to have a meta-analysis The term ‘meta-analysis’ is often used interchangeably with ‘systematic review’. Meta anlaysis- is actually a statistical technique used to combine the results of several studies addressing the same question into a single summary measure. Systematic reviews do not have to have a meta-analysis There are times when it is not appropriate (high degree of heterogeneity among the studies) or possible.

THE COCHRANE COLLABORATION International non-profit organisation that prepares, maintains, and disseminates systematic up-to-date reviews of health care interventions The Cochrane Collaboration was established in October 1992. The Collaboration is an International non-profit organisation that prepares, maintains, and disseminates systematic up-to-date reviews of health care interventions. It relies on funding from a number of different sources to complete its work.

FOREST PLOT Graphical representation of a meta analysis “Forest of lines” Originated in 80’s Modern forest plot - 1998 Forest plots are graphical representations of the meta-analysis. The word originated from the idea that graph had a forest of lines. The plot originated in the early eighties although the term forest plot was coined only in 1996. Forest plots in their modern form originated in 1998.

HOW TO READ A FOREST PLOT Individual studies Let us look at this forest plot of the stdies evaluating the efficacy of megnesium sulfate given antenatally to prevent cerebral palsy in infants. Each study is represented by a line – we can see that there are 4 studies in this meta analysis (A). The names of the individual studies and year of publication are indicated on the left (A).

HOW TO READ A FOREST PLOT Effect estimate for each study There is a box in the line for each study. The mid-point of the box represents the point effect estimate, that is, the mean effect estimate for each study. The area of the box represents the weight given to the study (A). This is designed so that eyes are drawn towards the studies that are given more weight. The diamond below the studies represents the overall effect (A). Pooled estimate

HOW TO READ A FOREST PLOT Confidence intervals of individual studies’ effect estimate CI of pooled estimate The width of the line shows the confidence intervals of the effect estimate of individual studies (A). The width of the diamond shows the confidence intervals for the overall effect estimate (A) as indicated in figures here. What do point estimate and 95% confidence intervals mean? Point estimate is best guess of the true effect in the population. 95% confidence intervals mean that there is a 95% chance that the true effect in the population will lie within the range. They also mean that if the trial is repeated, there is a 95% chance that the point estimate from the trial lies within the 95% confidence intervals obtained in the systematic review. These are based on the sample being representative and the assumption that there are no systematic errors that can bias the results.

HOW TO READ A FOREST PLOT There is a vertical line which corresponds to the value 1 in the plot shown. This is the line of no effect (A). This means that the intervention neither increases nor decreases the risk of developing the outcome of interest. Note also that effect estimates lying to the left of the line favours the experimental group, and those to the right ‘favour control’ group. This label however is interchangeable. If the line of no effect is included in the 95% confidence intervals, it indicates that there is no statistical significance for the effect of the intervention. If RR of 1/ line of no effect is not included in the 95% confidence intervals, the results are statistically significant. This is applicable for effect estimates of the individual studies and for the overall estimate. In the given eg., the 95% Confidence limits of the pooled estimate extend from 0.55 to 0.91, thus not crossing the line of no effect, in other words the intervention has a statistically significant effect on the outcome, More specifically, the pooled RR favours the intervention group, as there is 29% reduction in the risk of cerebral palsy. Favours intervention Favours control Line of no effect

WHAT ELSE DOES THE FOREST PLOT SHOW? Method/ Model used The Forest plot also provides the summary data entered for each study (A). In addition, it provides the weight for each individual study (A); the method and the model used to perform the meta-analysis, is indicated here (A). For eg, this meta analysis has used the Mantel Henzel method and fixed effects model for performing the meta analysis. Another important consideration in interpreting the forest plot is heterogeneity between studies, which is indicated here (A). In general, heterogeneity is taken to be significant if this parameter Isquared is greater than 50-60%. and the statistical significance of the analysis. For the time being, ignore the information on heterogeneity. In general, random effects model is used if heterogeneity between studies is significant. In other cases, the fixed effects model is used. Weight Effect estimate for each study Heterogeneity

FOREST PLOT FOR CONTINUOUS VARIABLES Mean difference instead of relative risk In case of quantitative measures, the weighted or standardised mean difference is used instead of relative risk. In this case, 0 indicates no effect. If 0 is included in the 95% confidence intervals, it indicates that there is no statistical significance. If 0 is not included in the 95% confidence intervals, the results are statistically significant. This is applicable for effect estimates for the individual study level and for the overall estimate. Whether the intervention is beneficial or harmful depends upon the context.

LEARNING POINTS IN THIS PODCAST Levels of evidence Systematic review and meta analysis Forest plot – Components Interpretation