Systematic reviews and Meta- analyses Alison Brettle, Research Fellow (Information) Salford Centre for Nursing, Midwifery and Collaborative Research University of Salford

Aims To discuss the role and process of systematic reviews and meta-analyses

Systematic Review A review of all the literature on a particular topic, which has been systematically identified, appraised and summarised giving a summary answer.

What is a systematic review?  An overview of primary research studies conducted according to explicit and reproducible methodology  A rigorous method of summarising research evidence  Shows what we know and don’t know about a topic area  Provides evidence of effectiveness (or not) by summarising and appraising relevant evidence

Systematic reviews aim  To find all relevant research studies (published and unpublished)  To assess each study on basis of defined criteria  Synthesise the findings in an unbiased way  Present a balanced and impartial summary of the findings taking any flaws into consideration

 Advantages  Disadvantages

Advantages of systematic reviews  Summarise evidence, keep people up to date without reading all published research literature  Allow large amounts of data to be assimilated (eg by busy clinicians, policy makers etc)  A clearer picture by collating results of research  Break down barriers of getting research into practice  Reduce bias – removes reviewers personal opinions, preferences and specialist knowledge  Explicit methods - allow the reader to assess how review has been compiled  More reliable conclusions because of methods used

Disadvantages of systematic reviews  Inconclusive conclusions  Applicability to practice?  Time to undertake  Quality?

CRD Guidance  Systematic reviews aim to identify, evaluate and summarise the findings of all relevant individual studies, thereby making the available evidence more accessible to decision-makers. When appropriate, combining the results of several studies gives a more reliable and precise estimate of an intervention’s effectiveness than one study alone.5, 6, 7, 8 Systematic reviews adhere to a strict scientific design based on explicit, pre-specified and reproducible methods. Because of this, when carried out well, they provide reliable estimates about the effects of interventions so that conclusions are defensible. As well as setting out what we know about a particular intervention, systematic reviews can also demonstrate where knowledge is lacking.4, 9 This can then be used to guide future research 

Systematic review models  Medical/Health care Cochrane Collaboration, NHS Centre for Reviews and Dissemination Usually includes “high quality” research evidence – RCTs Often includes meta-analysis (mathematical synthesis of results of 2+ studies that addressed same hypothesis in same way)  Social care/Social Sciences SCIE, EPPI Centre, Campbell Collaboration Often include wider range of studies including qualitative Often narrative synthesis of evidence

Systematic review process  Define/focus the question  Develop a protocol  Search the literature (possibly 2 stages scoping and actual searches)  Refine the inclusion/exclusion criteria  Assess the studies (data extraction tools, 2 independent reviewers)  Combine the results of the studies to produce conclusion– can be a qualitative or quantitative (meta-analysis)  Place findings in context – quality and heterogeniety of studies, applicability of findings

What type of study design? How effective is paracetamol at reducing pain? Does smoking increase the risk of oral cancer?

STRONGExperimental studies/ clinical trials Randomised controlled trials Non-randomised controlled trials Observational studies Cohorts Case-controls Cross-sectional surveys Case series Case reports WEAK Expert opinion, consensus

Experimental studies  Randomised controlled trial  Non-randomised controlled clinical trial  Evaluating the effectiveness of an intervention

Observational studies  Cohort  Case-control  Cross-sectional survey  Measuring the incidence of a disease; looking at the causes of disease; determining prognosis  Looking at the causes of disease; identification of risk factors; suitable for examining rare diseases  Measuring the prevalence of a disease; examining the association

What is a meta-analysis? Optional part of a systematic review Systematic reviews Meta-analyses

Meta-analysis  The process of using statistical methods to combine the results of different studies.  The aim is to integrate the findings, pool the data, and identify the overall trend of results (Dictionary of Epidemiology, 1995)  Focus is the direction and magnitude of effects across studies  A method of aggregating research results – statistical technique for amalgamating, summarising and reviewing previous research

What is meta-analysis?  Using meta-analysis, a wide variety of questions can be investigated, as long as a reasonable body of primary research studies exist.  Selected parts of the reported results of primary studies are entered into a database, and this "meta-data" is "meta-analysed", in similar ways to working with other data - descriptively and then inferentially to test certain hypotheses.  Meta analysis can be used as a guide to answer the question 'does what we are doing make a difference to X?', even if 'X' has been measured using different instruments across a range of different people. Meta-analysis provides a systematic overview of quantitative research which has examined a particular question.

When can you do meta-analysis?  Research must be  Empirical rather than theoretical  Have quantitative results  Examine the same constructs and relationships  Have findings that can be put in a comparable statistical form (e.g. effects sizes or odds ratios)  Are comparable given the question in hand  (adapted from DB Wilson, 1999)

Effect size  The effect size makes meta-analysis possible – it standardises findings across studies so they can be directly compared  Any standardised index can be an effect size (for example, odds ratio, relative risk) as long as it is comparable across studies, represents magnitude and direction of relationship of interest, is independent of sample size  What to include?  Must have clear inclusion and exclusion criteria  Published studies – publication bias?  Important to identify all studies that meet eligibility criteria

Strengths and weaknesses  Represents findings in a systematic way  Can find relationships across studies  Clarifies interpretation of studies  Can handle large numbers of studies  But  Doesn’t always capture more qualitative distinctions between studies  Comparability can require judgement  Inclusion of less robust studies  Selection bias (reporting of negative findings)

Meta-analysis Understanding the jargon and the blobs!

The likelihood of something happening V The likelihood of something not happening Odds Ratio, Relative Risk Measures of risk

Relative risk  A ratio of the probability of the event occuring in the treatment (exposed) group versus a control group (non exposed)  RR = Probability of event in treatment group Probability of event in control group  For example, if the probability of developing an infection in treatment group was 20% and among control 1%, then the relative risk would be 20 and would favour the control group.probability  Similarly if the probability of developing an infection in treatment group was 4% and among control 10%, then the relative risk would be 0.4 and would favour the treatment group.probability  A RR of 1.0 = no difference between groups

Odds ratio  The odds of the event in the intervention group divided by the odds of the event in the control group  OR of 1.0 = No difference between groups  OR<1.0 means event is less likely in the intervention group

Odds Ratio Graph (Blobbogram) 2 more than less than 1 1 Line of no significance LEFT E S M O RIGHT E

Odds Ratio 2 more than less than 1 1 Best estimate Confidence Interval (wobble factor)

2 more than less than 1 1 Odds Ratio (Blobbogram)

Confidence Interval Is the range within which the true size of effect (never exactly known) lies, with a given degree of assurance (95% or 99%).

Confidence Intervals (Wobble factor)

Confidence Interval (CI) = the wobble factor, how sure are we about the results? - the shorter the CI the more certain we are about the results

the number of people you would need to treat with a specific intervention to see one additional occurrence of a specific outcome Number needed to treat (NNT)

The p-value in a nutshell How often you would see a similar result by chance, when actually there was no effect by the drug or treatment. p=0.001 Very unlikely1 in 1000 p=0.05 Fairly unlikely1 in 20 p=0.5 Fairly likely1 in 2 p=0.75 Very likely3 in 4 Impossible Certain Absolutely 0 1

Practical Example Marik PE & Zaloga GP Meta-analysis of parenteral nutrition versus enteral nutrition in patients with acute pancreatitis. BMJ; 328:1407 Objective To compare the safety and clinical outcomes of enteral and parenteral nutrition in patients with acute pancreatitis.

Problem in a ‘nutshell’  Parenteral nutrition: intravenous feeding bypassing ‘eating and digestion’; typically through an infusion pump; complications: bacterial infection  Enteral nutrition: feeding through a feeding tube to the gut  Evidence gut is optimal route yet parenteral nutrition remains widespread  In acute pancreatitis parenteral nutrition standard care but evidence suggest enteral is feasible  In acute pancreatitis most sever complication is pancreatic infection with mortality of up to 80%  Studies report parenteral nutrition increases infection rates in critically ill patients and when compared enteral nutrition is associated with improved immune function and decreased infections  Studies under-powered; differences not always statistically significant; magnitude a treatment effect unknown

From the abstract  Data sources Medline, Embase, Cochrane controlled trials register, and citation review of relevant primary and review articles.  Study selection Randomised controlled studies that compared enteral nutrition with parenteral nutrition in patients with acute pancreatitis. From 117 articles screened, six were identified as randomised controlled trials and were included for data extraction.  Data extraction Six studies with 263 participants were analysed. Descriptive and outcome data were extracted. Main outcome measures were infections, complications other than infections, operative interventions, length of hospital stay, and mortality. The meta-analysis was performed with the random effects model*.  *Random effects models are used when observations are not taken from a simple random sampling to take account of a clustering or multilevel sampling

Relative risks and continuous data outcomes are presented with 95% confidence intervals and chi square tests for heterogeneity  Relative risk is a ratio of the probability of the event occurring in the exposed group versus a non- exposed group.ratioprobability  RR = Probability of event in enteral group Probability of event in parenteral group  For example, if the probability of developing an infection in enteral group was 20% and among parenteral 1%, then the relative risk would be 20 and would favour the parenteral group.probability  Similarly if the probability of developing an infection in enteral group was 4% and among parenteral 10%, then the relative risk would be 0.4 and would favour the enteral group.probability  Testing heterogeneity between studies: χ2 test with p≤ 0.05 indicating significant heterogeneity

Fig 1 Process of study selection of randomised controlled trials (TPN=total parenteral nutrition; ENT=enteral nutrition; PN=parenteral nutrition) Marik, P. E et al. BMJ 2004;328:1407 Copyright ©2004 BMJ Publishing Group Ltd.

Table 1 Demographic data of studies included in meta- analysis. Figures are for enteral nutrition/total parenteral nutrition, and scores are given as means (SDs) No of patientsRanson criteria Glasgow Score APACHE IISiting of nasojejunal tube McClave, /161.3 (0.35) / 1.3 (0.35) Enoscopic Windsor, /182/28 / 9.5Fluoroscopic Kalfarentzos, /204.2 (0.9) / 4.6 (1.1) 12.7 (2.6)/ 11.8 (1.9) Fluoroscopic Abou-Assi, /273.1 (0.5) / 2.5 (0.4) Fluoroscopic / endoscopic Olah, /482.6 (1.2) / 2.4 (1.6) Fluoroscopic Gupta, /98 / 10Blind

Table 2 Outcome data of studies included in meta-analysis (figures are for enteral nutrition/total parental nutrition) No of patient s Septic complications Other complications Surgical complications LoSMortality McClave, /162/29.7/ /0 Windsor, /180/30/51/512.5/1 5 0/2 Kalfarentz os, /205/103/52/440/391/2 Abou- Assi, /271/913/171/214.2 /18.4 6/8 Olah, /485/133/45/ /23.6 2/4 Gupta, /90/20/67/100/0

Table 3 Jadad quality score of trials included in meta-analysis YearRandomisation method BlindingWithdrawals/dr op outs accounted for Jadad score McClure, Not statedNoneYes2 Windsor, Odd/even hospital number NoneYes1 Kalfarentzos, Sealed number envelopes NoneYes3 Abou-Assi, Not statedNoneYes2 Olah, Birth dateNoneYes1 Gupta, Sealed number envelopes NoneYes3

Results  Infections:  Relative risk RR = 0.45, (CI ), p=0.004  Test for heterogeneity between studies p=0.59  Complications:  RR = 0.61 (0.31 – 1.22), p=0.16  Surgical interventions:  RR = 0.48 ( ), p=0.05  χ2 = 0.62, p=0.89  Length of hospital stay:  Mean reduction 2.9 days (CI 1.6 – 4.3)  χ2 = 16.5, p=  Mortality  RR = 0.66 (0.32 – 1.37), p=0.3

Risk of infection, complications other than infection, surgical intervention, and mortality; results from meta- analyses of randomised trials comparing enteral with parenteral nutrition in pancreatitis Marik, P. E et al. BMJ 2004;328:1407 Copyright ©2004 BMJ Publishing Group Ltd.

Random effects model of relative risk (95% confidence interval) of infections associated with enteral feeding compared with parenteral nutrition Marik, P. E et al. BMJ 2004;328:1407 Copyright ©2004 BMJ Publishing Group Ltd.

Limitations  Poor quality studies  No blinding – may overestimate intervention effect  Different inclusion/exclusion criteria (wide range of disease severity)  Small sample numbers leading to wide confidence intervals  Heterogeneity of studies  Possibility of publication bias  Conclusion  Evidence does not support use of parenteral nutrition in patients with acute pancreatitis

Further reading  Greenhalgh T (1997) How to read a paper: papers that summarize other papers (systematic reviews and meta- analyses), BMJ, 315:  Sheldon T (2000) Statistics for evidence based nursing, Evidence Based Nursing, 3; 4-6  Sheldon T (2000) Estimating treatment effects: real or the result of chance? Evidence Based Nursing, 3; 36-39

Further reading  Systematic reviews relevant to your area of practice Do they exist? Are they really applicable? To what extent to they marry up with your practice?

Useful resources  Cochrane Collaboration  Centre for Reviews and Dissemination Finding studies for systematic reviews  EPPI-Centre – Stages of a review  SCIE - The conduct of systematic research reviews for SCIE knowledge reviews pubID=111