SYSTEMATIC REVIEWS AND META-ANALYSIS

Objectives Define systematic review and meta-analysis Know how to access appraise interpret the results of a systematic review

Reasons to use reviews Sheer volume of literature* Save time doing exhaustive literature researches *Wyatt: about 25,000 biomedical journals in print worldwide. This number is rising by 4% pa - doubling time of 19 yrs

Review Any attempt to synthesise results and conclusions of 2 or more publications on a given topic e.g. editorials, working papers Problems Retrieval bias and publication bias Were all studies identified? Do not see how conclusions arrived at Reviewers may be biased / conflict of interest Languages excluded?

Were all studies that met the inclusion criteria for the review identified successfully? If not, were the results of the sample of studies included in the review representative of the results of all eligible studies? Of controlled trials identified in five separate subject areas within perinatal medicine, between 20 and 50 percent were identified using MEDLINE, compared with between 85 and 100 percent using the Oxford Database of Perinatal Trials. Chalmers et al. 1989

Retrieval bias What can be done about it? How can it be prevented? Systematic and comprehensive search for eligible reports How can it be prevented? Use of structured abstracts by investigators Improvement, extension and further development of Bibliographic retrieval systems Explicit search criteria

Publication bias A tendency among investigators, peer reviewers and journal editors to allow the direction and statistical significance of research findings to influence decisions regarding submission and acceptance for publication

Publication bias Examples Analysis of relative risks derived from studies examining the relationship between passive smoking and lung cancer suggests that studies in which no relationship has been detected may remain unpublished. Vandenbrouke, Br Med J 1988;296:391-392 However, estimates of the effect of these suggests that they would have little impact on the weight of evidence against passive smoking, 2004 A survey of authors or published reports of randomized trials revealed that between a quarter and a fifth of the trials that they have ever conducted had never been published, and that trials in which a new treatment had been found to be a superior to a standard treatment were more likely than others to have been published. Dickersin et al.Contr Clin Trials1987;8:343-353

Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ  2003;326:1167-1170 (31 May), Joel Lexchin, Lisa A Bero, Benjamin Djulbegovic, Otavio Clark Results 30 studies were included (1966 – 2002). Research funded by drug companies was less likely to be published than research funded by other sources. Studies sponsored by pharmaceutical companies were more likely to have outcomes favouring the sponsor than were studies with other sponsors (odds ratio 4.05; 95% confidence interval 2.98 to 5.51; 18 comparisons). None of the 13 studies that analysed methods reported that studies funded by industry was of poorer quality.

Publication bias What can be done about it? Grey literature Contact authors

Pitfalls of reviews “Current medical reviews do not routinely use scientific methods to identify, assess and synthesise information.” Mulrow, 1987 50 reviews in 4 major journals, 1985-86 no statement of methods 49 summary inappropriate 47

How can the situation improve? Better reviews (high quality, more relevant) Cochrane collaboration Systematic reviews Improved access to reviews CCPC, Databases, Effectiveness Bulletins Readers more skilled in making sense of reviews CASP

What is a systematic review? A review in which evidence on a topic has been systematically identified and summarised according to predetermined criteria. Specific clinical questions Predefined explicit methodology Reproducible Usually review of RCTs

Meta-analysis - statistical principles The use of statistical methods to summarise the results of independent studies into a single estimate giving more weight to results from larger studies. No direct comparison of patients Summary statistics are calculated for each trial Individual estimates are pooled - overall pooled estimate (if appropriate) Gives more precise estimate of effect size Interpretation / odds ratios

Types of review Reviews Systematic reviews Meta-analysis

Why use systematic reviews? Volume of literature Provide a basis for rational decision making Are health care effects consistent? Limit bias and reduce random error Provides more reliable results Required for ethics committees, funding agencies

Aim: To help people make well-informed decisions about health care by preparing, maintaining and promoting accessibility of systematic reviews of the effects of health care interventions. www.cochrane.org

6 principles of Cochrane Collaboration 5000 people in 50 countries Building on peoples’ existing enthusiasm and interests 50 Cochrane Review Groups - focus on particular areas of health Minimising duplication effort Avoidance of bias Keeping up to date Ensuring access

Format for a (Cochrane) review Contact Cochrane Develop a protocol Formulate the problem Locate and select studies Critical appraisal of studies Score for selection, attrition, performance, detection, blinding biases Collecting data Analysing and presenting results Interpreting results Improving and updating reviews

What is a Cochrane Review. http://www. cochrane. org/reviews/revstruc Cochrane Reviews investigate the effects of interventions for prevention, treatment and rehabilitation in a healthcare setting. They are designed to facilitate the choices that doctors, patients, policy makers and others face in health care. Most Cochrane Reviews are based on randomized controlled trials, but other types of evidence may also be taken into account, if appropriate. Cochrane reviews have the following general features:

General features of a Cochrane review Structured format Detailed methods section The quality of clinical studies to be incorporated into a review is carefully considered, using predefined criteria. A thorough and systematic search strategy, which includes searches for unpublished and non-English records If the data collected in a review are of sufficient quality and similar enough, they are summarised statistically in a meta-analysis, generally provides a better overall estimate of a clinical effect than the results from individual studies. Reviews aim to be relatively easy to understand for non-experts Multinational editorial teams try to ensure that a review is applicable in different parts of the world Reviews are updatable. Results from newly completed or identified clinical trials can be incorporated into the review after publication. Additionally, readers can send in comments and criticisms to any review, and reviews may be changed accordingly to improve their quality.

Systematic review protocol State objectives and eligibility criteria Identify potentially eligible studies Apply eligibility criteria Refine protocol Publish protocol

Elements of a systematic review Define the clinical question Identify all completed studies, un/published Select the studies that meet scientific validity criteria Look for evidence of bias Describe the scientific quality of the studies Assess if quality systematically related to results Describe studies with a forest plot Assess if similar enough to justify combining results Calculate summary measure of effect & CI Fletcher & Fletcher T12.1

How can we appraise a review? How can we systematically appraise a review? What questions should we ask?

Three basic types of question... Is it trustworthy? – Validity 10 questions to appraise a review Critical Appraisal Skills Programme (CASP) http://www.phru.org.uk/~casp/ Three basic types of question... Is it trustworthy? – Validity Screening questions Detailed questions on methodology What does it say? – Results Will it help? – Relevance

Is it trustworthy? – Validity Screening questions Did review address a clearly focused issue? Did authors review right type of study? Detailed questions on methodology Were all important relevant studies included? Did reviewers do enough to assess the quality of the included studies? Randomisation system, scoring system, >1 assessor If meta-analysis performed, was it reasonable to combine the results?

What does it say - Results Rates Relative and absolute risk, numbers needed to treat + confidence intervals Clinical significance Statistical significance

Will it help? – Relevance (What service providers should ask) Is my patient... Are the interventions available to my patient... Are the outcomes relevant to my patient... .....sufficiently different from people in the review to allow me to consider the findings inapplicable?

Design literature search for: Quantitative systematic review of randomised controlled trials comparing antibiotic with placebo for acute cough in adults. (BMJ 1998;316:906-10) Fahey T. et al. Aim: To establish whether antibiotics are effective in the treatment of acute cough in the community

Literature search Medical subject headings (MeSH) Databases Language? Study type Patients

Literature search Medical subject headings (MeSH) Databases Cough, bronchitis, sputum, respiratory tract infections, chest infection Databases Medline, EMBASE, Science Citation Index, Cochrane Controlled Trials Register Contacted authors - ?know of unpublished trials UK drug companies ?unpublished trials Language - not just English

Literature search Inclusion/exclusion criteria Study type Prospective trials, formal or quasi-randomisation Placebo controlled Patients >12 years Family practice clinic, community based o/px dept., hospital o/px dept. Acute cough, with/out sputum, no antibiotic in preceding week COAD excluded

Get class to interpret this table

Conclusions? Treatment with antibiotic Benefits Does not affect resolution of cough or alter course of illness Benefits Marginal for most patients with acute cough May be outweighed by side effects (RR=1.51, CI 0.86 to 2.64)