What are systematic reviews and why do we need them? Gavin Stewart Centre for Evidence-Based Conservation University of Birmingham, UK What are systematic reviews and why do we need them?

What is the problem? Good policy needs an evidence-base Quantity of information on a subject is large Accessibility of information is variable (some may be hard to find) Information quality is variable and so is the outcome

Quantity 1995 - 2 million articles were being published each year in 20,000 journals (medical example) a pile of paper 500 metres high if researchers tried to stay current by reading two articles per day, in one year they would fall 55 centuries behind! or, if you try to read everything of possible relevance, you would have to read 5,500 articles per day. Similar story for ecology? Have you read the latest issues of Nature, Science, Biological Conservation, Journal of Applied Ecology, Conservation Biology, TREE, Journal of Environmental Management etc ?

Accessibility & Variability You manage an upland NNR There are 58 references in the management handbook (and a further 77 you don’t know about?) Information from other side of the hill quality is variable And the outcomes are different What do you do?

What Is Systematic Review? Systematic review is a tool that provides empirical answers to scientific research questions using existing available evidence Key features Systematically locate data Critically appraise methodology synthesise evidence

They are not conventional Reviews Follow a strict methodological and statistical protocol more comprehensive minimising the chance of bias improves transparency, repeatability and reliability

Differences Between Traditional and Systematic Reviews (Adapted from Cook, D. J. et. al. (1997). Ann. Intern. Med. 126: 376-380) Feature Traditional Review Systematic Review Question Often broad in scope Focused question Sources & search Not usually specified, potentially biased Comprehensive sources & explicit search strategy Selection Rarely specified, Criterion-based selection, uniformly applied Appraisal Variable Rigorous critical appraisal, uniformly applied Synthesis Often a qualitative summary Quantitative summary* when appropriate Inferences Sometimes evidence-based Evidence-based *A quantitative summary that includes a statistical synthesis is a meta-analysis

Stages of a Review Identification of need for review Formulating a question Generating a search strategy Study relevance Study quality Data extraction Synthesis of data Recommendations

Identify and expand concepts Set the scope of the question Identification of need for review and formulating a question and protocol Define the hypothesis Identify and expand concepts Set the scope of the question

Generating a search strategy Multiple electronic databases and the internet using a range of Boolean search-terms Foreign language searches Include grey literature to avoid publication bias (see subsequent slides) Search bibliographies and contact experts

Appraising study relevance Use the question elements Subject e.g. arctic-alpine flora Intervention e.g. grazing (deer) Outcome e.g. Change in frequency at a site scale (5% decline is deleterious) If information is presented about arctic-alpines in relation to grazing and frequency is measured, the article is relevant irrespective of what it says!

Appraising study quality There is no such thing as a perfect study, all studies have weaknesses, limitations, biases Interpretation of the findings of a study depends on design, conduct and analysis, as well as on the population, interventions, and outcome measures The researchers in a primary study did not necessarily set out to answer your review question

What do we do with quality assessment results? Determine minimum quality threshold for inclusion Explore differences in quality as an explanation for heterogeneity in study results To weight individual study results in relation to their validity or the amount of information they contain Guide interpretation and overall recommendations

Data extraction Undertaken with synthesis in mind Standardised methodology with a priori rationale Pilot data extraction and ensure repeatability using two reviewers (good practice for data hygiene in any case)

Synthesising evidence Qualitative synthesis Meta-analysis to synthesise results across studies Bayesian synthesis of disparate data types particularly using experience as a prior

Review & Dissemination Unit Needs-led research Review & Dissemination Unit Decision-makers Funding Bodies Research Community Needs-Led Research