Systematic reviews – informing policy, practice and research Iain Chalmers Coordinator, James Initiative www.jameslindlibrary.org SysNet Annual Lecture, Cardiff University 23 April 2012

to clinicians & patients? 85% Research waste = over $85 Billion / year Questions relevant to clinicians & patients? Appropriate design and methods? Accessible full publication? Unbiased and usable report? Low priority questions addressed Important outcomes not assessed Clinicians and patients not involved in setting research agendas Over 50% studies designed without reference to systematic reviews of existing evidence Over 50% of studies fail to take adequate steps to reduce biases, e.g. unconcealed treatment allocation Over 50% of studies never published in full Biased under-reporting of studies with disappointing results Over 30% of trial interventions not sufficiently described Over 50% of planned study outcomes not reported Most new research not interpreted in the context of systematic assessment of other relevant evidence 50% 50% 50% 85% Research waste = over $85 Billion / year 3 3

to clinicians & patients? 85% Research waste = over $85 Billion / year Questions relevant to clinicians & patients? Appropriate design and methods? Accessible full publication? Unbiased and usable report? Low priority questions addressed Important outcomes not assessed Clinicians and patients not involved in setting research agendas Over 50% studies designed without reference to systematic reviews of existing evidence Over 50% of studies fail to take adequate steps to reduce biases, e.g. unconcealed treatment allocation Over 50% of studies never published in full Biased under-reporting of studies with disappointing results Over 30% of trial interventions not sufficiently described Over 50% of planned study outcomes not reported Most new research not interpreted in the context of systematic assessment of other relevant evidence 50% 50% 50% 85% Research waste = over $85 Billion / year 4 4

Does anyone here think that researchers should NOT review existing evidence systematically before developing treatment guidelines and planning new research?

Patients have suffered and died unnecessarily, and resources for health care and health research have been wasted, because existing research evidence has not been reviewed systematically. 6

JAMA 1992;268:240-248.

The human costs of failing to cumulate evidence from research scientifically “Advice on some life-saving therapies has been delayed for more than a decade, while other treatments have been recommended long after controlled research has shown them to be harmful.” Antman et al. JAMA 1992;268:240-8.

Oxford Textbook of Medicine, 2nd edn, 1987 “The clinical benefits of thrombolysis … remain to be established.”

Sometimes very important advances in knowledge come from systematic reviews of existing evidence

"It is essential that existing sources of evidence, especially systematic reviews, are considered carefully prior to undertaking research. Research which duplicates other work unnecessarily, or which is not of sufficient quality to contribute something useful to existing knowledge, is in itself unethical." Department of Health. Research Governance Framework for Health and Social Care, 2001, para 2.3.1

TGN1412 TGN 1412

18

Cumulative estimate from trials of the effect of aprotinin on perioperative blood transfusion, 1987-2002. Aprotinin better Aprotinin worse 19

New trials of aprotinin ignored previous trials 20

But was it the right question? Is tranexamic acid better than aprotinin? A comparison of aprotinin and lysine analogues in high-risk cardiac surgery. Ferguson et al. N Engl J Med. 2008 May 29;358(22):2319-31.

* *actually, the non-use Only 11 of 24 responding authors of trial reports that had been added to existing systematic reviews were even aware of the relevant reviews when they designed their new studies. 22

23

STUDIES IN ANIMALS 20 animal studies: “The results of this review did not show convincing evidence to substantiate the decision to perform trials with nimodipine in large numbers of patients. Stroke 2001;32:2433-8. Horn J, Limburg M. Calcium antagonists for acute ischemic stroke. The Cochrane Database of Systematic Reviews, 2000 “46 trials were identified of which 28 were included (7521 patients). No effect of calcium antagonists on poor outcome at the end of follow-up (OR 1.07, 95% CI 0.97/1.18), or on death at end of follow-up (OR 1.10, 95% CI 0.98/1.24) was found.” STUDIES IN HUMANS

2005 “…there is a relative scarcity of systematic reviews…” “…it would therefore be desirable to undertake further systematic reviews and meta-analyses to evaluate more fully the predictability and transferability of animal models.”

Can this policy be defended on scientific, ethical, or economic grounds?

No subsequent contact made. Richard Hughes, personal communication, 30 Nov 2011

To reiterate Patients have suffered and died unnecessarily, and resources for health care and health research have been wasted, because existing research evidence has not been reviewed systematically. 37

Austin Bradford Hill, 1965 Four questions to which readers want answers when reading reports of research. 1. Why did you start? 2. What did you do? 3. What answer did you get? 4. And what does it mean anyway? 38

Lord Rayleigh, 1842-1919 “One of the very few members of the higher nobility who won fame as an outstanding scientist.”

“If, as is sometimes supposed, science consisted in nothing but the laborious accumulation of facts, it would soon come to a standstill, crushed, as it were, under its own weight…The work which deserves, but I am afraid does not always receive, the most credit is that in which discovery and explanation go hand in hand, in which not only are new facts presented, but their relation to old ones is pointed out.” Lord Rayleigh, 1884

Things that should be done: Address uncertainties by reviewing existing evidence systematically. Ensure that new research begins and ends by referring to systematic reviews of other relevant evidence.

How can we expect patients and the public to trust that we have their interests at heart if we and our professional institutions continue to acquiesce in our failure to make systematic use of the results of research that they have funded?

Promote research on the effects of treatments…

Promote research on the effects of treatments… …but only if it meets scientific and ethical principles

Sir Paul Nurse, President of the Royal Society, urged research funders to trust scientists to decide where public funding should be directed. In speaking about initiatives that arise from reviews to decide priorities Sir Paul warned that “such initiatives may attract less creative and effective scientists who simply follow where resources are being made available.” The case for greater interaction between scientists and business was also made as being crucial to rebuilding an innovation-based economy. De Havilland Alert, 1 Dec 2011.

Finally, a challenge for you We need to be more efficient in preparing and updating systematic reviews.

The decline in costs of computer memory and gene sequencing Computer power: costs halve every 18 Months Gene sequencing faster! Why not systematic reviews?

The steps in a Systematic Review What is current time for each step? How can we make each easier/faster? Standardize Streamline Computerize

More efficient searching Steve Pritchard Alison Weightman

Better abstract screening Highlight the search terms that had been used - yellow in example Highlight methodological terms (from a supplementary list NOT used in the search) – pink in example. Allowing easy marking of definite, possible, and definitely not abstracts, and comparison of these markings across two reviewers. [Sente 6]

Your ideas please! For any review step, do you have tips to: Standardize? Streamline? Automate? Ideas, please, to: pglaszio@bond.edu.au or colesbm@cardiff.ac.uk