1. Systematic reviews – informing policy, practice and research
Iain Chalmers
Coordinator, James Initiative
SysNet Annual Lecture, Cardiff University
23 April 2012

3. to clinicians & patients? 85% Research waste = over $85 Billion / year
Questions relevant
to clinicians & patients?
Appropriate design
and methods?
Accessible
full publication?
Unbiased and
usable report?
Low priority questions addressed
Important outcomes not assessed
Clinicians and patients not involved in setting research agendas
Over 50% studies designed without reference to
systematic reviews of existing evidence
Over 50% of studies fail to take adequate steps to reduce biases, e.g. unconcealed treatment allocation
Over 50% of studies never published in full
Biased under-reporting of studies with disappointing results
Over 30% of trial interventions not sufficiently described
Over 50% of planned study outcomes not reported
Most new research not interpreted in the context of systematic assessment of other relevant evidence
50%
50%
50%
85% Research waste = over $85 Billion / year 3 3

4. to clinicians & patients? 85% Research waste = over $85 Billion / year
Questions relevant
to clinicians & patients?
Appropriate design
and methods?
Accessible
full publication?
Unbiased and
usable report?
Low priority questions addressed
Important outcomes not assessed
Clinicians and patients not involved in setting research agendas
Over 50% studies designed without reference to
systematic reviews of existing evidence
Over 50% of studies fail to take adequate steps to reduce biases, e.g. unconcealed treatment allocation
Over 50% of studies never published in full
Biased under-reporting of studies with disappointing results
Over 30% of trial interventions not sufficiently described
Over 50% of planned study outcomes not reported
Most new research not interpreted in the context of systematic assessment of other relevant evidence
50%
50%
50%
85% Research waste = over $85 Billion / year 4 4

5. Does anyone here think that researchers should NOT review existing evidence systematically before developing treatment guidelines and planning new research?

6. Patients have suffered and died unnecessarily, and resources for health care and health research have been wasted, because existing research evidence has not been reviewed systematically. 6

7. JAMA 1992;268:

9. The human costs of failing to cumulate evidence from research scientifically “Advice on some life-saving therapies has been delayed for more than a decade, while other treatments have been recommended long after controlled research has shown them to be harmful.” Antman et al. JAMA 1992;268:240-8.

10. Oxford Textbook of Medicine, 2nd edn, “The clinical benefits of thrombolysis … remain to be established.”

12. Sometimes very important advances in knowledge come from systematic reviews of existing evidence

15. "It is essential that existing sources of evidence, especially systematic reviews, are considered carefully prior to undertaking research. Research which duplicates other work unnecessarily, or which is not of sufficient quality to contribute something useful to existing knowledge, is in itself unethical." Department of Health. Research Governance Framework for Health and Social Care, 2001, para 2.3.1

16. TGN1412
TGN 1412

18. 18

19. Cumulative estimate from trials of the effect of aprotinin on perioperative blood transfusion,
Aprotinin better Aprotinin worse 19

20. New trials of aprotinin ignored previous trials20

21. But was it the right question? Is tranexamic acid better than aprotinin?
A comparison of aprotinin and lysine analogues in high-risk cardiac surgery.
Ferguson et al. N Engl J Med May 29;358(22):

22. * *actually, the non-use
Only 11 of 24 responding authors of trial reports that had been added to existing systematic reviews were even aware of the relevant reviews when they designed their new studies. 22

23. 23

25. STUDIES IN ANIMALS
20 animal studies: “The results of this review did not show convincing evidence to substantiate the decision to perform trials with nimodipine in large numbers of patients.
Stroke 2001;32:
Horn J, Limburg M.
Calcium antagonists for acute ischemic stroke.
The Cochrane Database of Systematic Reviews, 2000
“46 trials were identified of which 28 were included (7521 patients). No effect of calcium antagonists on poor outcome at the end of follow-up (OR 1.07, 95% CI 0.97/1.18), or on death at end of follow-up (OR 1.10, 95% CI 0.98/1.24) was found.”
STUDIES IN HUMANS

26. 2005
“…there is a relative scarcity of systematic reviews…” “…it would therefore be desirable to undertake further systematic reviews and meta-analyses to evaluate more fully the predictability and transferability of animal models.”

30. Can this policy be defended on scientific, ethical, or economic grounds?

31. No subsequent contact made.
Richard Hughes, personal communication,
30 Nov 2011

37. To reiterate Patients have suffered and died unnecessarily, and resources for health care and health research have been wasted, because existing research evidence has not been reviewed systematically. 37

38. Austin Bradford Hill, Four questions to which readers want answers when reading reports of research Why did you start? 2. What did you do? 3. What answer did you get? 4. And what does it mean anyway? 38

41. Lord Rayleigh,
“One of the very few members of the higher nobility who won fame as an outstanding scientist.”

43. “If, as is sometimes supposed, science consisted in nothing but the laborious accumulation of facts, it would soon come to a standstill, crushed, as it were, under its own weight…The work which deserves, but I am afraid does not always receive, the most credit is that in which discovery and explanation go hand in hand, in which not only are new facts presented, but their relation to old ones is pointed out.” Lord Rayleigh, 1884

44. Things that should be done: Address uncertainties by reviewing existing evidence systematically. Ensure that new research begins and ends by referring to systematic reviews of other relevant evidence.

45. How can we expect patients and the public to trust that we have their interests at heart if we and our professional institutions continue to acquiesce in our failure to make systematic use of the results of research that they have funded?

47. Promote research on the effects of treatments…

48. Promote research on the effects of treatments…
…but only if it meets scientific and ethical principles

49. Sir Paul Nurse, President of the Royal
Society, urged research funders to
trust scientists to decide where
public funding should be directed.
In speaking about initiatives that arise from reviews to decide priorities Sir Paul warned that “such initiatives may attract less creative and effective scientists who simply follow where resources are being made available.” The case for greater interaction between scientists and business was also made as being crucial to rebuilding an innovation-based economy. De Havilland Alert, 1 Dec 2011.

52. Finally, a challenge for you We need to be more efficient in preparing and updating systematic reviews.

53. The decline in costs of computer memory and gene sequencing
Computer power: costs halve every 18 Months
Gene sequencing faster!
Why not systematic reviews?

54. The steps in a Systematic Review
What is current time for each step?
How can we make each easier/faster?
Standardize
Streamline
Computerize

55. More efficient searching
Steve
Pritchard
Alison
Weightman

58. Better abstract screening
Highlight the search terms that had been used - yellow in example
Highlight methodological terms (from a supplementary list NOT used in the search) – pink in example.
Allowing easy marking of definite, possible, and definitely not abstracts, and comparison of these markings across two reviewers.
[Sente 6]

59. Your ideas please! For any review step, do you have tips to:
Standardize?
Streamline?
Automate?
Ideas, please, to: or