Evidence-Based Research Karen A. Robinson Johns Hopkins University Evidence-Based Research Network Bergen Norway 3 December 2014

Scientific knowledge is cumulative, and evidence should be cumulated scientifically. Research Synthesis: Formal methods for summarizing evidence for a research question by collecting studies addressing similar questions and evaluating the consistencies and variability in these studies Cornerstone of evidence-based medicine

Evidence-based Medicine “The conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence-based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research.” - Sackett et al., 1996 Sackett et al. Evidence based medicine: what it is and what it isn’t. BMJ 1996;312(7023):71-2.

Robinson, Karen A. Use of prior research in the justification and interpretation of clinical trials. The Johns Hopkins University, ProQuest, UMI Dissertations Publishing, “ While the use of research synthesis to make evidence- informed decisions is now expected in health care, there is also a need for clinical trials to be conducted in a way that is evidence-based. Evidence-based research is one way to reduce waste in the production and reporting of trials, through the initiation of trials that are needed to address outstanding questions and through the design of new trials in a way that maximizes the information gained.”

Evidence-Based Research Using evidence to inform research so that it is addressing questions that matter in a valid, efficient and accessible manner.

The scientific and ethical justification for studies requires the use of prior research “Medical research involving human subjects must conform to generally accepted scientific principles, [and] be based on a thorough knowledge of the scientific literature,...” - Paragraph 11, Declaration of Helsinki

Emanuel EJ, Wendler D, Grady C. What makes clinical research ethical? JAMA May 24-31;283(20): Justification and interpretation In embarking on a trial- Requirements for ethical clinical research: Social or scientific value Favorable risk-benefit ratio At completion of trial – Integration of new results with existing evidence: Place results within context of existing knowledge Prior research is needed to interpret the results of a trial and what they mean for our understanding research question

Do researchers use prior research in providing the justification and interpretation of the results of trials?

52 additional trials > 5,600 patients Fergusson, D; Glass, K C; Hutton, B, and Shapiro, S. Randomized controlled trials of aprotinin in cardiac surgery using clinical equipoise to stop the bleeding. Clinical Trials. 2005; 2(3)

“No trial is an island, entire of itself; every trial is a piece of the continent, a part of the main.” - Clarke and Chalmers, 1998 Classification of discussion sections of reports of RCTs in 5 major medical journals Date of Publication (Number of reports) 1997 (n=26) First trial1 Contained an updated systematic review integrating the new results 2 Discussed a previous review but did not attempt to integrate new results 4 No apparent systematic attempt to set new results in context of other trials 19

Classification of discussion sections of reports of RCTs in 5 major medical journals Date of Publication (Number of reports) 1997 (n=26) 2001 (n=33) 2005 (n=18) 2009 (n=29) 2012 (n=35) First trial13352 Contained an updated systematic review integrating the new results Discussed a previous review but did not attempt to integrate new results No apparent systematic attempt to set new results in context of other trials

Citation of Similar Trials

Objective To assess the extent of the use of prior research by measuring the citation of prior related trials in reports of randomised controlled trials (RCTs). Specifically, do reports of RCTs cite prior similar trials?

Methods Briefly: 1.Created cohorts of RCTs addressing the same question 2.Determined if RCTs in same cohort cite each other

Methods For each RCT, we calculated: Prior Research Citation Index (PRCI) Number of RCTs cited Total number of potential RCTs to cite Sample Size Citation Index (SSCI) Total number of participants in cited RCTs Total number of participants in potential RCTs to cite

Prior Research Citation Index Median PRCI 0.21 (95% CI 0.2 to 0.25, Mean =0.29, n=1,523 RCTs) PRCI = Number of RCTs cited / Total number of potential RCTs to cite Publication DateNumber of RCTs Median PRCI 2 years or earlier1, years or earlier years or earlier

Of 1,101 RCTs with 5 or more prior trials to cite 511 (46%) cited no prior trial or only one prior trial Citation of zero or one prior RCT

Citation of prior trials by sample size Sample Size Citation Index (SSCI) calculated for 1,261 RCTs Median SSCI 0.24 (95% CI 0.21 to 0.27) SSCI =Number of participants in RCTs cited / Total number of participants in potential RCTs to cite

Conclusions About 80% of prior studies about the same question were NOT cited in reports of RCTs – About half of the RCTs cited zero or one prior trial Information from 76% of participants enrolled in prior trials not acknowledged

Why happening? This is better now.

Rich get Richer If trial cited, 60% probability it will be cited again If trial NOT cited – more than 60% probability that it will remain uncited

What are causes of lack of citation? Cite systematic reviews instead Studies assessing factors that predict citation: – In 8 of 10 studies, factors other than methodological features of the study predicted subsequent citation – Other factors were significant predictors of citation, such as: geographical location of author results seen as supportive

Why and better now? Combined new search results (2011) with existing database Citation patterns in pharmacological CVD RCTs 86 meta-analyses with 580 trials Veronica Ivey Sawin. Recognition of prior research: Citation patterns in reports of clinical trials in cardiovascular research. ScM Thesis. Johns Hopkins University Bloomberg School of Public Health. 2013

Results – Not better Results in CVD: Median PRCI 0.25 (95% CI 0.23 to 0.27) Median SSCI 0.31 (95% CI 0.25 to 0.36) No difference between 2004 and 2011 datasets Approximately 75% of prior trials ignored and only 30% of trial participants represented

Difference over time? Black = PRCI Orange = SSCI

Supportive vs not? Trials were about 45% more likely to be cited by subsequent trials if results supportive. RR 1.45 (95% CI to 1.612) Inadequate, and biased, consideration of prior research remains – implications include ethically unjustifiable trials, wasted research and unnecessary risk for trial participants.

54% were not a single connected network: – 39% two islands – 4% 10 or more separate islands Robinson KA, Dunn AG, Tsafnat G, Glasziou P. Citation networks of trials: Feasibility of iterative bidirectional citation searching. Journal of Clinical Epidemiology 2014;67(7): “No trial is an island, entire of itself; every trial is a piece of the continent, a part of the main.” - Clarke and Chalmers, 1998

What should happen? Use evidence to: – Identify worthwhile questions – Design valid and informative studies – Report results within context of what is known

Identifying Questions For systematic reviews:

Identifying Questions For studies, use: – guidelines – systematic reviews

Identified gaps from evidence-based guidelines as: – Insufficient evidence, no recommendations made – Insufficient evidence, consensus recommendations made – “Needs further research”

Identifying research gaps from guidelines: 62 gaps identified, of which only one was fully specified (PICO) Only about 20% of the gaps were called out by guideline committees as research gaps.

Framework for Determining Research Gaps During Systematic Reviews Robinson KA, Saldanha IJ, Mckoy NA. Frameworks for determining research gaps during systematic reviews. Methods Future Research Needs Report No. 2 AHRQ Publication No. 11-EHC043-EF. Rockville, MD: Agency for Healthcare Research and Quality. June PMID: Robinson KA, Saldanha IJ, Mckoy NA. Development of a framework to identify research gaps from systematic reviews. Journal of Clinical Epidemiology 2011 Dec;64(12): PMID: Robinson KA, Akinyede O, Dutta T, Sawin VI, Li T, Spencer MR, Turkelson CM, Weston C. Framework for determining research gaps during systematic review: Evaluation. Rockville (MD): Agency for Healthcare Research and Quality (US) 2013 Feb. Report No. 13- EHC019-EF. PMID: Identification of gaps from systematic reviews in a systematic way Framework to facilitate the identification and characterization of gaps: Where the evidence falls short, as well as how and why

We applied the framework to 50 systematic reviews

Framework- Results 144 review questions in the 19 EPC reports (average 5.5 questions) and 31 Cochrane reviews (average 1.3 questions). A total of approximately 600 unique research gaps were identified and characterized. Number of gaps per question: average per question for EPC reports (95% CI 9.31 to 16.19) and 8.5 per question for Cochrane reviews (95% CI 6.23 to 10.32). Insufficient information (Reason A) was most frequent reason for the gaps, followed by inconsistency (C), not the right information (D), and biased information (B).

Challenges in Practicing EBR Ever increasing number of studies Methodologically poor studies Reporting bias Resources – time and money Skills and competencies

What do we need? Ongoing research – Empirical research to improve research prioritization, including to systematically identify gaps from systematic reviews and guidelines – Research synthesis methods – Implications of not practicing EBR

What do we need? Adequate support for research synthesis – Training – Support – Academic credit Expectations at funding agencies, ethics boards and journals – Criteria for ethics panels and funding agencies to determine if proposals follow EBR practices

What should happen? Use evidence to inform research so that it is answering questions that matter in a valid, efficient and accessible manner. Evidence-Based Research Worthwhile questions; Valid and informative studies; Results within context of what is known

All scientific work is incomplete…that does not confer upon us a freedom to ignore the knowledge we already have… -Sir Austin Bradford Hill Hill AB. The reasons for writing. BMJ. 1965;4:870 Evidence-Based Research Network