1. Does it Matter — Timeliness or Accuracy of Results? Results of a Research Program on Rapid Reviews Andrea C. Tricco PhD MSc CADTH Symposium 2015

2.  I have no actual or potential conflict of interest in relation to this presentation. Conflict of interest 2

3.  Background  Scoping Review of Rapid Reviews  International Survey of Rapid Review Producers  International Consensus-building Exercise with Key Stakeholders  Ultimate Goal of Our Research Program  Questions Outline 2

4. Background  Rapid reviews are a form of knowledge synthesis in which components of the systematic review process are simplified or omitted to produce information in a timely manner [Khangura 2012]  Evidence suggests that decision-makers are currently using rapid reviews to inform their decision-making processes  Few studies have examined the methodological characteristics of rapid reviews  We aimed to conduct a research program on rapid reviews to clarify the methods and perceptions of rapid review approaches 3

5. A scoping review of rapid review methods Submitted to BMC Medicine Methods project 1:

6. Objective & methods  Objective –To examine rapid review approaches, guidance, impact, and comparisons through a scoping review  Methods –Used methodologically rigorous scoping review methods proposed by Arksey and O’Malley (2005) –MEDLINE, EMBASE, the Cochrane Library, internet websites of rapid review producers, and reference lists were searched to identify articles for inclusion –Two reviewers independently screened citations and full-text articles, and abstracted data 5

7. Results Study flow figure N=3397 citations from MEDLINE, EMBASE, Cochrane Library, previous systematic reviews, and grey literature N=3397 citations from MEDLINE, EMBASE, Cochrane Library, previous systematic reviews, and grey literature N=3135 excluded titles and abstracts N=100 rapid reviews (plus 1 companion report) N=100 rapid reviews (plus 1 companion report) N=262 potentially relevant full-text articles N=161 excluded full-text reports 6

8. Results (continued) Word cloud figure for the frequency of terms 7

9. Study Characteristics No. of Rapid Reviews (n=100) Year of Publication 1997-20002 2001-200410 2005-200830 2009-201251 20135 Not Reported4 Corresponding author’s continent Europe (including UK)58 North America (Canada & United States)20 Australia15 Multiple continents3 Asia1 South America1 Not Reported2 Study characteristics Results (continued) 8

10. Study CharacteristicsNo. of Rapid Reviews (n=100) Article Type Application (82 with methods)84 Development7 Impact6 Comparison4 Topic of Review Intervention62 (74%) Frequency10 (12%) Causal association4 (5%) Diagnosis4 (5%) Patient experience2 (2%) Screening2 (2%) Not applicable16 Study characteristics (continued) 9

11. Results (continued) Methods characteristics 10

12. Results (continued) ApproachLiterature searchSearch limitScreeningData abstraction Risk of bias appraisal 1 >1 database, published only Both date and language One reviewer One person abstracts, other verifies One person assesses, other verifies 2 Updating the literature search of a previous review, published only NoneOne reviewer Not performed 3 >1 database, grey literature Both date and language One reviewer Not performed 4 >1 database, grey literature Either date or language One reviewer Not performed 5 >1 database, grey literature DateOne reviewer 5 most frequent rapid review approaches 11

13. Conclusions  Numerous rapid review approaches were identified  Little consistency exists in the field  Poor quality of reporting was observed  Prospective study comparing the results from rapid reviews to those obtained through systematic reviews is warranted. 12

14. An international survey of rapid review producers Submitted to the J of Clin Epi Methods project 2:

15. Objective and methods  Objective –To determine different rapid review approaches used by rapid review producers  Methods –International survey of 63 organizations administered via FluidSurvey –Survey pilot-tested prior to administration –Reminders to non-respondents sent every 2 weeks –Contacted through postal mail if no response –$10 financial incentive from Amazon 14

16. Results Study flow figure of participants 63 organizations contacted 41 responses (65%) 22 did not respond 15

17. Results (continued) Summary of rapid review characteristics Review Characteristics Count (%) Duration of Review (weeks) 1-1262 (70) 12-2618 (20) 26-366 (7) ≥ 522 (2) Commissioning Agency Government Agencies & Health Ministries69 (78) Healthcare Organizations, Hospitals & Community Health51 (58) Healthcare Professionals13 (15) Industry4 (5) Target Audience Government Agencies & Health Ministries73 (83) Healthcare Professionals46 (52) Patients19 (22) Researchers21 (24) 16

18. Results (continued) Rationale providedCount (%) Decision-maker timeline57 (66) Focused or brief question8 (9) Lack of resources5 (6) Increase efficiency (including timeliness)4 (5) Broad understanding of an area4 (5) Identify topics requiring a systematic review 2 (2) Update a systematic review2 (2) Well-established intervention1 (1) Evidence is unclear1 (1) Rationale for conducting rapid reviews 17

19. Results (continued) Review StageMost frequent streamlined approachCount (%) Identifying relevant studiesUsed previous review(s) as a starting point79 (92) Limitations on search strategyLimited review by date of publication75 (88) Study selectionScreening conducted by ONE reviewer only68 (85) Data AbstractionData abstraction performed by ONE reviewer only67 (84) Quality (risk of bias) appraisal process Risk of bias assessed by ONE reviewer only68 (86) SynthesisNarrative summary75 (90) Summary results of most frequently streamlined approaches 18

20. Conclusions  Results are consistent with scoping review of rapid reviews  Rapid reviews usually conducted in 1-12 weeks  Government agencies and health ministries are primary commissioners  Many different streamlined methods are being used. 19

21. International consensus-building exercise regarding rapid reviews Submitted to the J of Clin Epi Methods project 3:

22. Objective & methods  Objective –To conduct a consensus-building exercise to select a rapid review approach that will be prospectively tested in a diagnostic study  Methods –Editors, healthcare providers, researchers, policy-makers, and industry stakeholders (including participants of the CADTH rapid review summit) –Asked to rank the 5 most frequent rapid review approaches identified in our scoping review and survey using FluidSurvey –Results presented to participants, followed by a facilitated discussion (online and in-person) and re-ranking exercise using FluidSurvey 21

23. Results Study flow figure of participants 26 individuals contacted 113 responses (72%) 3 did not respond 130 individuals contacted 40 did not respond 22 Online delphi In-person delphi

24. Summary of ranking results by approach Results (2) Rapid review approach FeasibilityTimelinessComprehensivenessRisk of Bias Approach 1 1 st 2 nd 4 th 1 st Approach 2 2 nd 1 st 5 th Approach 3 3 rd 2 nd Approach 4 4 th 2 nd 4 th Approach 5 5 th 1 st 3 rd *Ranked based on the distribution of "very" and "extremely" on the 7-point Likert scale, except Risk of Bias was ranked on distribution of “not at all” and “very” 23

25.  The highest ranked method was: Approach 1 ‒ Most feasible (72%, n=81 out of 113 responses) ‒ Lowest perceived risk of bias (12%, n=12 out of 103) ‒ 2nd in timeliness (37%, n=38 out of 102) ‒ 5th in comprehensiveness (5%, n=5 out of 100)  We will use the information from the consensus-building exercise to select the rapid review approach for a prospective study. Conclusion 24

26. Ultimate goal of this research “We can give you results within 4 months, but the meta- analysis estimates will be inaccurate by 35%” Rapid review definition (Shannon Kelly) Identify 5 frequently used methods Diagnostic study to test a rapid review approach Identify and characterize rapid review methods 25

27.  Will use these results from our research program to inform a diagnostic study: −Index test: Rapid Review Approach −Reference standard: Systematic Review  Collaboration between 3 Canadian Knowledge Synthesis Centers  Targeting CIHR and PCORI Proposed diagnostic study Diagnostic Accuracy of Rapid reviews compared To Systematic reviews (DARTS) 26

28. Summary  This research program provides up-to-date information on rapid review methods reported in the literature, as well as stakeholder experiences and perceptions regarding rapid reviews  Poor quality of reporting was observed in the literature  Rapid reviews have many names and approaches and some methods might be more desirable than others  A prospective study comparing the results of rapid reviews to those obtained through systematic reviews is necessary. 27

29. Acknowledgements  Funding: Canadian Institutes of Health Research/Drug Safety and Effectiveness Network −Operating grant to update 2 systematic reviews, international survey, Delphi −New investigator award  Research team: Jesmin Antony, Wasifa Zarin  Co-investigators: Drs. Straus, Moher, Hutton, Sherifali 28

30. Questions? triccoa@smh.ca 30