1. “MEETING THE CHALLENGE OF MAINTAINING COCHRANE REVIEWS” Author: Dr David Tovey FRCGP Editor in Chief, The Cochrane Library

2. Plan AHRQ 2012 Annual Conference 2  Introduction  Previous Cochrane project  “Fit for purpose” project  Future challenges and opportunities for co- operation  Questions

3.  Sally Hopewell, Yemisi Takwoingi, Alex Sutton, Rachel Marshall and Bazian Ltd.  This project was funded by the NIHR Cochrane – NHS Engagement Award Scheme (project number 10/4000/01). The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the Department of Health. Acknowledgements

4.  Cochrane Musculoskeletal Group  Cochrane Pain, Palliative and Supportive Care Group  Cochrane Infectious Disease Group  Cochrane Wounds Group  Cochrane Neonatal Group  Cochrane Airways Group  Karla Soares-Weiser 4 AHRQ 2012 Annual Conference

5. Introduction: Protecting Archie Cochrane’s vision AHRQ 2012 Annual Conference 5  “It is surely a great criticism of our profession that we have not organised a critical summary by speciality and subspecialty adapted periodically of all relevant randomised controlled trials”

6. Introduction: Protecting Archie Cochrane’s vision AHRQ 2012 Annual Conference 6  “It is surely a great criticism of our profession that we have not organised a critical summary by speciality and subspecialty adapted periodically of all relevant randomised controlled trials” Credible & Safe

7. Introduction: the size of the challenge AHRQ 2012 Annual Conference 7

8. Introduction: the size of the challenge AHRQ 2012 Annual Conference 8

9. Introduction: the size of the challenge AHRQ 2012 Annual Conference 9

10. Introduction: the size of the challenge AHRQ 2012 Annual Conference 10 “The boulder in my rucksack”

11. Previous Cochrane projects: the updating officer project AHRQ 2012 Annual Conference 11  8 reviews selected  On average these reviews each took 6.4 months (range 3-11 months) to update from receipt of the search strategy to submission for editorial review.  The main challenges:  lack of familiarity with individual Review Groups’ methods  out of date methodology within the existing reviews  lack of subsequent author commitment  lack of wish for ongoing commitment  Would imply the need for a small army of updating officers!!

12. “Fit for purpose”project AHRQ 2012 Annual Conference 12 Three objectives of the project:  To work with external stakeholders to develop a customisable prioritisation tool – prioritisation by topic  To develop a decision tool for determining whether and when to update Cochrane Reviews – prioritisation by status  To explore whether targeted consultancy could facilitate updating and identify process efficiencies

13. “Fit for purpose”project AHRQ 2012 Annual Conference 13 Three objectives of the project:  To work with external stakeholders to develop a customisable prioritisation tool – prioritisation by topic  To develop a decision tool for determining whether and when to update Cochrane Reviews – prioritisation by status  To explore whether targeted consultancy could facilitate updating and identify process efficiencies

14. Methods  Project conducted by Bazian, co-applicants in the NHS engagement award.  Stakeholders in the NHS (clinicians, consumers, commissioners etc.) formed a panel.  The panel had two meetings, and communicated via email before and after meetings.

15. Results: prioritisation criteria in the tool

16.  The tool was tested on 19 Cochrane Reviews, which found:  There is a level of judgement when assigning scores.  A certain level of knowledge of the field is helpful when assigning scores  Needs to be customised (notable absentee parameters..)  Recommended that the results of using the tool are discussed with external stakeholders including patient and carer representatives. Results: testing the tool

17. “Fit for purpose” project AHRQ 2012 Annual Conference 17 Three objectives of the project:  To work with external stakeholders to develop a customisable prioritisation tool – prioritisation by topic  To develop a decision tool for determining whether and when to update Cochrane Reviews – prioritisation by status  To explore whether targeted consultancy could facilitate updating and identify process efficiencies

18.  We refined and amalgamated two complementary methodologies for prioritising systematic review updates :  a qualitative tool based on a broad range of updating signals (Loudon 2008).  formal statistical methods which assess when the inclusion of new studies is likely to change a review’s conclusions (Sutton 2009). Methods

19. Output: Decision tool

20. Step 1: Is the clinical question answered or no longer relevant?

21. Step 2: Are there any new factors to consider?

22. Steps 3 & 4: Are there new studies? Are the conclusions likely to change?

23. Statistical prediction tool

24. About metarank  Based on minimal information on the new evidence  assumes an update strategy is in place such that number of new studies and their sample sizes are known  ‘Signals’ of the need to update implemented as a STATA user-written function  Performs simulation of several meta-analyses, each with one or more new studies of different sizes

25. Metarank output  Summary of results for each meta-analysis includes details of the original meta-analysis and the signals detected  Table with all reviews in the dataset ranked in order of priority  by a given signal  or the total of all signals triggered  Includes descriptive information for the collection e.g.  average number of trials and participants in the collection of meta- analyses  number of trials in the largest meta-analysis  largest number of participants in a meta-analysis

27. Advantages of metarank  Assessment of the likelihood of any definable criteria changing in a meta-analysis  Ongoing studies can be taken into account  aid in predicting potential ‘shelf life’ of a review in the light of new or accruing evidence  To provide information with respect to further research needed such as the number of new trials and the number of participants  useful information to end users of reviews in assessing the stability or validity of a review

28. Limitations of metarank  Relies on availability of some information on new trials  Simplistic approach used in deriving the predictive distribution for new studies  Other issues external to a review may need to be taken into account such as  the rate at which new evidence on a topic evolves  public heath significance etc.  No criteria developed to establish when enough evidence has accrued on a given topic and review that it is deemed decisive and not worth conducting further primary research

29.  The decision tool provides a set of criteria that can be used to assess whether to update a Cochrane Review.  The tool can be applied to a single Cochrane Review or can be used to prioritise a suite of reviews (e.g. those from an individual Cochrane Review Group) Decision tool: summary

30. Results: testing the tool AHRQ 2012 Annual Conference 30  One Cochrane Review Group so far  Cautiously positive - “helpful structure to assess each review’s eligibility for updating” “a transparent way to explain decisions around updating to stakeholders”  Time consuming but might improve with experience  Some assumptions around the statistical tool “ too crude”  Need to have wider experience and use

31. “Fit for purpose” project AHRQ 2012 Annual Conference 31 Three objectives of the project:  To identify those Cochrane Reviews that NHS stakeholders regard as the most important to update – prioritisation by topic  To develop a decision tool for determining whether and when to update Cochrane Reviews – prioritisation by status  To explore whether targeted consultancy could facilitate updating and identify process efficiencies

32. Aims  We aimed to provide short and focused periods of assistance, to incentivise authors, and to address barriers to updating.  The intended approach was to retain both the responsibility for producing the update within the authors, and editorial support for within the CRGs. 32 AHRQ 2012 Annual Conference

33. Methods  One-year project (July 2010 to July 2011)  Involved 6 CRGs  Each CRG offered 37 days’ of assistance for updating 1 to 6 Cochrane Reviews (between November 2010 and April 2011)  Tasks performed by reviewers from inside and outside Cochrane without specific topic knowledge 33 AHRQ 2012 Annual Conference

34. Tasks offered to CRGs and authors 34 AHRQ 2012 Annual Conference

35. Results: assessment of tasks 35 AHRQ 2012 Annual Conference

36. Results: CRG and author feedback  Evaluation form: all CRGs responded, most authors responded (1 on holiday, 1 no response)  Scale 1-5 (1 greatly improved, 5 greatly worsened)  All authors and managing editors who responded to the question on rolling out an updating service for The Cochrane Collaboration were in favour of an updating service (2 CRGs did not respond) AuthorsCRGs Author motivation1.61.8 Speed of update1.62.5 Quality of update1.82.75 Overall impression of assistance 1.441.5 36 AHRQ 2012 Annual Conference

37. Results: status of updates  We worked on 14 Cochrane Review updates  Mean time spent: 35.5 hours (median 26.71 hours, range 4.5 to 109.75 hours)  Lots of activity but....  In October 2011, no updates yet published 37 AHRQ 2012 Annual Conference

38. Future challenges: AHRQ 2012 Annual Conference 38  Whose priorities?  Expectations and the increasing professionalisation of systematic reviews  The rise and rise of methods advances  How to balance updates versus new reviews?  Authorship of updates

39. Conclusion AHRQ 2012 Annual Conference 39  Updating is critical to patient safety and credibility  Increasing recognition of need to prioritise (but this isn’t as easy as it seems)  Transparency to user/reader is critical  Different approaches but no “one size fits all” solution

40. Conclusion: How can we work more effectively together? AHRQ 2012 Annual Conference 40  Sharing intelligence  “Keep up” initiative  Shared surveillance  Sharing the workload  Sharing data  “Wiki” approaches  Novel approaches  Use of data mining and semantic technologies  Limited search and “Summary updates”

41. AHRQ 2012 Annual Conference 41 thank you for listening dtovey@cochrane.org

42. Authorship of updates AHRQ 2012 Annual Conference 42  Scenario: A review is updated that originally included 11 studies and now has 17. The text includes 35% the same text as the previous version, across all sections. In addition a summary of findings table has been added. The author team has changed completely and none of the previous authors have contributed to the update.  Questions How should the work of the previous authors be credited?

43. ICMJE guidance on contributorship AHRQ 2012 Annual Conference 43 Authorship credit should be based on 1. Substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; 2. Drafting the article or revising it critically for important intellectual content; and 3. Final approval of the version to be published. Authors should meet conditions 1, 2, and 3.

44. Authorship of updates AHRQ 2012 Annual Conference 44 A. As authors of the updated review B. Lead author included on the update C. Should be negotiated on an individual basis D. Original authors not included as authors but acknowledged within the review E. Some other approach