1. New methodological standards for Cochrane reviews first output from the MECIR project edit Rachel Churchill Co-ordinating Editor representative on Steering Group School of Social and Community Medicine, University of Bristol Julian Higgins Methods Groups representative on Steering Group MRC Biostatistics Unit, Cambridge, UK & Centre for Reviews and Dissemination, University of York

2. Outline Background and purpose of MECIR Development of methodological standards for conduct of intervention reviews The first set of standards Some examples Next steps and discussions around implementation

3. Background Coordinated by MARS (Methods Application and Review Standards) – Co-convened by Julian Higgins and Rachel Churchill to facilitate interaction between Methods Groups and CRGs; supports Editor in Chief and CEU Coordinating Group from MARS and the CEU (Rachel Churchill, Julian Higgins, Jackie Chandler, David Tovey and Toby Lasserson)

4. MECIR Stands for Methodological Expectations of Cochrane Intervention Reviews Purpose is –to specify methodological expectations for Cochrane protocols, reviews, and updates of reviews on the effects of interventions –to ensure that these methodological expectations are supported and implemented across the Collaboration

5. Development process MECIR Working Groups 1.Developing a question and deciding the scope of the review 2.Searching for studies 3.Selecting studies and collecting data 4.Assessing risk of bias in studies 5.Analysing data and undertaking meta-analyses 6.Interpretation and presenting results Coordinating Group 1. WGs submitted long list of items 2. WGs asked to consider Mandatory items vs Highly Desirable vs Good practice and Rationale for each Collaboration-wide consultation All registered entities (16/6/11 – 8/8/11) 51% responded - CRGs (35/53) - MGs (8/15) - Centres (5/14) - Centre branches (2/15) - Fields (5/11) - CEU, TWG, Author rep

8. Consultation process Further work by Coordinating Group, before extensive consultation and feedback from all Cochrane entities All registered entities (16 June ‘11 – 8 August ‘11) 51% entities responded –CRGs (35/53) –MGs (8/15) –Centres (5/14) –Centre branches (2/15) –Fields (5/11) –CEU, TWG, Author rep

9. Consultation feedback (Aug – Oct 2011) High levels of support for most items Decision around status item description, expectation and rationale (merging or omitting as appropriate) and provided associated Handbook reference for each standard 100 standards, reduced to 80 Provided responses to comments and final decision regarding status, description, and rationale for each item (see Annex 2) www.editorial-unit.cochrane.org/mecir

10. Summary of MECIR standards 80 standards for the conduct of CIRs structured around: –Setting research question to inform the scope of the review –Setting the eligibility criteria for including studies in the review –Selecting outcomes to be addressed –Planning the review methods –Searching for studies –Selecting studies into the review –Collecting data from included studies –Assessing risk of bias in included studies –Synthesizing the results of included studies –Summarizing the findings –Reaching conclusions

11. Some examples

12. Example of one we got right

13. Pre-defining clear and unambiguous criteria for interventions and comparators (standard 7) (Mandatory) Agreed/Disagreed: CRGs: 30 / 0 Centres: 7 / 0 Fields: 4 / 0 Methods Groups: 6 / 0 Left unchanged Define in advance the eligible interventions and the interventions against which these can be compared in the included studies

14. Example of one we had to work on

15. Excluding studies without useable data (standard 40) [Draft] (Mandatory) Agreed/Disagreed: CRGs: 23 / 5 Centres: 6 / 1 Fields: 4 / 0 Methods Groups: 6 / 0 Three suggestions to change to Highly Desirable Do not exclude studies on the basis of there being no usable data (either because the data are inconvenient format or because they are not reported at all)

16. Excluding studies without useable data (standard 40) Hugely important Clarify that this relates to the formal “exclusion” of studies from the review We will not implement this in our group RevMan will protest if a study is ‘included’ with no data at all agree if the outcomes of interest were measured as an outcome somewhat confusing should be rephrased

17. Excluding studies without useable data (standard 40) (Mandatory) Include studies in the review irrespective of whether measured outcome data are reported in a ‘usable’ way.

18. Some key ones

19. Some key conduct standards Consider important potential adverse effects: Mandatory Clarify role of outcomes: Mandatory Search trials registers: Mandatory Double (independent) data extraction: Mandatory Risk of bias tables: Mandatory Summary of findings tables: Highly desirable The five GRADE considerations (study limitations, consistency of effect, imprecision, directness and publication bias) to assess the quality of the body of evidence: Mandatory

20. The new stuff

21. Standards not currently in the Handbook Rerun or update searches for all relevant databases within six months before publication of the review or review update, and screen the results for potentially eligible studies: Mandatory, BUT Incorporate fully any studies identified in the rerun or update of the search within six months before publication of the review or review update: Highly Desirable Compare magnitude and direction of effects reported by studies with how they are presented in the review, taking account of legitimate differences: Mandatory Consider in advance whether issues of equity and relevance of evidence to specific populations are important...: Highly desirable

22. Other MECIR outputs Guidance on implementation of standards for updates to be produced –to be produced Reporting standards (for protocols, reviews, updates) –in draft –Working Groups to be invited to input –consultation: Collaboration and external (users) Good practices and Common errors –in draft –will continually evolve

23. Facilitating implementation A mixture of Distribution to entities and individuals Handbook amendments –prominent incorporation of standards throughout –extension/clarification of guidance as appropriate Training materials (Training Working Group) Check lists for authors, editors, peer reviewers –building on existing author checklist (Editorial Resources Committee) Modifications to RevMan Baseline audit

24. Concluding remarks MECIR conduct standards are (almost all) clarifications of existing Handbook guidance Adopting the standards should ensure that Cochrane intervention reviews are consistently of high quality We hope you will support and implement them –and contribute to development of further expectations, and future evaluations of them all The standards will be updated over time For more, see www.editorial-unit.cochrane.org/mecir Finally, THANK YOU to all individuals and entities who have contributed so helpfully so far to the development of the expectations...

25. Particular thanks to Working Groups Doug Altman Mohammed Ansari (Methods lead, WG1) Sally Bell-Syer Patrick Bossuyt Deborah Caldwell Christopher Cates Rachel Churchill (Co-Eds lead, WG4) Mike Clarke (Co-Eds co-lead, WG2) Jan Clarkson (Co-Eds co-lead, WG6) Philippa Davies Marina Davoli (Co-Eds lead, WG1) Ruth Foxlee Chantelle Garritty Davina Ghersi (Co-Eds co-lead, WG2) Julie Glanville (Methods co-lead, WG2) Peter Herbison Julian Higgins Sophie Hill (Co-Eds lead, WG3) Toby Lasserson Edith Leclercq Carol Lefebvre (Methods co-lead, WG2) Jessie McGowan Rachel Marshall Ruth Mitchell Donal O’Mathuna Anna Noel-Storr Georgia Salanti (Methods lead, WG5) Doug Salzwedel Margaret Sampson Jelena Savovic Holger Schünemann (Methods lead, WG 6) Ian Shemilt Nandi Siegfried Jonathan Sterne (Methods lead, WG4) Britta Tendal (Methods lead, WG3) David Tovey Peter Tugwell Lucy Turner Claire Vale Julia Walters Helen Worthington (Co-Eds lead, WG 5 and Co-Eds co-lead, WG6) Janelle Yorke