Presentation is loading. Please wait.

Presentation is loading. Please wait.

COMMENTS ON COCHRANE REVIEWS Approaches to managing feedback WORKSHOP 22 September 2013 Cochrane Colloquium, Quebec City.

Similar presentations


Presentation on theme: "COMMENTS ON COCHRANE REVIEWS Approaches to managing feedback WORKSHOP 22 September 2013 Cochrane Colloquium, Quebec City."— Presentation transcript:

1 COMMENTS ON COCHRANE REVIEWS Approaches to managing feedback WORKSHOP 22 September 2013 Cochrane Colloquium, Quebec City

2 Introductions We are: John Hilton Editor, Cochrane Editorial Unit Gavin Stewart Associate Editor, Wiley Jo Garner Senior Business Analyst, Wiley Toby Lasserson Senior Editor, Cochrane Editorial Unit Feedback Editor, Airways Group + 19 other roles Disclosures: Management of comments is part our jobs as employees of our respective organisations but otherwise we have no actual or potential conflict of interest in relation to this presentation. Who are you?

3 Introductions We are: John Hilton Editor, Cochrane Editorial Unit Gavin Stewart Associate Editor, Wiley Jo Garner Senior Business Analyst, Wiley Toby Lasserson Senior Editor, Cochrane Editorial Unit Feedback Editor, Airways Group + 19 other roles Disclosures: Management of comments is part our jobs as employees of our respective organisations but otherwise we have no actual or potential conflict of interest in relation to this presentation. Who are you?

4 Introductions We are: John Hilton Editor, Cochrane Editorial Unit Gavin Stewart Associate Editor, Wiley Jo Garner Senior Business Analyst, Wiley Liz Dooley Managing Editor, ARI Group ME Support + 21 other roles Disclosures: Management of comments is part our jobs as employees of our respective organisations but otherwise we have no actual or potential conflict of interest in relation to this presentation. Who are you?

5 Workshop Part 1: Whats the latest? Overview, data, guidance, support What do you need? What do you need to know? Part 2: Case studies Some real-life examples What would you do? And why? Any barriers, hurdles or brick walls? Part 3: The way ahead Improving the process and the display How can you help? Part 4: Is it worth it? Isnt everyone using Twitter these days? What do you think?

6 Submitting comments

7

8

9 Comments are also submitted via other channels: Direct contact to CRG From OVID Direct contact to Editor in Chief or CEU Summaries website (not at the moment)

10 Processing comments Comment submission to Wiley & CEU Comment triage by Wiley Comment assessment by ME & FE Comment management What does CEU do? Comments go to Editor in Chief, and two editors EiC looking if intervention required (with CRG) Editors looking for cross-CRG issues (e.g. multiple submissions) Support Tracking

11 Processing comments Comment submission to Wiley & CEU Comment triage by Wiley Comment assessment by ME & FE Comment management What gets filtered out: Comments or enquiries about the website functionality, access, display, etc Comments that are offensive, nonsensical, or inconsequential Comments that arise from system errors or system testing Duplicates

12 Processing comments Comment submission to Wiley & CEU Comment triage by Wiley Comment assessment by ME & FE Comment management CRG-led process Managing Editor Feedback Editor Co-ordinating Editor(s) Feedback contributor(s)

13 Processing comments Comment submission to Wiley & CEU Comment triage by Wiley Comment assessment by ME & FE Comment management CRG-led process Managing Editor Feedback Editor Co-ordinating Editor(s) Feedback contributor(s) Author(s) Editor(s)

14 Processing comments Comment submission to Wiley & CEU Comment triage by Wiley Comment assessment by ME & FE Comment management CRG-led process Managing Editor Feedback Editor Co-ordinating Editor(s) Feedback contributor(s) Author(s) Editor(s) Editor-in-Chief CEU Editor(s) Steering Group, CEO Interfering third parties

15 Acting on comments Questions, questions, questions Is it in English? Is it coherent and not clearly wrong? Is it about the content of the review? Should it be published? How much editing does it need? Are the authors going to respond? And when? If authors wont or cant respond, what then? How long will the whole process take? Is anyone in editorial base conflicted? Has contributor declared conflicts? Does it trigger an amendment? Does it trigger an update? Does it raise broader issues?

16 Acting on comments Questions, questions, questions Is it in English? Is it coherent and not clearly wrong? Is it about the content of the review? Should it be published? How much editing does it need? Are the authors going to respond? And when? If authors wont or cant respond, what then? How long will the whole process take? Is anyone in editorial base conflicted? Has contributor declared conflicts? Does it trigger an amendment? Does it trigger an update? Does it raise broader issues? What have I missed? Process Judgement call? Workflow?

17 Comment outcomes 1. Validity 1. Valid: take action 2. Not valid: take no further action 2. Usefulness 1. Translate 2. Edit 3. Drastically cut or ask for resubmission 3. Suitability for publication 1. Publish 2. Dont publish 4. Author response 1. Not needed 2. Needed but not forthcoming 3. Received and published 5. Action on review 1. No action needed 2. Review amendment 3. Review update 4. Changes at next update

18 How many comments?

19 Which groups? In the first half of 2013…

20 Which groups? In the first half of 2013… 4Anaesthesia Group Heart Group Pregnancy & Childbirth Group Wounds Group

21 Which groups? In the first half of 2013… 4Anaesthesia Group Heart Group Pregnancy & Childbirth Group Wounds Group 6Airways Group

22 Which groups? In the first half of 2013… 4Anaesthesia Group Heart Group Pregnancy & Childbirth Group Wounds Group 6Airways Group 8Acute Respiratory Infections Group

23 Which groups? In the first half of 2013… 4Anaesthesia Group Heart Group Pregnancy & Childbirth Group Wounds 6Airways Group 8Acute Respiratory Infections Group And 32 other CRGs received at least 1 comment

24 Characteristics of comments Study inclusion/exclusion Interpretation of findings Conflict of interest Scope Methods Typos, errors

25 Characteristics of comments Study inclusion/exclusion Interpretation of findings Conflict of interest Scope Methods Typos, errors One sentence 10 pages Detailed, well-written referenced letter Semi-coherent scribble Focused on single aspect of review Multiple issues, not all related to review

26 Characteristics of comments Study inclusion/exclusion Interpretation of findings Conflict of interest Scope Methods Typos, errors One sentence 10 pages Detailed, well-written referenced letter Semi-coherent scribble Focused on single aspect of review Multiple issues, not all related to review Retired academics Cochranites Students Clinicians Pharmaceutical companies or manufacturers Persistent single-issue enthusiasts

27 Where do comments come from?

28 What about the unapproved comments? Example 1 Review Speed cameras for the prevention of road traffic injuries and deaths Contributor Maller Comment When there is accident happened, who will take responsibility? After several day or long time, how to know the scene when it happened? Car video recorder could record accident and store video data with gps information on memory card, people could get accident reconstruction, it is very easy. more view:

29 What about the unapproved comments? Example 2 Review Contributor Comment

30 What about the unapproved comments? Example 3 Review Probiotics for the prevention of Clostridium difficile associated diarrhea in adults and children Contributor Rachel Bock Comment Test comment - please delete

31 Known unknowns? Who is submitting comments: students, clinicians, Cochranites, pharma companies? (and why?) How many comments are published, and how long after submission? How many reviews amended or updated as a result of feedback? What aspects are commentors focusing on? (study inclusion, interpretation, outcomes, etc?) Any more? What would it be useful to know? ANY VOLUNTEERS?

32 Cochrane Editorial & Publishing Policy Resource

33 Cochrane Editorial & Publishing Policy Resource

34 Cochrane Editorial & Publishing Policy Resource

35 Cochrane Organisational Policy Manual

36 Cochrane Handbook

37 Cochrane Editorial Unit website

38 So whats in those new guidelines? Guidance on how feedback should be managed When to publish, when not to publish Guidance on reasonable timescales Roles of ME and FE Meeting expectations of contributors Where to get support What to do with non-English comments Your feedback will be very welcome.

39 Feedback editors: neglected? Communication to and between FEs has been lacking Need Role description (purpose, expectations) Induction package? Page on Cochrane Training page? Independence of role Cochrane.org feature Forum

40 So what else do you/we need?

41 Case studies

42 Whats in that CEU proposal? Use the term comments Decouple comments from review, but link to version Display comments parallel to review version Make comments citable (DOI) Expectations about reasonable turnaround time Transparency and tracking Archie workflow for comments So that was a while ago. What happened?....

43 Cochrane-Wiley Roadmap A set of work streams fulfilling agreed contractual projects Next 18 months Feedback scheduled for Q Project will involve: CEU, Wiley, Cochrane Tech team CRGs You?

44 Goals for feedback project Make process more efficient and transparent Phased approach to first improve the user experience for: the Contributor the Reader of feedback De-couple publishing feedback from the review or protocol Make the individual comment citable in its own right

45 Requirements for user feedback 1. A Contributor is able to submit a comment 2. The system shall save and acknowledge the comment 3. Wiley Editorial may moderate the comment 4. The system shall trigger an editorial workflow that may result in publication of the comment along with a response 5. A Reader will be able to view comments and responses 6. A Reader will be able to cite the comment 7. A Search User will be able to search comments and responses

46 Proposed approach – submission

47

48 Proposed approach – display

49 Improve processing A system for moderation Separate the publishing of feedback from the review, retain a link between them: Feedback as a new Content type? Changes to Archie, through XDPS and to Online Library Add the ability to cite individual comments & author responses Improve the robustness of the feedback-response loop Feedback items having their own workflow?

50 Why all the bother? So we can …amend reviews in the light of new evidence…to reflect the emergence of new data, valid feedback, solicited or unsolicited, from whatever source Chalmers I, Haynes B. Reporting, updating, and correcting systematic reviews of the effects of health care. BMJ 1994;209:862-5.

51 But where are all the comments?

52 Not just Cochrane

53 It can work…

54 Even then, will authors engage? Peter Gøtzsche, Andreas Lundh and BMJ editors looked at BMJ rapid responses Even when comments could invalidate research or reduce reliability over half the time authors couldnt be bothered to respond. Are they using the shadow system - scribbled marginalia, chats in labs, peer review, discussions at conferences, journal clubs And Twitter, of course.

55 So, why no comments? Why is it that comment forums provided by online scholarly journals for post-publication peer review are so consistently underutilized, even though online forums concerning countless subjects, from culture to politics, are booming? Technophobia? Time? Worry about appearing to be unacademic Commenting on scientific articles does not advance my career Prefer a published letter (despite delays) Disincentives: fear of upsetting, being wrong, giving away ideas Is the question not whether have comments failed, but whether are they succeeding somewhere else?

56 Whats different about Cochrane? Updating! Comments are part of a continuous process of improvement and updating Culture of acceptance of criticism Expectations of authors So what could we or you do? Proactive solicitation More encouragement on website – expose comments, prompt for comments Altmetrics Activate specific social networks and establish rapport


Download ppt "COMMENTS ON COCHRANE REVIEWS Approaches to managing feedback WORKSHOP 22 September 2013 Cochrane Colloquium, Quebec City."

Similar presentations


Ads by Google