Presentation on theme: "AWHILES Study Day 13 May 2011 Julie Browne External Relations Manager, Wales Deanery, PGMDE University of Cardiff."— Presentation transcript:
AWHILES Study Day 13 May 2011 Julie Browne External Relations Manager, Wales Deanery, PGMDE University of Cardiff
Published research: how do you know if its any good? …or
The role of the editorial office in safeguarding the integrity and academic rigour of the scholarly archive, with particular reference to the medical education literature
What Id like to cover today The role of journals in safeguarding quality Medical education journals Peer review – what is it? Peer review – how is it done? Peer review – why do it? Publication ethics
Meet Jamie …
How do you know if a published paper is likely to be reliable?
Jamie again …
What are the characteristics of a high-quality journal?
Quality in journals Metrics People Content Systems
Leg 1: Metrics Citations (including self-citation) Indexes Rejection rates Speed of peer review/acceptance to publication Impact factor/immediacy
ISI Web of Knowledge: Journal Citation Reports
Trend spotting: Impact factor
Leg 2: People –Editorial board, editor, editorial staff –Publisher –Learned society –Peer reviewers –Authors
Leg 3: Content High quality papers (goes without saying – well discuss this further in a minute!) Letters and commentaries sections (post- hoc peer review) Editorials and guidance on scope and mission Retractions, corrections, errata Participation in journalology research
Leg 4: Editorial Systems Transparent processes, clearly announced – how we do things, what you can expect from us Published policies: e.g. ethics, peer review, handling of complaints, handling of editors papers Membership of COPE or other editors group An ethics committee A statement regarding editorial independence (e.g. All content in Academic Medicine reflects the views of the authors and does not reflect the official policy of the AAMC unless clearly specified. The BMA grants editorial freedom to the editor of the BMJ. The views expressed in the journal are those of the authors and may not necessarily comply with BMA policy.)
But, above all… A rigorous, effective and fair peer review system
The purpose of peer review is to keep egg off authors faces Drummond Rennie (JAMA)
Completion of research Writing and preparation Submission Internal review Decision Revision Resubmission Re-review Acceptance Publication Rejection The review process External review
Common reasons for rejection Did not address an important scientific issue Research question not clearly stated Has been done before Results are unsurprising/does not add enough new information to be interesting Inappropriate study design/methodological weaknesses/methods not described clearly enough Sample size too small/response rate too low Incorrect or unjustified conclusions Statistical analysis incorrect or inappropriate Wrong journal/readership; would not interest our readers Too badly written to understand
But sometimes something murky is lurking under the surface….
Fabrication: invention of data or cases Falsification: wilful distortion of data including failure to report missing or negative data Plagiarism: copying of ideas, data or words without attribution Failing to get consent from an ethics committee for research and/or conducting research in humans without informed consent Misreporting authorship contributions Not revealing conflicts of interest Murky things: misconduct
Peer review to the rescue?!
More from Jamie …
How widespread is research and publication misconduct? How much does the peer review system catch?
Things inevitably slip through …
Some big cases (1) Jon Sudbø In January 2006 it was revealed that a paper by Norwegian researcher Jon Sudbø published in The Lancet in October 2005 was based upon fraudulent patient data. The article had suggested that some types of NSAIDs diminish the risk of oral cancer in smokers. However, it turned out that he had made up all the patient data; a quarter of all the subjects in The Lancet study had the same birthday!
Some big cases (2) Andrew Wakefield 1996 A paper by Wakefield and colleagues published in The Lancet suggested that there was a connection between the MMR vaccine and autism and colitis No subsequent research was able to reproduce his findings. The paper was retracted Immunisation rates dropped from 92% to 73%, almost certainly leading to a rise in mortality Wakefield was struck off by GMC for over 30 charges of misconduct including four counts of dishonesty. Wakefield continues to defend his reputation.
Malcolm Pearce 1994: Obstetrician Malcolm Pearce claimed in BJOG to have successfully relocated an ectopic pregnancy, with a healthy subsequent birth. A junior doctor at St Georges blew the whistle. The patient had never existed. Some big cases (3)
OK, so these are big cases. But does the little stuff really matter that much …?
The journalology literature 56% of newly appointed consultants reported witnessing research fraud Geggie (2001) J Med Ethics 27:344–6 On average 2% of scientists admit to have falsified research at least once and up to 34% admit other questionable research practices Fanelli (2009). PLoS ONE 4(5): e % of submissions to a Chinese journal were plagiarized. Zhang. (2010) Nature 467:153 30% of submitted manuscripts included plagiarism from a previous publication of the senior author. 16% of submitted manuscripts included plagiarism from another investigators work and/or website. McFarlin et al (2010) IJ Exercise Sci (3):3 9.5% of authors in the BMJ were guest authors: i.e. they did not make any significant contribution to the paper Bates et al (2004) JAMA. 292(1):86-8.
How dangerous is it? erodes public trust in the scientific community is extremely costly to investigate and police may create a flawed knowledge base (systematic reviews) may negatively influence funders and politicians may divert funding and effort away from reputable research may influence the political process, esp in controversial areas e.g. stem cell research (Hwang Woo Suk) can create false hope among patients
Corrections, Retractions and "Expressions of Concern" The retraction or expression of concern, so labelled, should appear on a numbered page in a prominent section of the print journal as well as in the online version, be listed in the Table of Contents page, and include in its heading the title of the original article. The text of the retraction should explain why the article is being retracted and include a complete citation reference to that article. Uniform Guidelines for manuscripts submitted to biomedical journals (The Vancouver Guidelines)
But retracted papers live on … Examination of one Nature paper by former Bell Labs physicist Jan Hendrik Schön, published in 2000 and retracted in 2003, revealed that it's been noted in research papers 17 times since, although the drop-off after retraction was steep: prior to being pulled, the paper was cited 153 times. Unger K. Couzin J. Even Retracted Papers Endure. Science 2006;312(5770):40-41 The difference between citation levels of flawed originals and corrected republications does not approach statistical significance until eight to twelve years post-republication. Peterson GM. The effectiveness of the practice of correction and republication in the biomedical literature. J Med Libr Assoc. 2010; 98(2): 135–139.
Brice, J. and J. Bligh. Author misconduct: not just the editors' responsibility. Medical Education, (1): p Smith, R. The trouble with medical journals. London: RSM Books 2006 Hames, I. Peer review and manuscript management in scientific journals: Guidelines for good practice. 2007, London: John Wiley and Sons. Wager, E., F. Godlee, and T. Jefferson. How to survive peer review. 2002, London: BMJ Books. Bligh, J. and J. Brice, New Year reflections on professionalism. Medical Education, (1): p Organisations: ICMJE: World Association of Medical Editors: Committee on Publication Ethics: Further information