1. Traditional Peer Review ULS Scholarly Communications Lunch and Learn #14
Office of Scholarly Communication and Publishing
University Library System
University of Pittsburgh
August 21, 2014

2. Traditional Peer Review
What is peer review?
Why should librarians care about peer review?
How does it work? - the process
Research ethics and peer review
Problems and limitations of traditional peer review
NEXT MONTH: Innovations in peer review

3. What is peer review?
Peer review is expert, independent, unbiased scrutiny of research to:
self-regulate quality standards within an academic discipline
reinforce the scientific method
provide credibility for research results
determine suitability for scholarly publication
Scholarly peer review is different from:
professional peer review
clinical peer review

4. Why should librarians care about peer review?
essential to critical evaluation of information sources
adds value and credibility to resources we provide
critical for the library as publisher
librarians may serve as reviewers ourselves
important to the clients we serve
Major changes are underway - we need to understand them!

5. History of peer review
1752: Royal Society of London establishes ‘Committee on Papers’
reviews papers for the Philosophical Transactions of the Royal Society
journal already in existence for 86 years
Early days – reviewers were not looking at scientific validity so much as searching for good papers that would promote the journal or editors.
Most early peer review was not independent or external, and most was conducted by editors themselves.
External review did not catch on until the mid 20th century.

6. Some characteristics of peer review
Conducted by independent experts
Free from personal and professional bias
Anonymous
single blind: identity of reviewer is not known to author
double blind: identity of author and reviewer unknown to each other
Norms and processes vary by discipline

7. What is being evaluated?
Research methodology
Scientific soundness
Statistical design and analysis
General presentation of results
Degree to which conclusions are supported by evidence
Can the research be replicated by another researcher?

8. What is being evaluated?
Originality of research
Novelty and overlap with similar research
Relevance: importance to the discipline
Anticipated level of interest
Appropriateness for scope of journal (or journal section)

9. Traditional peer review: THE PROCESS

10. Responsibility of the Author
Removing identifying information from the submission
Do not include authorship in the submission document
Remove identifying information from the document properties
Do not attempt to discover the identity of the reviewer and, if an author accidentally discovers the reviewer’s identity, do not contact them regarding the submission

11. Responsibility of the Reviewer
Disclose to the editor any conflicts of interest
Only review submissions for which you have expertise
Agree to review only if you can do so in a timely manner
Decline reviews similar to your own current work in preparation or submission
Don’t share the submission or its details with others
Support criticisms with evidence
Do not attempt to discover the author’s identity, and if you accidentally do so, inform the editor immediately

12. Reviewing: A Labor of Love?
Reviewers typically do not get paid
Reviewers are not publicly acknowledged for individual reviews
Some journals give an annual award for the most active reviewer (Oh yeah! A plaque!)
A single review can take several hours to complete

13. RESEARCH ETHICS and PEER REVIEW

14. Test Your Review Ethics Knowledge
You are an editor of a journal that practices double- blind peer review. An author of a recently-reviewed (but not yet published) article claims that the rejection received from Reviewer #2 is due to personal bias – the author alleges that Reviewer #2 discerned their identity from the very unique data set that they were using and, due to a feud between Reviewer #2 and the author’s dissertation advisor, rejected the article for non-scholarly reasons. The identity of Reviewer #2 is, in fact, the person alleged to be in a feud with the author’s advisor. What do you do?

15. Options:
A. Nothing. Reviewers are allowed to reject the article for any reason.
B. Investigate the connection between Reviewer #2 and the dissertation advisor in question, but leave the review as it stands.
C. Thank the author for their concern and recruit another peer reviewer to take the place of Reviewer #2, throwing out Reviewer #2’s comments.
D. Encourage the author to take any methodological or theoretical concerns of Reviewer #2 into consideration, but favor the responses of Reviewers 1 and 3 more heavily.
Choose this if the review is pure vitriol
Choose this one if the review has substance

16. Test Your Review Ethics Knowledge
As a reviewer for a double-blind peer reviewed medical journal, you notice that the author of a paper on different types of treatment for a disease repeatedly cites works from a particular pharmaceuticals group. You know that there are many competing pharmaceuticals companies with different treatments available and suspect that the author may be affiliated with this group in some way. However, due to the double-blind process, you do not know the name or affiliation of the author, and no information is given in the article. Is this a problem, and if so, what do you do?

17. Options: A. Report the issue to the editor with relevant evidence.
B. Conduct your own investigation, ing the members of the cited pharmaceutical group to inquire if any of them have submitted a paper to the journal recently, then report your findings to the editor.
C. Write your own paper responding to this article with criticisms about Conflict of Interest, then submit it to the same journal before you have completed your review.
D. Nothing. People from pharmaceuticals groups are allowed to write papers too.

18. Part 2:
As the editor in this instance, with the reviewer presenting all relevant evidence, what do you do?

19. Options:
A. Nothing. Conflicts of interest are not the domain of your journal.
B. Contact the author of the paper and request a statement of any conflicts of interest.
C. Immediately reject the paper for ethical reasons.
D. Contact the author’s organization informing them of ethical misconduct.

20. Test Your Review Ethics Knowledge
You are an author of a journal article investigating the dialect of an unstudied Inuit fishing community. While your article is being formatted for publication, you attend a presentation at a conference about dialects of Inuit by a Professor Richard Smith and, to your shock, one of his slides contains the exact same data that you reported in your unpublished article! The community is small and you are pretty sure you would know all of the other researchers who have done work in the area, and Richard Smith is not one of them. You notice that Professor Smith doesn’t cite the source of his data and you suspect that he was a reviewer of your article. What should you do?

21. Options:
A. This is one of the hazards of publishing in academic journals. There is nothing you can do.
B. During the question and answer session, publicly accuse Professor Smith of ‘scooping’ your data and demand answers.
C. Contact the editor of the journal you are publishing in, presenting all of the evidence and your concerns.
D. Call the U.S. copyright office and file a complaint against Professor Smith.

22. Part 2:
You are the editor of the journal who receives the author’s complaint. You know that Professor Richard Smith was one of the reviewers for this article. What do you do?

23. Options: A. Say thank you to the author and promise to investigate.
Answer: all of the above, in this order, if there is a breach of reviewer conduct!
Options:
A. Say thank you to the author and promise to investigate.
B. Review the files and the timeline to see if Professor Smith could have scooped the data.
C. Contact Professor Smith directly to request an explanation.
D. Contact the reviewer’s institution requesting an investigation.
E. Remove the reviewer from your review database and consider reporting the case in your journal.

24. Traditional peer review: limitations and challenges

25. Traditional peer review: limitations and challenges
Labor intensive
Reviewers are overtaxed and under-rewarded
Time-consuming; slows the pace of research
Process is open to bias, self-interest, and cronyism
Valuable research can be lost through subjective filtering for relevance and importance
Limits transparency and accountability
Like Churchill said: “Democracy is the worst form of government, except all those other forms that have been tried from time to time.”

26. NEXT MONTH Innovations in Peer Review
Open peer review
The preprint publishing model
Post-publication peer review (PubPeer)
Filtering for methodology only (PLoS One and PeerJ)
Rewarding and rating peer reviewers
Independent peer review, pre-submission (Rubriq)
No hybrid Open Access
No ‘delayed’ Open Access
Be listed in the Directory of Open Access Journals (unless the journal is too new for DOAJ eligibility)
Be a member of the Open Access Scholarly Publishers Association or adhere to its Code of Conduct
Have publicly available a standard article fee schedule