Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.archivedOER Public Archive Home Page

Enhancing Peer Review at the National Institutes of Health Andrea Kopstein, PhD, MPH AEA Annual Meeting Denver, CO, November, 2008 National Institutes of Health U.S. Department of Health and Human Services

ENHANCING PEER REVIEW PRIORITY 1: ENGAGE THE BEST REVIEWERS

Electronic Reviews: Expanding Peer Review Platforms Study Sections Necessity ● Clinical reviewers Preference ● Physicists, computational biologists New Opportunities ● Fogarty, International Reviewers Our Goal: 15% of all reviews to be electronic in 2008 Electronic Reviews Video Enhanced Discussions Asynchronous Electronic Discussions

Expected Benefits for AED Greater flexibility for scheduling and running peer review meetings. Expand potential reviewer base - eliminate travel and time constraints. Allow for more thoughtful and thorough discussions. Potentially less confrontational environment Improved management of conflicts. Website is straightforward for reviewers and requires little training. Reduced costs

AED Progress …. Improvements to Interface (Version 3.1 now being used). Enhancements to AED: Unread comments Unscoring Identification of conflicts- scoring table Assigned reviewer display, etc. 170 review meetings done using AED in 2008 Wider adoption and acceptance – 5 new NIH IC users and 4 agencies educated about AED in FY2008.

The AED Evaluation Web-based survey to reviewers and SROs. Fixed choice and open-ended questions. Response rates: Reviewers - 56% (786/1460). Scientific Review Officers - 60% (55/92) Both target populations had two follow-up reminders Although response rates were somewhat lower than desired, the summary of qualitative comments provide some insight into the AED process and AED is an ongoing initiative

SOME EVALUATION FINDINGS – SUMMER 2008

Reviewer Satisfaction with AED Technology

Scientific Review Officer (SRO) Satisfaction with AED Technology

Reviewers: Was the duration of discussion sufficient to allow an appropriate level of consideration of applications?

SRO: Compared to a face-to-face meeting, do you feel the quality of deliberation was:

REVIEWERS: Some of the most frequent positive responses to open questions: #1 response – AED works well! #2 - AED provides a review that is more fair because of deliberation time Many times a reviewer may misinterpret certain facts on an application. If one is not a primary reviewer it is impossible to check out these facts. With this system there is plenty of time to check things out if something does not ring true, or if there is a strong disagreement about a certain point.

REVIEWERS: Frequent positive responses…… This format relies EXCLUSIVELY on the strengths of an opinion, not on the aggressiveness of a reviewer and their ability to speak loud. The technology works well Not having to travel is a big benefit

REVIEWERS: Some of the most frequent negative responses to open questions: Complaints about some reviewers tending to ignore applications for which they were not assigned or going offline for extended periods of time. In a face-to-face meeting, it is possible to absorb the discussion as it occurs. Passive listening not viable in AED More difficult to make your case for an applications than in face-to-face meetings

REVIEWERS: Negative responses - continued Sometimes questions are asked but then get no response A major disadvantage is the an inability to talk to colleagues during the meeting Technology was slow, navigation issues, trouble for MACs

SROs: Top positive responses to open questions: #1- Easier to recruit stellar reviewers Easier to get clinical reviewers Easier to get international reviewers Easier to schedule Good for reviewing diverse applications

SROs: Top positive responses…….. Allows multitasking while the review meeting is ongoing The technology works well SROs and Reviewers like not having to travel Meeting can be run from any location where there is Internet access

SROs: Some of the most frequent negative responses to open questions: Loss of direct interaction with reviewers – personal rapport lost Concerned that unassigned reviewers don’t review or score carefully Spotty reviewer participation in discussion, sometimes difficult to engage More Labor intensive than face-to-face meeting for smaller numbers of applications Sometimes applications are nitpicked because there is so much time