1. Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.archivedOER Public Archive Home Page

2. Enhancing Peer Review at the National Institutes of Health Andrea Kopstein, PhD, MPH AEA Annual Meeting Denver, CO, November, 2008 National Institutes of Health U.S. Department of Health and Human Services

3. ENHANCING PEER REVIEW PRIORITY 1: ENGAGE THE BEST REVIEWERS

4. Electronic Reviews: Expanding Peer Review Platforms Study Sections Necessity ● Clinical reviewers Preference ● Physicists, computational biologists New Opportunities ● Fogarty, International Reviewers Our Goal: 15% of all reviews to be electronic in 2008 Electronic Reviews Video Enhanced Discussions Asynchronous Electronic Discussions

5. Expected Benefits for AED Greater flexibility for scheduling and running peer review meetings. Expand potential reviewer base - eliminate travel and time constraints. Allow for more thoughtful and thorough discussions. Potentially less confrontational environment Improved management of conflicts. Website is straightforward for reviewers and requires little training. Reduced costs

6. AED Progress …. Improvements to Interface (Version 3.1 now being used). Enhancements to AED: Unread comments Unscoring Identification of conflicts- scoring table Assigned reviewer display, etc. 170 review meetings done using AED in 2008 Wider adoption and acceptance – 5 new NIH IC users and 4 agencies educated about AED in FY2008.

7. The AED Evaluation Web-based survey to reviewers and SROs. Fixed choice and open-ended questions. Response rates: Reviewers - 56% (786/1460). Scientific Review Officers - 60% (55/92) Both target populations had two follow-up reminders Although response rates were somewhat lower than desired, the summary of qualitative comments provide some insight into the AED process and AED is an ongoing initiative

8. SOME EVALUATION FINDINGS – SUMMER 2008

9. Reviewer Satisfaction with AED Technology

10. Scientific Review Officer (SRO) Satisfaction with AED Technology

11. Reviewers: Was the duration of discussion sufficient to allow an appropriate level of consideration of applications?

12. SRO: Compared to a face-to-face meeting, do you feel the quality of deliberation was:

13. REVIEWERS: Some of the most frequent positive responses to open questions: #1 response – AED works well! #2 - AED provides a review that is more fair because of deliberation time Many times a reviewer may misinterpret certain facts on an application. If one is not a primary reviewer it is impossible to check out these facts. With this system there is plenty of time to check things out if something does not ring true, or if there is a strong disagreement about a certain point.

14. REVIEWERS: Frequent positive responses…… This format relies EXCLUSIVELY on the strengths of an opinion, not on the aggressiveness of a reviewer and their ability to speak loud. The technology works well Not having to travel is a big benefit

15. REVIEWERS: Some of the most frequent negative responses to open questions: Complaints about some reviewers tending to ignore applications for which they were not assigned or going offline for extended periods of time. In a face-to-face meeting, it is possible to absorb the discussion as it occurs. Passive listening not viable in AED More difficult to make your case for an applications than in face-to-face meetings

16. REVIEWERS: Negative responses - continued Sometimes questions are asked but then get no response A major disadvantage is the an inability to talk to colleagues during the meeting Technology was slow, navigation issues, trouble for MACs

17. SROs: Top positive responses to open questions: #1- Easier to recruit stellar reviewers Easier to get clinical reviewers Easier to get international reviewers Easier to schedule Good for reviewing diverse applications

18. SROs: Top positive responses…….. Allows multitasking while the review meeting is ongoing The technology works well SROs and Reviewers like not having to travel Meeting can be run from any location where there is Internet access

19. SROs: Some of the most frequent negative responses to open questions: Loss of direct interaction with reviewers – personal rapport lost Concerned that unassigned reviewers don’t review or score carefully Spotty reviewer participation in discussion, sometimes difficult to engage More Labor intensive than face-to-face meeting for smaller numbers of applications Sometimes applications are nitpicked because there is so much time