1. CSR Advisory Council Meeting May 19, 2014 Editorial Board Review A Few Good Reviewers Don Schneider, Ph.D.

2. Reviewer pools Past experience Cost considerations Toward Review by the Best

3. Reviewer StatusNumbers HHMI300 NIH R37700 NAS2,000 R0126,000 CSR needs16,000 Reviewer Pools

4. Format modeled on journal manuscript review First piloted in 2008 with 6 SBIR panels, just in time for TR01s, Challenge grants, DP1 etc. Two stages – First Stage – Mail reviewers – Second Stage – Editors Past Experience

5. First Stage/Mail Reviewers Subject matter experts –Provide depth in review Focus on scientific and technical merit 2-3 first stage mail reviewers per application Submit full critiques Give overall impact and criterion scores –Overall impact scores not factored into final priority score

6. Hold face-to-face meeting Recruit broad experts – Provide perspective in review (assign about 15 applications each) Focus on impact and significance Assign 3 second stage reviewers per application Consider first stage critiques in review Write overall impact paragraph Give overall impact score – Final priority score based on second stage only Second Stage/Editors

7. Provides both depth and breadth in review Optimizes use of the best reviewers Scales well for large numbers of applications (second stage discusses a fraction of the applications) Rationale

8. Perceived Advantages Involves no travel/teleconference for first stage reviewers Allows small, interactive face-to-face meetings Promotes better scoring and assessment of impact Lessens travel and lodging expenses and inconveniences

9. Review# of ApplicationsCost/application Regular R01 F2F$518 DP1/Pioneer EB+I244$280 DP2/New Innov EB593$124 DP5/Early Indep EB+I84$875 Cost Considerations (Alicia Caffi)

10. Recruitment of large numbers of reviewers Timeline – Tight, two sequential reviews – (in the 17 week cycle) More staff time required (SROs) Some sense of isolation by first stage reviewers Challenges

11. Each application examined by at least 5 reviewers Interactive, thoughtful discussions Overall scoring by second stage members Reviewers and staff like final review products Review Outcomes

12. Survey conducted by A Kopstein of reviewers participating in SBIR pilots 2008 Outcomes were generally positive – Majority willing to participate in either review stage in future – Editorial Board Review: Increases expert review »3/4 ths of respondents Preferred for their own applications »2/3 rds of respondents Survey

13. Hopes for a few good reviewers? Discussion