1. 1 Amy Rubinstein, Ph.D., Scientific Review Officer Adrian Vancea, Ph.D., Program Analyst Office of Planning, Analysis and Evaluation Study on Direct Ranking of Applications: Advantages & Limitations

2. 2 Applications are assigned to 3 reviewers who provide preliminary impact scores (1-9) and critiques. After panel discussion of each of the top 50% of applications, all panel members vote on a final overall impact score. Each application’s score is derived from the average of all panel members’ votes and multiplied by 10 (resulting in final scores of 10-90). R01 applications are assigned a percentile based on the scores of applications reviewed in the relevant study section in that round and the previous 2 rounds. Current System for Evaluating and Ranking Applications Reviewed in CSR Study Sections

3. 3 The number of applications reviewed by NIH is at or near historic highs and award rates are at historic lows. It can be difficult to differentiate between the top 1-20% of applications reviewed in study sections using raw scores. Concerns about the potential for an application to be funded results in compression of scores in the 1-3 range (final scores between 10 and 30). The current system of percentiles is used to rank applications reviewed in different study sections. However, score compression results in many applications with the same percentile, making funding decisions more difficult. Why Consider Direct Ranking?

4. 4 1%: 10-1311%: 2521%: 33 2%: 14-1512%: 2622%: 34 3%: 16-1713%: 2724%: 35 4%: 18-1914%: 2825%: 36 6%: 2016%: 2927%: 37 8%: 21-2217%: 3029%: 38 9%: 2319%: 3131%: 39 10%: 2420%: 3233%: 40 Percentile Base Report Council Date: 2015/01 IC: CSR

5. 5

6. 6 Reviewers not be forced to give applications higher (worse) overall impact scores than they think the applications deserve. Reviewers required to distinguish between applications of similar quality and separate the very best from the rest. Reviewers have the opportunity to re-rank applications after hearing the discussion of all applications, something that is less practical with the current system. Potential Advantages of a Rank Order Method

7. 7 New Investigators are reviewed in a separate cluster but must be integrated into the final rank order of applications that are reviewed. Applications cannot be ranked with respect to applications in the previous two rounds as is done with the percentile system. Reviewers in study sections that cover highly diverse scientific areas may find direct ranking more difficult. Private ranking may lack the transparency of the current system where reviewers who vote out of the range set by assigned reviewers must provide justification during or after the discussion. Challenges Associated with Direct Ranking

8. 8 Carried out in parallel with the current review system in the 2014_10 and 2015_01 council rounds. Applications were scored as usual; reviewers were asked to privately rank their top 10 R01 applications discussed on a separate column on the score sheet. Rank data was analyzed for informational purposes and not used to influence funding decisions. Pilot Study for Direct Ranking of Applications

9. 9 32 chartered scientific review groups (SRGs) from the 2014_10 and 2015_01 council rounds Number of discussed R01 applications per SRG ranges from 12 to 39 (average 26.12) Number of reviewers per SRG ranges from 13 to 31 (average 22.97) Participating Study Sections

10. 10 Measure correlation between the percentiles/scores and direct ranking results –Each application has an associated percentile/score –Associate an “average rank” with each application –Expect good correlation Propose a method for breaking up ties using the ranking results Visualize correlation between ranking and percentiles. Data Analysis

11. 11 NP = Not present, CF = Conflict, NR = Not ranked Source Data Format

12. 12 Data with Imputed Ranks

13. 13 Next step is to calculate average the rank for each application Data with Imputed Ranks

14. 14 For application A, only 19-1=18 reviewers can rank –Average rank = average of the 18 ranks = 83/18 = 4.61 For application B, only 19-2=17 reviewers can rank –Average rank = average of the 17 ranks = 165/17 = 9.71 Applic ation R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19Avg Rank A 36367257CF64464356244.61 B 147513567121015589NP1510 14CF9.71 Average Rank

15. 15 Data with Imputed Ranks

16. 16 Correlation Coefficient Between Rank and Percentile

17. 17 B is better than A R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19 Avg Rank A 143CFNP9136151432457799CF7 7.94 B 57NP8547CF14134145126745 6.53 How can one differentiate between two applications? We want something natural and easy to understand. 19-5=14 common reviewers considered to rank/compare both applications 5 reviewers that consider A better than B 9 reviewers that consider B better than A Comparing Applications with Similar Percentiles

18. 18 Appl Indices 7 14% 8 14% 9 16% 10 18% 11 21% 12 21% 13 21% 14 21% 7 14% 7 7 14/20 7 14/19 7 14/17 7 11/20 7 12/21 7 16/21 7 16/20 8 14% 8 8 12/17 8 12/16 11 9/16 8* 9/18 13 11/19 8 14/16 9 16% 9 10 8/15 11 10/14 12 8/13 13 12/19 9* 6/12 10 18% 10 11 10/17 12 9/17 13 11/17 10 9/13 11 21% 11 9/14 13 10/19 11 11/15 12 21% 12 13 12/19 12 10/14 13 21% 13 15/19 14 21% 14 14 out of 20 common reviewers ranked 7 as better than 8 * Indicates ties Direct Comparison Matrix

19. 19 Application AApplication B score (Average)27 percentile11% score range2,3, 42, 3 reviewers preference by score3/19 (16%) 13/19 (68%) awarded each application the same score ranking (Average)6.16.2 ranking range 1- NR2-NR reviewers preference13/18 (72% of reviewers)5/18 (28% of reviewers)A stronger than B Comparing Two Applications with the Same Percentile

20. 20 Visualization of Binning for All SRGs

21. 21 Visualization of Binning for Single SRG

22. 22 Helped reviewers prioritize applications and improved score spreading. Reviewers more engaged in discussions because of the need to rank. Difficult to rank applications that the reviewer did not read. May provide some complementary information but should not replace current system. Reviewer Comments

23. 23 Does ranking add value to the peer review process? Could the rank ordering exercise be used as a tool by SROs to help panels spread scores and become more engaged in discussion? Can rank ordering be used by Program Staff to break ties or provide more information needed for funding decisions? Questions and Next Steps

24. Dr. Ghenima Dirami, SRO, Lung Injury and Repair study section Dr. Gary Hunnicutt, SRO, Cellular, Molecular and Integrative Reproduction study section Dr. Raya Mandler, SRO, Molecular and Integrative Signal Transduction study section Dr. Atul Sahai, SRO, Pathobiology of Kidney Disease study section Dr. Wei-qin Zhao, SRO, Neurobiology of Learning Memory study section Dr. Adrian Vancea, Program Analyst, Office of Planning, Analysis and Evaluation Dr. Amy Rubinstein, SRO, Gene and Drug Delivery Systems Study Section Direct Ranking Pilot Working Group Members

25. 25 Post Ranking Pilot Office of Planning, Analysis and Evaluation Q & A