1. Amy Rubinstein, Ph.D. Scientific Review Officer Direct Ranking of Applications: Pilot Study

2. Applications are assigned to 3 reviewers who provide preliminary impact scores (1-9) and critiques. After panel discussion of each of the top 50% of applications, all panel members vote on a final overall impact score. Each application’s score is derived from the average of all panel members’ votes and multiplied by 10 (resulting in final scores of 10-90). R01 applications are assigned a percentile based on the scores of applications reviewed in the relevant study section in that round and the previous 2 rounds. Current System for Evaluating and Ranking Applications Reviewed in CSR Study Sections

3. The number of applications reviewed by NIH is at or near historic highs and award rates are at historic lows. It can be difficult to differentiate between the top 1-20% of applications reviewed in study sections. Concerns about the potential for an application to be funded results in compression of scores in the 1-3 range (final scores between 10 and 30). The current system of percentiles is used to rank applications reviewed in different study sections. However, score compression results in many applications with the same percentile, making funding decisions more difficult. Why Consider Direct Ranking?

4. Distribution of scores for discussed applications

5. Council RoundScore 20Score 25Score 30 2014/0561218 2014/0171118 2013/1071118 2013/0561118 2013/0191522 2012/1091522 2012/0581522 2012/0181522 2011/1081421 2011/0581421 2011/0171421 2010/1071320 2010/0571320 2010/0171219 2009/1071319 CSR All Percentiles at scores of 20, 25, 30

7. Andrea Hollingshead, University of Southern California (rank order methods and procedures) Reid Hastie, University of Chicago (behavioral and psychological aspects of ranking and rating systems) David Budescu, Fordham University (statistical considerations when designing scoring and ranking systems) Donald Saari, University of California Irvine (effects of different voting methods on order of preference and how outcomes can be manipulated) Symposium on Ranking: August 2013

8. Reviewers would not be forced to give applications higher (worse) overall impact scores than they think the applications deserve. Reviewers would be required to distinguish between applications of similar quality and separate the very best from the rest. Reviewers will have an opportunity to re-rank applications after hearing the discussion of all applications, something that is less practical with the current system. Potential Advantages of a Rank Order Method

9. New Investigators are reviewed in a separate cluster but must be integrated into the final rank order of applications that are reviewed. Applications cannot be ranked with respect to applications in the previous two rounds as is done with the percentile system. Reviewers in study sections that cover highly diverse scientific areas may find direct ranking more difficult. Direct ranking may not allow reviewers to indicate applications of essential equivalent quality. Private ranking may lack the transparency of the current system where reviewers who vote out of the range set by assigned reviewers must provide justification during or after the discussion. Possible Disadvantages of Direct Ranking

10. Will be carried out in parallel with current review system. Applications will be scored as usual; reviewers will be asked to privately rank the top 10 R01 applications discussed on a separate score sheet. Rank data will be analyzed internally but average rank information will not be shared with Program staff or be used to influence funding decisions. Pilot Study for Direct Ranking of Applications

11. Average rank, minimum rank, maximum rank for each application. Variance, standard deviation Percentage of eligible reviewers ranking a given application in the top 10. Data Analysis

12. How does the rank order of directly ranked applications compare to the order obtained by percentiles? Does the rank order method produce fewer ties than the percentile method and therefore better spreading of scores? How much consensus among reviewers is evident in the rank system? (A calculation of variance and the percentage of eligible reviewers ranking an application in the top 10 will be noted.) How do reviewers perceive the new system? Do they find it difficult to remember earlier discussions? Do they find it difficult to judge the relative rank of applications that are of similar quality? (Reviewers will be asked to provide optional, open-ended feedback after the pilot). How might NIH use data from a rank ordering system? (Results will be discussed within CSR and with Program staff using aggregate data. No rank orders for specific applications will be shared with Program Directors). Pilot Evaluations: Questions to Answer

13. Dr. Ghenima Dirami, SRO, Lung Injury and Repair study section Dr. Tomas Drgon, SRO, Biostatistics Methods and Research study section Dr. Gary Hunnicutt, SRO, Cellular, Molecular and Integrative Reproduction study section Dr. Raya Mandler, SRO, Molecular and Integrative Signal Transduction study section Dr. Atul Sahai, SRO, Pathobiology of Kidney Disease study section Dr. Wei-qin Zhao, SRO, Neurobiology of Learning Memory study section Dr. Adrian Vancea, Program Analyst, Office of Planning, Analysis and Evaluation Direct Ranking Pilot Working Group Members