1. NIH Peer Review: Continuity and Change
Toni Scarpa
Center for Scientific Review
National Institutes of Health
Department of Health and Human Services
APS/Cardiovascular Section
Salt Francisco, April 2006

2. Peer Review: An N.I.H. “Conception”
Is the heart and soul of NIH
Has produced an effective partnership between the federal government and research institutions
Has created the best academic medical centers, the best biomedical/behavioral research and biotechnology
Has made possible the best cures and the best prevention
Has been admired and imitated here and abroad
Has protected NIH against outside influence

3. Center for Scientific Review

4. Applications received for all of NIH and applications referred for CSR review, FY 1998-2004
80,000
60,000
Number of applications
40,000
20,000
1998
2000
2002
2004
Fiscal year
Applications received for all of NIH
Applications assigned for review by CSR

5. CSR 4 Review Divisions & 23 IRGs
Division of Biologic
Basis of Disease
Elliot Postow, Ph.D.
Division of Molecular
and Cellular Mechanisms
Donald Schneider, Ph.D.
Division of Physiology
and Pathology
Michael Martin, Ph.D.
Division of Clinical and
Population-Based Studies
Anita Miller Sostek, Ph.D
AIDS and Related
Research IRG (AARR)
Ranga V. Srinivas, Ph.D.
Bioengineering Sciences and
Technologies IRG (BST)
Sally Amero, Ph.D.
Cardiovascular Sciences
IRG (CVS)
Joyce Gibson, D.Sc.
Behavioral & Biobehavioral
Processes IRG (BBBP)
Karen Sirocco, Ph.D.
Biology of Development and
and Aging (BDA)
Sherry Dupere, Ph.D.
Digestive Sciences IRG
(DIG)
Mushtaq Khan, Ph.D., DVM
Brain Disorders and Clinical
Neuroscience IRG (BDCN)
Dana Plude, Ph.D.
Endocrinology, Metabolism, Nutrition, and Reproductive Sciences (EMNR)
Sooja Kim, Ph.D.
Biological Chemistry and Macromolecular Biophysics IRG (BCMB)
John Bowers, Ph.D.
Hematology
IRG (HEME)
Joyce Gibson, D.Sc.
Health of the Population
IRG (HOP)
Robert Weller, Ph.D.
Immunological Sciences
IRG (IMM)
Calbert Laing, Ph.D.
Integrative, Functional
and Cognitive Neuroscience
IRG (IFCN)
Christine Melchior, Ph.D.
Cell Biology IRG (CB)
Marcia Steinberg, Ph.D.
Risk, Prevention, and Health
Behavior IRG (RPHB)
Michael Micklin, Ph.D.
Infectious Diseases and
Microbiology IRG (IDM)
Alex Politis, Ph.D.
Musculoskeletal, Oral, and Skin Sciences IRG (MOSS)
Daniel McDonald, Ph.D.
Genes, Genomes, and Genetics IRG (GGG)
Richard Panniers, Ph.D.
Surgery, Radiology, and
Bioengineering IRG (SRB)
Eileen Bradley, D.Sc.
Oncological Sciences
IRG (ONC)
Syed Quadri, Ph.D.
Renal and Urological Sciences IRG (RUS)
Daniel McDonald, Ph.D.
Molecular, Cellular and
Developmental
Neuroscience IRG (MDCN)
Carole Jelsema, Ph.D.
Respiratory Sciences IRG (RES)
Mushtaq Khan, Ph.D., DVM

6. CSR Mission Statement
To see that NIH grant applications receive fair, independent, expert, and timely reviews -- free from inappropriate influences -- so NIH can fund the most promising research.

7. Necessary Changes in CSR Peer Review Operations
CSR Operations Current Systems New Systems?
Complexity and Impact
Time

8. Changes in CSR Operations
Increase communications between CSR, the ICs, our reviewers and applicants
Increase uniformity
Increase efficiency
Facilitate work of IC program staff

9. Potential of Knowledge Management Tools for Peer Review
Collexis Software or Others
Knowledge management solutions
Fingerprinting and text retrieving
Disease coding
Benefits for Peer Review
Assigning applications to Integrated Review Groups or Study Sections
Selecting reviewers (one application, multiple applications)
Nine pilots are underway to begin to assess these benefits

10. Study Section Realignment
Review of one IRG every month
Total review every 2 years

11. Required Changes in Current Systems
Shorten the review cycle

12. This is Not an Ford Assembly Line
Evaluate Scientific
Merit of Applications
Receipt
Refer

13. Shortening the NIH Review Cycle, Initial Steps
For most research grants, we are posting summary statements within one month after the study section meeting instead of two to three months after the meeting (effective Oct 05)
We are conducting a pilot study to speed the review process for new investigators so they may revise and resubmit for the very next review cycle 4 months earlier than before (effective Feb 06)

14. Desirable Changes in CSR Review
Shorten the review cycle
Address concern that clinical research is not properly evaluated
Improve the assessment of innovative, high- risk/high-reward research

15. “The judging of grants has become a charade.”
The American Society for Cell Biology
“The judging of grants has become a charade. To be funded, the experimental plan has become a litany of experiments already accomplished so that everything is feasible. When grants come back with unfundable scores, new investigators may not have sufficient resources to do the experiments that “show feasibility.”
Zena Werb
President, ASCB
Newsletter August 2005

16. Possible Changes in Current Systems
Shorten the review cycle
Address concern that clinical research is not properly evaluated
Improve the assessment of innovative, high- risk/high-reward research
Do more to recruit and retain more high-quality reviewers

17. Expanding Peer Review’s Platforms
Study Sections
Electronic Reviews
Telephone Enhanced Discussions
Video Enhanced Discussions
Asynchronous Electronic Discussions
Necessity ● Clinical reviewers
Preference ● Physicists, computational biologists
New Opportunities ● Fogarty, International Reviewers

18. The First NIH Study Section 1945
The Last NIH Study Section

19. Possible New Systems
If we didn’t have any peer-review system and we had to design one from scratch, what would it look like?

21. Questions—Applications
Should we have 3 or 365 deadlines for most applications?
Should applications (Rs) be shortened? Should appendices be eliminated or reduced in size?
Is there more value in having 2-3 reviewers reading 25-page applications or reviewers reading 5-page applications?

22. Questions—Study Section Meetings
What is the ideal number of members to have serve on a study section?
Is one study section with reviewers efficient?
What is the intellectual contribution of individual reviewers in large study sections?

23. Questions—Scoring
Is it proper or desirable to have reviewers voting on priority scores for each application referred to their study section?
Is consensus always good? Or should we focus on score variance?

24. Applications Received for All of NIH FY 1998-2004
80,000
60,000
Number of applications
40,000
20,000
1998
2000
2002
2004
Fiscal year

25. Number of Research Grant Applications/Applicant

26. CSR Applications Reviewed, Regular and SEP May Council Only

27. Study Section Application/Reviewer Ratio October Council Only

28. Two groups of challenges/opportunities
Reviewers
Decrease the number of reviewers and increase the quality
Increase the number of applications reviewed without extra workload
Recruit and retain the best reviewers
SRA
Increase efficiency
Recruit and train

29. Possible Short Term Approaches for Increasing Efficiency for Reviewers and CSR
Replace Many SEPs with Smaller Parallel Study Sections
Enlarge Study Section Membership and Decrease Frequency of Participation
Pre Meeting Streamlining
Various Review Platforms
Hybrid Review Platforms
Staggering Application Deadlines
2 instead of 3 reviews
Shorten Applications
More Structured Applications and Reviews

30. Size of Grant Applications
RO1
Will increase number of applications reviewed by reviewers
Will decrease the number of reviewers in a study section
May be combined with a change in format of the application, more consonant to review criteria
May be combined with scoring individual criteria (i.e. relevance, innovation, etc
Strong support by councils and scientific leadership
May result in more innovation

31. This is CSR

32. Coronary Heart Disease
Age-Adjusted Death Rates in U.S.:
Actual (blue) vs Expected (yellow)
500
400
300
200
100
1950
1955
1960
1965
1970
1975
1980
1985
1990
1995
2000
Deaths per 100,000
Year
~ 514,000 Actual Deaths
in 2000
~ 1,329,000 Projected Deaths
815,000 Deaths
Prevented in 2000

33. The True Value of Peer Review