Presentation is loading. Please wait.

Presentation is loading. Please wait.

NIH Peer Review: Continuity and Change

Similar presentations

Presentation on theme: "NIH Peer Review: Continuity and Change"— Presentation transcript:

1 NIH Peer Review: Continuity and Change
Toni Scarpa Center for Scientific Review National Institutes of Health Department of Health and Human Services APS/Cardiovascular Section Salt Francisco, April 2006

2 Peer Review: An N.I.H. “Conception”
Is the heart and soul of NIH Has produced an effective partnership between the federal government and research institutions Has created the best academic medical centers, the best biomedical/behavioral research and biotechnology Has made possible the best cures and the best prevention Has been admired and imitated here and abroad Has protected NIH against outside influence

3 Center for Scientific Review

4 Applications received for all of NIH and applications referred for CSR review, FY 1998-2004
80,000 60,000 Number of applications 40,000 20,000 1998 2000 2002 2004 Fiscal year Applications received for all of NIH Applications assigned for review by CSR

5 CSR 4 Review Divisions & 23 IRGs
Division of Biologic Basis of Disease Elliot Postow, Ph.D. Division of Molecular and Cellular Mechanisms Donald Schneider, Ph.D. Division of Physiology and Pathology Michael Martin, Ph.D. Division of Clinical and Population-Based Studies Anita Miller Sostek, Ph.D AIDS and Related Research IRG (AARR) Ranga V. Srinivas, Ph.D. Bioengineering Sciences and Technologies IRG (BST) Sally Amero, Ph.D. Cardiovascular Sciences IRG (CVS) Joyce Gibson, D.Sc. Behavioral & Biobehavioral Processes IRG (BBBP) Karen Sirocco, Ph.D. Biology of Development and and Aging (BDA) Sherry Dupere, Ph.D. Digestive Sciences IRG (DIG) Mushtaq Khan, Ph.D., DVM Brain Disorders and Clinical Neuroscience IRG (BDCN) Dana Plude, Ph.D. Endocrinology, Metabolism, Nutrition, and Reproductive Sciences (EMNR) Sooja Kim, Ph.D. Biological Chemistry and Macromolecular Biophysics IRG (BCMB) John Bowers, Ph.D. Hematology IRG (HEME) Joyce Gibson, D.Sc. Health of the Population IRG (HOP) Robert Weller, Ph.D. Immunological Sciences IRG (IMM) Calbert Laing, Ph.D. Integrative, Functional and Cognitive Neuroscience IRG (IFCN) Christine Melchior, Ph.D. Cell Biology IRG (CB) Marcia Steinberg, Ph.D. Risk, Prevention, and Health Behavior IRG (RPHB) Michael Micklin, Ph.D. Infectious Diseases and Microbiology IRG (IDM) Alex Politis, Ph.D. Musculoskeletal, Oral, and Skin Sciences IRG (MOSS) Daniel McDonald, Ph.D. Genes, Genomes, and Genetics IRG (GGG) Richard Panniers, Ph.D. Surgery, Radiology, and Bioengineering IRG (SRB) Eileen Bradley, D.Sc. Oncological Sciences IRG (ONC) Syed Quadri, Ph.D. Renal and Urological Sciences IRG (RUS) Daniel McDonald, Ph.D. Molecular, Cellular and Developmental Neuroscience IRG (MDCN) Carole Jelsema, Ph.D. Respiratory Sciences IRG (RES) Mushtaq Khan, Ph.D., DVM

6 CSR Mission Statement To see that NIH grant applications receive fair, independent, expert, and timely reviews -- free from inappropriate influences -- so NIH can fund the most promising research.

7 Necessary Changes in CSR Peer Review Operations
CSR Operations Current Systems New Systems? Complexity and Impact Time

8 Changes in CSR Operations
Increase communications between CSR, the ICs, our reviewers and applicants Increase uniformity Increase efficiency Facilitate work of IC program staff

9 Potential of Knowledge Management Tools for Peer Review
Collexis Software or Others Knowledge management solutions Fingerprinting and text retrieving Disease coding Benefits for Peer Review Assigning applications to Integrated Review Groups or Study Sections Selecting reviewers (one application, multiple applications) Nine pilots are underway to begin to assess these benefits

10 Study Section Realignment
Review of one IRG every month Total review every 2 years

11 Required Changes in Current Systems
Shorten the review cycle

12 This is Not an Ford Assembly Line
Evaluate Scientific Merit of Applications Receipt Refer

13 Shortening the NIH Review Cycle, Initial Steps
For most research grants, we are posting summary statements within one month after the study section meeting instead of two to three months after the meeting (effective Oct 05) We are conducting a pilot study to speed the review process for new investigators so they may revise and resubmit for the very next review cycle 4 months earlier than before (effective Feb 06)

14 Desirable Changes in CSR Review
Shorten the review cycle Address concern that clinical research is not properly evaluated Improve the assessment of innovative, high- risk/high-reward research

15 “The judging of grants has become a charade.”
The American Society for Cell Biology “The judging of grants has become a charade. To be funded, the experimental plan has become a litany of experiments already accomplished so that everything is feasible. When grants come back with unfundable scores, new investigators may not have sufficient resources to do the experiments that “show feasibility.” Zena Werb President, ASCB Newsletter August 2005

16 Possible Changes in Current Systems
Shorten the review cycle Address concern that clinical research is not properly evaluated Improve the assessment of innovative, high- risk/high-reward research Do more to recruit and retain more high-quality reviewers

17 Expanding Peer Review’s Platforms
Study Sections Electronic Reviews Telephone Enhanced Discussions Video Enhanced Discussions Asynchronous Electronic Discussions Necessity ● Clinical reviewers Preference ● Physicists, computational biologists New Opportunities ● Fogarty, International Reviewers

18 The First NIH Study Section 1945
The Last NIH Study Section

19 Possible New Systems If we didn’t have any peer-review system and we had to design one from scratch, what would it look like?


21 Questions—Applications
Should we have 3 or 365 deadlines for most applications? Should applications (Rs) be shortened? Should appendices be eliminated or reduced in size? Is there more value in having 2-3 reviewers reading 25-page applications or reviewers reading 5-page applications?

22 Questions—Study Section Meetings
What is the ideal number of members to have serve on a study section? Is one study section with reviewers efficient? What is the intellectual contribution of individual reviewers in large study sections?

23 Questions—Scoring Is it proper or desirable to have reviewers voting on priority scores for each application referred to their study section? Is consensus always good? Or should we focus on score variance?

24 Applications Received for All of NIH FY 1998-2004
80,000 60,000 Number of applications 40,000 20,000 1998 2000 2002 2004 Fiscal year

25 Number of Research Grant Applications/Applicant

26 CSR Applications Reviewed, Regular and SEP May Council Only

27 Study Section Application/Reviewer Ratio October Council Only

28 Two groups of challenges/opportunities
Reviewers Decrease the number of reviewers and increase the quality Increase the number of applications reviewed without extra workload Recruit and retain the best reviewers SRA Increase efficiency Recruit and train

29 Possible Short Term Approaches for Increasing Efficiency for Reviewers and CSR
Replace Many SEPs with Smaller Parallel Study Sections Enlarge Study Section Membership and Decrease Frequency of Participation Pre Meeting Streamlining Various Review Platforms Hybrid Review Platforms Staggering Application Deadlines 2 instead of 3 reviews Shorten Applications More Structured Applications and Reviews

30 Size of Grant Applications
RO1 Will increase number of applications reviewed by reviewers Will decrease the number of reviewers in a study section May be combined with a change in format of the application, more consonant to review criteria May be combined with scoring individual criteria (i.e. relevance, innovation, etc Strong support by councils and scientific leadership May result in more innovation

31 This is CSR

32 Coronary Heart Disease
Age-Adjusted Death Rates in U.S.: Actual (blue) vs Expected (yellow) 500 400 300 200 100 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 Deaths per 100,000 Year ~ 514,000 Actual Deaths in 2000 ~ 1,329,000 Projected Deaths 815,000 Deaths Prevented in 2000

33 The True Value of Peer Review

Download ppt "NIH Peer Review: Continuity and Change"

Similar presentations

Ads by Google