Presentation on theme: "Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular."— Presentation transcript:
Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular Section Salt Francisco, April 2006
Peer Review: An N.I.H. Conception Is the heart and soul of NIH Has produced an effective partnership between the federal government and research institutions Has created the best academic medical centers, the best biomedical/behavioral research and biotechnology Has made possible the best cures and the best prevention Has been admired and imitated here and abroad Has protected NIH against outside influence
Applications received for all of NIH and applications referred for CSR review, FY 1998-2004 0 20,000 40,000 60,000 80,000 1998200020022004 Fiscal year Number of applications Applications received for all of NIHApplications assigned for review by CSR
CSR 4 Review Divisions & 23 IRGs Division of Biologic Basis of Disease Elliot Postow, Ph.D. AIDS and Related Research IRG (AARR) Ranga V. Srinivas, Ph.D. Endocrinology, Metabolism, Nutrition, and Reproductive Sciences (EMNR) Sooja Kim, Ph.D. Immunological Sciences IRG (IMM) Calbert Laing, Ph.D. Oncological Sciences IRG (ONC) Syed Quadri, Ph.D. Infectious Diseases and Microbiology IRG (IDM) Alex Politis, Ph.D. Division of Physiology and Pathology Michael Martin, Ph.D. Cardiovascular Sciences IRG (CVS) Joyce Gibson, D.Sc. Integrative, Functional and Cognitive Neuroscience IRG (IFCN) Christine Melchior, Ph.D. Renal and Urological Sciences IRG (RUS) Daniel McDonald, Ph.D. Hematology IRG (HEME) Joyce Gibson, D.Sc. Digestive Sciences IRG (DIG) Mushtaq Khan, Ph.D., DVM Division of Clinical and Population-Based Studies Anita Miller Sostek, Ph.D Surgery, Radiology, and Bioengineering IRG (SRB) Eileen Bradley, D.Sc. Health of the Population IRG (HOP) Robert Weller, Ph.D. Risk, Prevention, and Health Behavior IRG (RPHB) Michael Micklin, Ph.D. Brain Disorders and Clinical Neuroscience IRG (BDCN) Dana Plude, Ph.D. Behavioral & Biobehavioral Processes IRG (BBBP) Karen Sirocco, Ph.D. Division of Molecular and Cellular Mechanisms Donald Schneider, Ph.D. Bioengineering Sciences and Technologies IRG (BST) Sally Amero, Ph.D. Biology of Development and and Aging (BDA) Sherry Dupere, Ph.D. Biological Chemistry and Macromolecular Biophysics IRG (BCMB) John Bowers, Ph.D. Cell Biology IRG (CB) Marcia Steinberg, Ph.D. Genes, Genomes, and Genetics IRG (GGG) Richard Panniers, Ph.D. Molecular, Cellular and Developmental Neuroscience IRG (MDCN) Carole Jelsema, Ph.D. Respiratory Sciences IRG (RES) Mushtaq Khan, Ph.D., DVM Musculoskeletal, Oral, and Skin Sciences IRG (MOSS) Daniel McDonald, Ph.D.
CSR Mission Statement To see that NIH grant applications receive fair, independent, expert, and timely reviews -- free from inappropriate influences -- so NIH can fund the most promising research.
Time Complexity and Impact CSR Operations Current Systems New Systems? Necessary Changes in CSR Peer Review Operations
Increase communications between CSR, the ICs, our reviewers and applicants Increase uniformity Increase efficiency Facilitate work of IC program staff Changes in CSR Operations
Potential of Knowledge Management Tools for Peer Review Collexis Software or Others Knowledge management solutions Fingerprinting and text retrieving Disease coding Benefits for Peer Review Assigning applications to Integrated Review Groups or Study Sections Selecting reviewers (one application, multiple applications) Nine pilots are underway to begin to assess these benefits
Study Section Realignment Review of one IRG every month Total review every 2 years
Required Changes in Current Systems Shorten the review cycle
This is Not an Ford Assembly Line ReceiptRefer Evaluate Scientific Merit of Applications EnterpriseArchitecture@mail.nih.gov
Shortening the NIH Review Cycle, Initial Steps For most research grants, we are posting summary statements within one month after the study section meeting instead of two to three months after the meeting (effective Oct 05) We are conducting a pilot study to speed the review process for new investigators so they may revise and resubmit for the very next review cycle 4 months earlier than before (effective Feb 06)
Desirable Changes in CSR Review Shorten the review cycle Address concern that clinical research is not properly evaluated Improve the assessment of innovative, high- risk/high-reward research
The judging of grants has become a charade. The American Society for Cell Biology The judging of grants has become a charade. To be funded, the experimental plan has become a litany of experiments already accomplished so that everything is feasible. When grants come back with unfundable scores, new investigators may not have sufficient resources to do the experiments that show feasibility. Zena Werb President, ASCB Newsletter August 2005
Possible Changes in Current Systems Shorten the review cycle Address concern that clinical research is not properly evaluated Improve the assessment of innovative, high- risk/high-reward research Do more to recruit and retain more high-quality reviewers
Expanding Peer Reviews Platforms Electronic Reviews Telephone Enhanced Discussions Video Enhanced Discussions Asynchronous Electronic Discussions Study Sections Necessity Clinical reviewers Preference Physicists, computational biologists New Opportunities Fogarty, International Reviewers
The First NIH Study Section 1945 The Last NIH Study Section 2005
If we didnt have any peer-review system and we had to design one from scratch, what would it look like? Possible New Systems
QuestionsApplications Should we have 3 or 365 deadlines for most applications? Should applications (Rs) be shortened? Should appendices be eliminated or reduced in size? Is there more value in having 2-3 reviewers reading 25-page applications or 10-15 reviewers reading 5-page applications?
QuestionsStudy Section Meetings What is the ideal number of members to have serve on a study section? Is one study section with 50-70 reviewers efficient? What is the intellectual contribution of individual reviewers in large study sections?
QuestionsScoring Is it proper or desirable to have 50-70 reviewers voting on priority scores for each application referred to their study section? Is consensus always good? Or should we focus on score variance?
Applications Received for All of NIH FY 1998-2004 0 20,000 40,000 60,000 80,000 1998200020022004 Fiscal year Number of applications
Number of Research Grant Applications/Applicant
CSR Applications Reviewed, Regular and SEP May Council Only
Study Section Application/Reviewer Ratio October Council Only
Two groups of challenges/opportunities Reviewers –Decrease the number of reviewers and increase the quality –Increase the number of applications reviewed without extra workload –Recruit and retain the best reviewers SRA –Increase efficiency –Recruit and train
Possible Short Term Approaches for Increasing Efficiency for Reviewers and CSR Replace Many SEPs with Smaller Parallel Study Sections Enlarge Study Section Membership and Decrease Frequency of Participation Pre Meeting Streamlining Various Review Platforms Hybrid Review Platforms Staggering Application Deadlines 2 instead of 3 reviews Shorten Applications More Structured Applications and Reviews
Size of Grant Applications RO1 Will increase number of applications reviewed by reviewers Will decrease the number of reviewers in a study section May be combined with a change in format of the application, more consonant to review criteria May be combined with scoring individual criteria (i.e. relevance, innovation, etc Strong support by councils and scientific leadership May result in more innovation