Presentation on theme: "NIH Peer Review: Continuity and Change"— Presentation transcript:
1 NIH Peer Review: Continuity and Change Toni ScarpaCenter for Scientific ReviewNational Institutes of HealthDepartment of Health and Human ServicesAPS/Cardiovascular SectionSalt Francisco, April 2006
2 Peer Review: An N.I.H. “Conception” Is the heart and soul of NIHHas produced an effective partnership between the federal government and research institutionsHas created the best academic medical centers, the best biomedical/behavioral research and biotechnologyHas made possible the best cures and the best preventionHas been admired and imitated here and abroadHas protected NIH against outside influence
4 Applications received for all of NIH and applications referred for CSR review, FY 1998-2004 80,00060,000Number of applications40,00020,0001998200020022004Fiscal yearApplications received for all of NIHApplications assigned for review by CSR
5 CSR 4 Review Divisions & 23 IRGs Division of BiologicBasis of DiseaseElliot Postow, Ph.D.Division of Molecularand Cellular MechanismsDonald Schneider, Ph.D.Division of Physiologyand PathologyMichael Martin, Ph.D.Division of Clinical andPopulation-Based StudiesAnita Miller Sostek, Ph.DAIDS and RelatedResearch IRG (AARR)Ranga V. Srinivas, Ph.D.Bioengineering Sciences andTechnologies IRG (BST)Sally Amero, Ph.D.Cardiovascular SciencesIRG (CVS)Joyce Gibson, D.Sc.Behavioral & BiobehavioralProcesses IRG (BBBP)Karen Sirocco, Ph.D.Biology of Development andand Aging (BDA)Sherry Dupere, Ph.D.Digestive Sciences IRG(DIG)Mushtaq Khan, Ph.D., DVMBrain Disorders and ClinicalNeuroscience IRG (BDCN)Dana Plude, Ph.D.Endocrinology, Metabolism, Nutrition, and Reproductive Sciences (EMNR)Sooja Kim, Ph.D.Biological Chemistry and Macromolecular Biophysics IRG (BCMB)John Bowers, Ph.D.HematologyIRG (HEME)Joyce Gibson, D.Sc.Health of the PopulationIRG (HOP)Robert Weller, Ph.D.Immunological SciencesIRG (IMM)Calbert Laing, Ph.D.Integrative, Functionaland Cognitive NeuroscienceIRG (IFCN)Christine Melchior, Ph.D.Cell Biology IRG (CB)Marcia Steinberg, Ph.D.Risk, Prevention, and HealthBehavior IRG (RPHB)Michael Micklin, Ph.D.Infectious Diseases andMicrobiology IRG (IDM)Alex Politis, Ph.D.Musculoskeletal, Oral, and Skin Sciences IRG (MOSS)Daniel McDonald, Ph.D.Genes, Genomes, and Genetics IRG (GGG)Richard Panniers, Ph.D.Surgery, Radiology, andBioengineering IRG (SRB)Eileen Bradley, D.Sc.Oncological SciencesIRG (ONC)Syed Quadri, Ph.D.Renal and Urological Sciences IRG (RUS)Daniel McDonald, Ph.D.Molecular, Cellular andDevelopmentalNeuroscience IRG (MDCN)Carole Jelsema, Ph.D.Respiratory Sciences IRG (RES)Mushtaq Khan, Ph.D., DVM
6 CSR Mission StatementTo see that NIH grant applications receive fair, independent, expert, and timely reviews -- free from inappropriate influences -- so NIH can fund the most promising research.
7 Necessary Changes in CSR Peer Review Operations CSR Operations Current Systems New Systems?Complexity and ImpactTime
8 Changes in CSR Operations Increase communications between CSR, the ICs, our reviewers and applicantsIncrease uniformityIncrease efficiencyFacilitate work of IC program staff
9 Potential of Knowledge Management Tools for Peer Review Collexis Software or OthersKnowledge management solutionsFingerprinting and text retrievingDisease codingBenefits for Peer ReviewAssigning applications to Integrated Review Groups or Study SectionsSelecting reviewers (one application, multiple applications)Nine pilots are underway to begin to assess these benefits
10 Study Section Realignment Review of one IRG every monthTotal review every 2 years
11 Required Changes in Current Systems Shorten the review cycle
12 This is Not an Ford Assembly Line Evaluate ScientificMerit of ApplicationsReceiptRefer
13 Shortening the NIH Review Cycle, Initial Steps For most research grants, we are posting summary statements within one month after the study section meeting instead of two to three months after the meeting (effective Oct 05)We are conducting a pilot study to speed the review process for new investigators so they may revise and resubmit for the very next review cycle 4 months earlier than before (effective Feb 06)
14 Desirable Changes in CSR Review Shorten the review cycleAddress concern that clinical research is not properly evaluatedImprove the assessment of innovative, high- risk/high-reward research
15 “The judging of grants has become a charade.” The American Society for Cell Biology“The judging of grants has become a charade. To be funded, the experimental plan has become a litany of experiments already accomplished so that everything is feasible. When grants come back with unfundable scores, new investigators may not have sufficient resources to do the experiments that “show feasibility.”Zena WerbPresident, ASCBNewsletter August 2005
16 Possible Changes in Current Systems Shorten the review cycleAddress concern that clinical research is not properly evaluatedImprove the assessment of innovative, high- risk/high-reward researchDo more to recruit and retain more high-quality reviewers
21 Questions—Applications Should we have 3 or 365 deadlines for most applications?Should applications (Rs) be shortened? Should appendices be eliminated or reduced in size?Is there more value in having 2-3 reviewers reading 25-page applications or reviewers reading 5-page applications?
22 Questions—Study Section Meetings What is the ideal number of members to have serve on a study section?Is one study section with reviewers efficient?What is the intellectual contribution of individual reviewers in large study sections?
23 Questions—ScoringIs it proper or desirable to have reviewers voting on priority scores for each application referred to their study section?Is consensus always good? Or should we focus on score variance?
24 Applications Received for All of NIH FY 1998-2004 80,00060,000Number of applications40,00020,0001998200020022004Fiscal year
25 Number of Research Grant Applications/Applicant
26 CSR Applications Reviewed, Regular and SEP May Council Only
27 Study Section Application/Reviewer Ratio October Council Only
28 Two groups of challenges/opportunities ReviewersDecrease the number of reviewers and increase the qualityIncrease the number of applications reviewed without extra workloadRecruit and retain the best reviewersSRAIncrease efficiencyRecruit and train
29 Possible Short Term Approaches for Increasing Efficiency for Reviewers and CSR Replace Many SEPs with Smaller Parallel Study SectionsEnlarge Study Section Membership and Decrease Frequency of ParticipationPre Meeting StreamliningVarious Review PlatformsHybrid Review PlatformsStaggering Application Deadlines2 instead of 3 reviewsShorten ApplicationsMore Structured Applications and Reviews
30 Size of Grant Applications RO1Will increase number of applications reviewed by reviewersWill decrease the number of reviewers in a study sectionMay be combined with a change in format of the application, more consonant to review criteriaMay be combined with scoring individual criteria (i.e. relevance, innovation, etcStrong support by councils and scientific leadershipMay result in more innovation
32 Coronary Heart Disease Age-Adjusted Death Rates in U.S.:Actual (blue) vs Expected (yellow)50040030020010019501955196019651970197519801985199019952000Deaths per 100,000Year~ 514,000 Actual Deathsin 2000~ 1,329,000 Projected Deaths815,000 DeathsPrevented in 2000