Summary of NIH Enhancing Peer Review Implementation Changes to NIH Proposals due on or after January 25, 2010 Slide Content Provided by Dr. Michael Sesma, NIH

The NIH Peer Review Process has changed New Aspects in Review Enhanced review criteria New scoring system Criterion scoring Structured critiques Clustering of New Investigator Applications Score order of review * These changes have been implemented for applications considered for FY2010 funding and for ARRA FOAs

Enhanced Review Criteria Core review criteria order: Core review criteria order: Significance* Significance* Investigator(s)* Investigator(s)* Innovation* Innovation* Approach* Approach* Environment* Environment* Additional review criteria & considerations expanded Additional review criteria & considerations expanded Side-by-side comparison available Side-by-side comparison availablehttp://grants.nih.gov/grants/peer_review_process.htm Overall Impact  Overall Impact/Priority Score *Will receive individual criterion scores

Enhanced Review Criteria Core review criteria for Career Award: Core review criteria for Career Award: Candidate* Candidate* Career Development Plan/Career Goals & Objectives/Plan to Provide Mentoring * Career Development Plan/Career Goals & Objectives/Plan to Provide Mentoring * Research Plan* Research Plan* Mentor(s), Consultant(s), Collaborator(s)* Mentor(s), Consultant(s), Collaborator(s)* Environment and Institutional Commitment* Environment and Institutional Commitment* Additional review criteria & considerations Additional review criteria & considerations Side-by-side comparison available Side-by-side comparison availablehttp://grants.nih.gov/grants/peer_review_process.htm Overall Impact  Overall Impact/Priority Score *Will receive individual criterion scores

Changes Happening NOW: Enhanced Review Criteria Overall Impact/Priority Score Reflects the reviewers’ assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved In consideration of: Core criteria Core criteria Additional review criteria (RFA or PAR) Additional review criteria (RFA or PAR) Additional review criteria – as applicable Additional review criteria – as applicable

Enhancing Peer Review at NIH: Enhanced Review Criteria Additional Review Criteria Protections for Human Subjects Protections for Human Subjects Inclusion of Women, Minorities, and Children Inclusion of Women, Minorities, and Children Vertebrate Animals Vertebrate Animals Resubmission Applications Resubmission Applications Renewal Applications Renewal Applications Revision Applications Revision Applications Biohazards Biohazards Budget and Period Support Budget and Period Support Select Agent Research Select Agent Research Applications from Foreign Organizations Applications from Foreign Organizations Resource Sharing Plans Resource Sharing Plans As applicable for the project proposedAs applicable for the project proposed Reviewers will consider in the determination of scientific and technical merit Reviewers will consider in the determination of scientific and technical merit Reviewers will not give separate scores for these items.Reviewers will not give separate scores for these items.

Enhancing Peer Review at NIH: New Scoring System 1 = Exceptional 9 = Poor Reduces number of rating discriminationsReduces number of rating discriminations Provides rating descriptorsProvides rating descriptors –To improve reliability –To encourage use of the entire range Scores rounded to fewer digitsScores rounded to fewer digits 9-Point Scale

Enhancing Peer Review at NIH: New Scoring System New Score Descriptors Phases of Process ImpactScoreDescriptor High Impact 1Exceptional 2Outstanding 3Excellent Moderate Impact 4 Very Good 5Good 6Satisfactory Low Impact 7Fair 8Marginal 9Poor

Enhancing Peer Review at NIH Structured Critiques Phases of Process Bullet comments on Strengths/WeaknessesBullet comments on Strengths/Weaknesses Decrease variability of review Decrease variability of review Increase quality of information in critiques Increase quality of information in critiques More succinct, better organized More succinct, better organized Encourage evaluative statements Encourage evaluative statements Scores for five review criteriaScores for five review criteria Ensure that reviewers address all review Ensure that reviewers address all review criteria and considerations criteria and considerations Required comments:Required comments: –Protections for Human Subjects –Inclusion Plans –Vertebrate Animal Welfare –Biohazards –Budget

Enhancing Peer Review at NIH Clustering of New Investigator Applications Where feasible, NI applications will be clustered Where feasible, NI applications will be clustered NI and ESI applications will be identified for reviewers NI and ESI applications will be identified for reviewers Expectations for preliminary data or track record should not be the same as for established investigators Expectations for preliminary data or track record should not be the same as for established investigators Where feasible, discussion order based on: Where feasible, discussion order based on: – Clustering of New Investigator applications – Clustering of clinical applications – Clustering of similar activity codes – Preliminary overall impact/priority scores

Enhancing Peer Review Website:Enhancing Peer Review Website:( NOT-OD :NOT-OD :( Side-by-side comparison of enhanced and former Side-by-side comparison of enhanced and former review criteria review criteria( Guidelines for ReviewersGuidelines for Reviewers( Additional Information Enhancing Peer Review at NIH

13 Enhancing Peer Review Overview and Timeline January 2009 May/June 2009 January 25, 2010 Submissions Phase out of A2 applications Identification of Early Stage Investigator (ESI) applications Phase out of A2 applications Identification of Early Stage Investigator (ESI) applications Enhanced review criteria New scoring Criterion scoring Structured critiques Clustering of New Inv. Applications Score order review Enhanced review criteria New scoring Criterion scoring Structured critiques Clustering of New Inv. Applications Score order review Alignment of applications & review criteria Shorter Research Plans Alignment of applications & review criteria Shorter Research Plans

14 Enhancing Peer Review at NIH Changes Happening NOW (Beginning with Submissions after January 25, 2010*, FY2011 funding) Alignment of applications & review criteria Alignment of applications & review criteria Restructured Applications Restructured Applications Shorter Research Plans: new page limits Shorter Research Plans: new page limitshttp://grants.nih.gov/grants/guide/notice-files/NOT-OD htmlhttp://enhancing-peer-review.nih.gov/restructured_applications.html *NIH Plans a Fall Release of new Forms *Stay Tuned for updates

Implementation 1. New FOAs are being posted without electronic application packages  Read the new instructions NOW 2. By December, new application packages will be posted, along with new Parent Announcements, and updated active FOAs  Paper and Electronic 3. Applicants must return to the FOA & download new forms and instructions for due dates on or after January 25, 2010  Resubmissions must also use new application forms and instructions (and shorter page limits)

Alignment of Application with Review Criteria

Restructuring the Research Plan: Significance, Innovation, Approach

Revisions to Biographical Sketch: Investigator(s) Personal Statement added: Personal Statement added: –Briefly describe why your experience and qualifications make you particularly well- suited for your role in the project Publications revised: Publications revised: –Limit the list of publications or manuscripts to no more than 15 –Make selections based on recency, importance to the field, and/or relevance to the application Page limit remains at 4 (except DP1/DP2) Page limit remains at 4 (except DP1/DP2)

Revisions to Application: Environment  Instructions added to Resources: –Provide a description of how the scientific environment will contribute to the probability of success of the project –For ESIs describe the institutional investment in the success of the investigator  Instructions added to Research Plan –In Select Agent Research section, describe the biocontainment resources available at all performance sites

20 Page Limit Revisions Shorter Research Plans *FOA page limits should be followed if they differ from application instructions.

What do the changes mean for the Applicants and Grant Administrators? Applicants must return to FOA or reissued Parent Announcement to download new forms. Forms will be available by December 1. Applications submitted using incorrect forms will be delayed and may not be reviewed! A SINGLE UPLOAD for the Research Strategy (previously a 3-file upload) increases applicant control over the look and feel of the application Simpler preparation for applicants Simpler preparation for applicants Easier reading for reviewers Easier reading for reviewers

Identifying New Application Forms New forms are required for both paper PHS 398 and electronic SF 424 (R&R) New forms are required for both paper PHS 398 and electronic SF 424 (R&R) –Funding opportunity announcement (FOA) or Parent Announcement –NIH Forms & Applications Page: Be sure to choose the correct application package: Be sure to choose the correct application package: –SF 424 (R&R): ADOBE_FORMS_B –PHS 398: Revision date “June 2009”

General Information Sources Enhancing Peer Review at NIH Webpages Overview of Peer Review Enhancements Video

The NIH Peer Review Process Enhancing Peer Review Initiative Enhancing Peer Review Initiative Office of Extramural Research Office of Extramural Research Peer Review Process Peer Review Process Peer Review Policies & Practices Peer Review Policies & Practices Center for Scientific Review Center for Scientific Review Additional Information

Decoding Your NIH Grant Number 1 R01 EB A1 1 R01 EB A1 Application Type Activity Code Institute Code Serial Number Support Year Extension 1 = new 2 = renewal 3 = supplement 5 = non- competing competing continuation continuation R = Research project P = Program project or Center or Center T = Training (institutional) (institutional) F = Fellowship (individual) (individual) K = Career Development Development U = Cooperative agreement agreement AA = NIAAA AG = NIA AI = NIAID AR = NIAMS AT = NCCAM CA = NCI DA = NIDA DC = NIDCD DE = NIDCR DK = NIDDK EB = NIBIB ES = NIEHS EY = NEI GM = NIGMS HD = NICHD HG = NHGRI HL = NHLBI LM = NLM MD = NCMHD MH = NIMH NR = NINR NS = NINDS RR = NCRR TW = FIC Unique, up to six digits Years of Continuous Funding A1 = first resubmission resubmission A2 = second resubmission resubmission