1. The NIH Peer Review Process Sally A. Amero, Ph.D. NIH Review Policy Officer Office of Extramural Research 2010 NIH Regional Seminars

2. 22 The NIH Peer Review Process National Institutes of Health Primary Federal agency in the US for conducting and supporting biomedical research 27 Institutes & Centers (ICs)  Extramural and intramural programs  24 ICs have funding authority

3. 33 The NIH Peer Review Process Two-tiered process: – Initial peer review Scientific Review Groups (SRGs) “Study Sections” – I/C Advisory Council or Board (Council) Per year: – Nearly 80,000 applications – Over 18,000 reviewers

4. 44 The NIH Peer Review Process Overview Application received Assignments made  Initial peer review Funding considerations SRG; study section Institutes or Centers (ICs) IC or CSR*Duals possible Scientific Review Officer Program Officer  Second level of review  Funding decisions Council or Board (IC)IC Director  Award! *CSR = NIH Center for Scientific Review Institute Director Award

5. 55 The NIH Peer Review Process Division of Receipt and Referral, CSR Check for completeness Determine area of research Assign an identification number Assign a grant number Assign to NIH IC for possible funding Assign to a Scientific Review Group CSR = Central receiving point for all competing applications submitted to the NIH

6. 66 The NIH Peer Review Process CSR Review Most R01s, fellowships, and small business applications Some Program Announcements (PAs, PARs) Some Requests for Applications (RFAs) Institute/Center Review IC-specific features Program projects Training grants Career development awards RFAs Referral to an SRG Review locus is stated in Funding Opportunity Announcement

7. 77 The NIH Peer Review Process To Request a Scientific Review Group Cover letter of application – Application title – FOA # and title – Request: – Assignment to particular SRG or study section – Assignment to particular IC for funding consideration – Disciplines involved, if multidisciplinary – Explanation for late application Not all requests can be honored.

8. 88 The NIH Peer Review Process Cover Letter of Application Instructions – List one request per line – Place SRG & IC review requests on separate lines – Place positive & negative requests on separate lines – Include name of IC or SRG, followed by a dash and acronym – Explain each request in a separate paragraph

9. 99 The NIH Peer Review Process Confirmation of Receipt Email notifications – eSubmissions: Grants.gov and NIH – Paper submissions: Email to confirm assignment eRA Commons (http://era.nih.gov/commons/index.cfm)http://era.nih.gov/commons/index.cfm – Grant application number – SRG assignment – SRO contact information – Funding IC(s) assignment(s) – New Investigator or Early Stage Investigator status – Council round – Multi-PD/PI status

10. 10 The NIH Peer Review Process Information Center for Scientific Review: http://cms.csr.nih.gov/PeerReviewMeetings/CSRIRGDescription/ http://www.csr.nih.gov/committees/rosterindex.asp Institutes and Centers: http://era.nih.gov/roster/index.cfm Areas of IC interest: http://www.nih.gov/icd/index.html

11. 11 The NIH Peer Review Process Scientific Review Officer (SRO) First level of peer review – Designated Federal Official – Extramural scientist administrator – Identifies and recruits reviewers – Manages conflicts of interest – Oversees arrangements for review meetings – Presides at review committee meetings – Prepares and releases summary statements

12. 12 The NIH Peer Review Process Peer Reviewers Recruitment – Expertise – Stature in field – Mature judgment – Impartiality – Managed conflicts of interest – Balanced representation Gender Geography Diversity Seniority – Availability SRG rosters are posted 30 days before the SRG meeting! http://era.nih.gov/roster/index.cfm

13. 13 The NIH Peer Review Process Types of Scientific Review Groups (SRGs) “Chartered” SRGs – Multiyear terms – Formal appointment process – May include temporary members for special expertise Special Emphasis Panels (SEP) – Ad hoc membership – Often meet only once

14. 14 The NIH Peer Review Process Types of Reviewers Regular reviewers – Written critiques, criterion scores, preliminary impact/priority scores – Final impact/priority scores Other Contributing Reviewers – Written critiques, criterion scores, preliminary impact/priority scores – Cannot submit final impact/priority scores

15. 15 The NIH Peer Review Process Reviewer Assignments ≥ Three qualified reviewers (2 + 1) Based on scientific content of application Expertise of reviewer Suggestions from PI on types of expertise – not names! Suggestions from Program staff Suggestions from SRG members Managing conflicts of interest Balancing workload

16. 16 The NIH Peer Review Process Conflicts of Interest (COI) – Financial – Employment – Personal – Professional – SRG membership – Other interests Two COI vouchers are submitted by each SRG member. Pre-meeting and Post-meeting COI between a reviewer and an application:

17. 17 The NIH Peer Review Process Scientific Review Groups (SRGs) Make recommendations, not funding decisions! – Scientific and technical merit – Budget and project duration – Bars to award – human subjects, vertebrate animals, biohazards – Resource Sharing Plans – Other administrative factors  Impact/priority scores  Criterion scores  Written critiques

18. 18 The NIH Peer Review Process Scientific Review Groups (SRGs) Confidentiality – All materials, discussions, documents retrieved or – destroyed (except those in the public domain) – Reviewers sent guidance with applications – Application information secure – All questions must be referred to SRO – Meetings closed to the public – Program staff may observe

19. 19 The NIH Peer Review Process Overall Impact/Priority Score Reflects the reviewers’ assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved Five (or more) scored criteria Additional review criteria Additional review considerations

20. 20 The NIH Peer Review Process Overall Impact –Scored criteria: Significance* Investigator(s)* Innovation* Approach* Environment* –Additional review criteria –Additional review considerations Enhanced Review Criteria – Research Grants * Receive individual criterion scores See Side-by-side comparison (http://grants.nih.gov/grants/peer_review_process.htm)http://grants.nih.gov/grants/peer_review_process.htm

21. 21 The NIH Peer Review Process Impact and Significance Overall Impact Reviewers will provide an overall impact score to reflect their assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the (following) five core review criteria, and additional review criteria (as applicable for the project proposed).

22. 22 The NIH Peer Review Process Impact and Significance Significance – Does the project address an important problem or critical barrier to progress in the field? – If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? – How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

23. 23 The NIH Peer Review Process Impact and Significance Overall Impact – Not a sixth review criterion – Not necessarily the arithmetic mean of the criterion scores – Is the synthesis/integration of the scored and additional criteria – Reviewer(s) assess the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved – Likelihood (i.e., probability) is primarily derived from the investigator(s), approach and environment criteria. – Sustained powerful influence is primarily derived from the significance and innovation criteria.

24. 24 The NIH Peer Review Process Impact and Significance Significance – Is evaluated and scored independently of other criteria – Evaluation assumes that the “aims of the project are achieved” and/or will be “successfully completed.” – Reviewers should evaluate within the context of a (research) field(s). – Research field(s) may vary widely, so it would be helpful if reviewers identify in their reviews the research field(s) within which the project addresses an important problem or critical barrier to progress.

25. 25 The NIH Peer Review Process Additional Review Criteria –Are factored into overall impact/priority score –Not assigned individual scores “Review Criteria at a Glance” (http://grants.nih.gov/grants/peer/reviewer_guidelines.htm) Additional, specific criteria for mechanism, or FOA Protections for Human Subjects from Research Risks Inclusion of Women, Minorities & Children Vertebrate Animals Biohazards Resubmissions Renewals Revisions

26. 26 The NIH Peer Review Process Additional Review Considerations –Not factored into overall impact/priority score –Not assigned individual scores “Review Criteria at a Glance” (http://grants.nih.gov/grants/peer/reviewer_guidelines.htm) Additional, specific considerations Applications from Foreign Organizations Select Agents Research Resource Sharing Plans Budget & Period of Support

27. 27 The NIH Peer Review Process Enhanced Review Criteria – Fellowships Overall Impact/Merit – Scored criteria Fellowship Applicant Sponsors, Collaborators, and Consultants Research Training Plan Training Potential Institutional Environment & Commitment to Training

28. 28 The NIH Peer Review Process Enhanced Review Criteria – Fellowships Overall Impact/Merit –Additional Review Criteria Protections for Human Subjects from Research Risks Inclusion of Women, Minorities, & Children Vertebrate Animals Biohazards Resubmissions Renewals – Additional Review Considerations Responsible Conduct of Research Select Agents Research Resource Sharing Plans Budget & Period of Support

29. 29 The NIH Peer Review Process NIH Scoring System New system – Numerical scores 1.0 (exceptional) to 9.0 (poor) Final impact/priority score = average of individual scores x 10 Individual criterion scores Ranked by percentile for certain mechanisms – Not Discussed (ND) - streamlining – Other designations (NR, DF, AB, NP, etc.) Final impact/priority scores range from 10 through 90!

30. 30 The NIH Peer Review Process NIH Scoring System Preliminary scores (before the SRG meeting) – Entered by assigned reviewers and discussants in secure website – Made available to other SRG members Final overall impact/priority scores (at the SRG meeting) – Voted by private ballot – All eligible SRG members vote Reviewers are instructed to revise their criterion scores after the meeting.

31. 31 The NIH Peer Review Process New Score Descriptors Phases of Process ImpactScoreDescriptor High Impact 1Exceptional 2Outstanding 3Excellent Moderate Impact 4Very Good 5Good 6Satisfactory Low Impact 7Fair 8Marginal 9Poor

32. 32 The NIH Peer Review Process Streamlining Allows discussion of more meritorious applications – Research projects ~ 50% – Shared instrumentation ~ 40% – Fellowship applications ~ 30% – RFAs – pre - arranged limits Requires full concurrence of SRG Not discussed at SRG meeting Summary statement: – Reviewer critiques – Individual criterion scores – No final overall impact/priority score (ND)

33. 33 The NIH Peer Review Process Streamlining Score order of review – SRG discusses most meritorious applications first – Entire SRG decides when to stop, which applications will not be discussed in panel Other order of review (e.g., IC assignment, mechanism) – SRO prepares a list of average preliminary scores – Distributes to SRG – Entire SRG decides which applications to discuss

34. 34 The NIH Peer Review Process Pre-Meeting SRG Procedures SRO – Performs administrative review of applications – Recruits reviewers, arranges for meeting date and site – Assigns 3 SRG members to each application – Makes applications available to reviewers Internet Assisted Review (IAR) site or on CDs Usually about six weeks before the SRG meeting – Instructs reviewers in review procedures – Monitors posting of initial scores and critiques in IAR

35. 35 The NIH Peer Review Process Reviewer Instructions and Guidelines Reviewer Orientation Review Criteria at a Glance Overall Impact versus Significance Scoring System and Procedure Critique Template Instructions NIH Certification Forms Budget Information Human Subjects, Protection and Inclusion Vertebrate Animals Resource Sharing Plans Available at: http://grants.nih.gov/grants/peer/reviewer_guidelines.htm#general_guidelines

36. 36 The NIH Peer Review Process Structured Critiques Phases of Process New summary statement format –Bulleted comments from reviewers –Less text –Criterion scores –Decreases variability –Increases quality of information in critiques –More succinct, better organized –Encourages evaluative statements –Ensures that reviewers address all review criteria and considerations

37. 37 The NIH Peer Review Process Templates for Reviewers Phases of Process Links to definitions of review criteria http://grants.nih.gov/ grants/peer/reviewer_guidelines. htm#general_guidelines

38. 38 The NIH Peer Review Process Pre-Meeting SRG Procedures Reviewers – Examine assignments – Complete and submit Conflict of Interest and Confidentiality voucher – Read applications, prepare written critiques using formatted template – Enter preliminary scores into IAR – Read and consider other critiques and preliminary scores – Make travel and hotel arrangements Preliminary scores and critiques may be due several days or a week in advance!

39. 39 The NIH Peer Review Process SRG Meeting Procedures Late grant material – Acceptance at discretion of SRO Correcting errors or omissions, e.g. – Additional letters of support or collaboration – Corrected budget form pages – Information that a manuscript has been accepted Cannot modify Research Plan, except for new letters – Cannot be accepted with fewer than 30 days before the SRG meeting

40. 40 The NIH Peer Review Process SRG Meetings Agenda – Call to Order - Chairperson – Policy and instructions - SRO – Discuss applications one at a time – Where feasible: In score order Cluster NI applications Cluster clinical applications – Score each application after its discussion – Discuss other considerations Budget Resource Sharing Plans Remember the Mock Study Section tomorrow!

41. 41 The NIH Peer Review Process Clustering of New Investigator Applications New Investigator (NI): A PD/PI who has not yet competed successfully for a substantial NIH research grant Early Stage Investigator (ESI): An NI who is within 10 years of completing the terminal research degree or medical residency (or equivalent)

42. 42 The NIH Peer Review Process Clustering of New Investigator Applications New Investigator Policy – Support New Investigators at success rates equivalent to that of established investigators submitting new applications – Where feasible, NI applications are clustered in review – NI and ESI applications are identified for reviewers – Expectations for preliminary data or track record should – not be the same as for established investigators

43. 43 The NIH Peer Review Process SRG Meeting Procedures Discussion format – Members with conflicts excused – Initial levels of enthusiasm stated (assigned reviewers and discussants) – Primary reviewer - explains project, strengths, weaknesses – Other assigned reviewers and discussants follow – Open discussion (full panel) – Levels of enthusiasm (assigned reviewers) re-stated – Individual SRG members vote – Other review considerations discussed (budget)

44. 44 The NIH Peer Review Process SRG Meeting Procedures Reviewer workload ~ 6 – 8 as “reviewer” ~ 2 – 3 as “discussant” Don’t assume reviewers will – See the unstated – Grasp nuances – Understand your lingo – Look things up – Read your mind!

45. 45 The NIH Peer Review Process SRG Meeting Procedures If 60 applications/SRG meeting ~ 50% streamlined 30 applications to discuss and score Clarity and brevity are essential! Leaves ~ 14 minutes/application ~ 3 - 4 minutes/reviewer If 9 hour SRG meeting ~ ½ hour introduction, streamlining ~ 1 hour lunch, 2 x 15 minute breaks

46. 46 The NIH Peer Review Process Alternate Styles of Review Teleconferences Editorial-style review Video-enhanced discussions Internet Assisted Meetings

47. 47 The NIH Peer Review Process After the Review NIH Program Officer = Point of Contact Wait for summary statement Read summary statement carefully before calling! A favorable score does not guarantee funding!

48. 48 The NIH Peer Review Process After the Review eRA Commons (http://era.nih.gov/commons/index.cfm)http://era.nih.gov/commons/index.cfm – Final Impact/Priority Score available three days after the SRG meeting – Summary statement available 4 – 8 weeks after meeting Available to Program Officers at that time Confidential document Available to: – PD/PIs – NIH officials – Advisory Council members

49. 49 The NIH Peer Review Process Summary Statement First page – NIH Program Officer (upper left corner) Name Contact information – Final Impact/Priority Score or ND – Percentile (if applicable) – Codes Human subjects Vertebrate animals Inclusion plans – Budget request

50. 50 The NIH Peer Review Process Summary Statement - continued Subsequent Pages – Description (provided by applicant) – Resumé and Summary of Discussion (if discussed) – Reviewer critiques – essentially unedited Follow review criteria for mechanism Protections for Human Subjects Inclusion Plans Vertebrate Animals Biohazards Budget – Administrative Notes

51. 51 The NIH Peer Review Process Appeals Process Consider your options if the outcome is unfavorable – Revise and resubmit application Consider critiques in summary statement Address critiques in introduction and text – Appeal review outcome Procedural deficiencies Factual errors May result in re-review of same application by different SRG Discuss your options with your Program Officer!

52. 52 The NIH Peer Review Process Appeals Process Appeals are submitted to the NIH Program Officer All correspondence must be in writing All correspondence from the PO must be in hard copy Must address procedural flaws in the review, not differences of scientific opinion Can be submitted up to thirty days after Council Results of a re-review cannot be appealed further

53. 53 The NIH Peer Review Process Advisory Council/Board Second level of review – Make recommendations to IC Director: Research priority areas Policy Appeals Funding Quality of SRG review – Concur with SRG recommendations – Modify SRG recommendations – Deferral for re-review – Cannot change final impact/priority score from SRG –

54. 54 The NIH Peer Review Process Advisory Council/Board Members – Scientists from the extramural research community – Public representatives – Appointed to multi-year terms – Appointed as Special Government Employees – Subject to Standards of Ethics for federal employees Rosters: http://www1.od.nih.gov/cmo/committee/index.html http://www1.od.nih.gov/cmo/committee/index.html Schedule: http://www1.od.nih.gov/cmo/committee/index.html http://www1.od.nih.gov/cmo/committee/index.html

55. 55 The NIH Peer Review Process Advisory Council/Board Committee – Subject to the Federal Advisory Committee Act (FACA) – Balanced representation Expertise Stature in field Mature judgment Impartiality Managed conflicts of interest Balanced representation – Gender, Diversity – Geography, Seniority

56. 56 The NIH Peer Review Process Funding Considerations Funding decisions – IC Director – Scientific and technical merit (initial peer review) – Council recommendations – Relevance to program priorities in IC – Number of meritorious applications received – Availability of funds – Advise of IC Program Staff

57. 57 The NIH Peer Review Process Enhancing Peer Review Initiative http://enhancing-peer-review.nih.gov/ Office of Extramural Research Peer Review Process http://grants.nih.gov/grants/peer_review_process.htm Peer Review Policies & Practices http://grants.nih.gov/grants/peer/peer.htm Center for Scientific Review http://cms.csr.nih.gov/AboutCSR/Welcome+to+CSR/ Additional Information

58. 58 The NIH Peer Review Process Contact Information Sally Amero, Ph.D. NIH Review Policy Officer Extramural Research Integrity Liaison Officer Office of Extramural Programs Office of Extramural Research National Institutes of Health ameros@od.nih.gov