1. Overview Benefits of Serving as a Reviewer Summary of Recent Changes Review Time-Line Review Criteria Review Scoring “Impact” vs. “Significance” Ethics of Reviewing Current Controversies Reviewer Qualifications How to Become a Reviewer

2. Benefits of Serving as a Reviewer Get a front row seat to the future Become a more successful scientist Learn more Meet new colleagues Become a better mentor Give back Shape the future www.csr.nih.gov

3. Summary of Recent Changes Emphasis on “impact” Bullet point format of reviews “Tutorial” comments not encouraged Discussion ordered by preliminary score New investigator applications separated

4. Review Time Line Pre-assignment phase – 2 weeks Review and submission phase – 4 to 6 weeks Read phase – 5 days Meeting phase – 2 days Edit phase – 3 days

5. Scored Review Criteria Impact Significance Investigators Innovation Approach Environment

6. Impact Summarize significant overall strengths and weaknesses Assess the likelihood that the project will exert a sustained powerful influence on the field

7. Significance Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability or clinical practice be improved? How will successful completion of the aim change the concepts, methods, technologies, treatments, services or preventive interventions that drive this field?

8. Investigator Are the researchers well suited to the project? If Early Stage or New Investigators (only R01), do they have appropriate experience and training? If established, have they demonstrated an ongoing record of accomplishments that have advanced their fields? If the project is collaborative or multi-PI, do the investigators have complementary and integrated expertise; are their leadership approach, governance and organizational structure appropriate for the project?

9. Innovation Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field or novel in a broad sense? Is it a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation or interventions proposed?

10. Approach Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the project? Are potential problems, alternative strategies, and benchmarks for success presented? If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed?

11. Frequent Mistakes Description substituted for critique Contradiction between strengths and weaknesses Extreme brevity, especially if application is not discussed Focus on the disease as opposed to the application Mismatching between scores and narrative Focus on minutiae, as in a manuscript review Verbosity

12. Additional Review Criteria* Protection for human subjects Inclusion of woman, minorities and children Vertebrate animals Biohazards Resubmission Renewal Revision (competing supplement) Multiple PI plan *can affect overall score

13. Protection for Human Subjects* Evaluate risks as acceptable or unacceptable Evaluate protections as adequate or inadequate Items for evaluation –Risk to subjects –Adequacy of protection against risks –Potential benefits to subjects and others –Importance of the knowledge to be gained –Data and safety monitoring for clinical trials –Data and Safety Monitoring Plan –Justification for exempt status *can affect overall score via “approach”

14. Inclusion of Women, Minorities and Children* *can affect overall score via “approach” A = Acceptable; U = Unacceptable

15. Vertebrate Animals* Evaluate as acceptable or unacceptable –Proposed use of animals –Species, strains, ages, sex and numbers –Justification for use of animals and for appropriateness of the species and numbers proposed –Adequacy of veterinary care –Procedures to limit discomfort, distress, pain and injury –Methods of euthanasia Worksheet available from CSR website *can affect overall score via approach

16. Additional Review Considerations* Application from foreign organizations Select agents Resource sharing plans Data sharing Model organisms Genome-wide association studies Budget and period of support *do NOT affect overall score

17. “Impact” vs. “Significance” Three key phrases for “Impact” –Likelihood—derived from assessment of investigator, approach and environment –Sustained powerful influence—derived from significance and innovation –Research field—should be identified “Impact” is not –a 6 th review criterion –the mean of scored criteria

18. “Cheever Types” Type 1 –project is important –applicant is able to do the work

19. “Cheever Types” Type 1 –project is important –applicant is able to do the work Type 2 –project is important –applicant not able to do the work

20. “Cheever Types” Type 1 –project is important –applicant is able to do the work Type 2 –project is important –applicant not able to do the work Type 3 –project not important –applicant is able to do the work

21. “Cheever Types” Type 1 –project is important –applicant is able to do the work Type 2 –project is important –applicant not able to do the work Type 3 –project not important –applicant is able to do the work Type 4 –project not important –applicant not able to do the work

22. Review Scoring

23. Categories of Weakness Minor –An easily addressable weakness that does not substantially lessen impact Moderate –A weakness that lessens impact Major –A weakness that severely limits impact

24. Budget and Period of Support Is the budget realistic? Reductions can be recommended –Insufficient budget justification –Insufficient information about work in later years –Project can be completed in fewer years or with smaller budget Panel recommendation summarized by SRO Budget does not affect overall priority score Overlap concerns noted in written comments (not discussed)

25. Additional Comments to Applicant Elaboration of ideas that are too complex for bullet points Further explain deficiencies in the application Avoid “tutorial” suggestions for improvement

26. Before the Meeting Pre-assignment phase –Declare conflicts of interest Review and submission phase –Read the instructions! –Make travel arrangements –Review, complete and sign pre-review certification –Read, evaluate and write critique for assigned applications –Upload preliminary scores and critiques before deadline Read phase –Read other critiques for assigned applications –Prepare for discussion during meeting

27. Overview of the Meeting Welcome and call to order by chair Panel members introduce themselves Introductory remarks and instruction by SRO Order of review –New investigator R01 applications –Other R01 applications –R21 applications Order determined by average of preliminary overall scores from assigned reviewers ~ 50% of applications are discussed

28. How Reviews are Discussed Conflicted members leave the room Applicant and title announced by chair Preliminary overall scores from reviewers Discussion by primary reviewer Additional comments by other reviewers Human subjects and animal welfare concerns Panel discussion Summary by chair Revised overall scores announced Voting outside range Budget recommendations Conflicted members called back

29. How to be Effective During the Meeting Submit initial reviews before the deadline Read comments from other reviewers before the meeting Suggestions from experienced reviewers Relax Talk about what matters Don’t rehash Focus on the application Don’t mince words Avoid mentoring Paint the big picture Paint a balanced picture Give key background No unreasonable demands No favoritism by discipline Be flexible and have fun

30. After the meeting Edit phase –Revise criterion scores to reflect discussion –Revise critiques to reflect changes after discussion –Sign the electronic Conflict of Interest form

31. Ethics of Reviewing Conflicts acknowledged appropriately No use of non-public information for private purposes No consultation with anyone other than SRO Applications discussed only at the designated time (no outside discussion) No review-related communication with applicant Protection of confidentiality –No discussion with other individuals –All review materials password-protected –All review materials destroyed after meeting No allegations of research misconduct during discussion or in critique (discuss confidentially with SRO)

32. Current Controversies American Association of Immunologists –“Study sections must engage in an exceedingly thoughtful process, with much discussion, the sharing of varied opinions, and the vocal expression of differing views. Providing this meaningful feedback is an essential part of the peer review process.” Francis Collins, Director NIH (reply) –“I sense that the central issue may be providing feedback to applicants. Although we have made an effort to separate scientific assessment and advice to applicants because we do not want reviewers to re-write the applications, the review template does specifically contain a box where reviewers are invited to give feedback to applicants.”

33. Reviewer Qualifications Substantial independent research experience Prior peer-reviewed grants (R01 or equivalent) Understand the review process Dedicated to high quality, fair reviews Willing to consider service for a 4-year term Other considerations –Diversity of geographic representation –Diversity of gender, race and ethnicity

34. Characteristics of Great Reviewers Fairness and objectivity Ability to articulate views succinctly Ability to engage in productive exchanges Ability to discuss non-assigned applications Ability to work collegially in a group setting Ability to facilitate or focus discussion Effort to ensure fairness and consistency of scoring throughout the meeting

35. How Much Time Does it Take? Reading a 25-page application ─ 2 to 4 hours Writing a critique ─ 1 to 1.5 hours Uploading critiques ─ 1 hour Typical number of assigned applications ─ 8 to 10 Meeting ─ 3 days, including travel Meetings per year ─ 3

36. How to Become a Reviewer Notify professional society or research dean –Nomination to CSR National Registry of Volunteer Reviewers Express interest to a current reviewer –Scientific Review Officers appreciate suggestions Send CV to SRO who oversees a review group that could use your expertise –Descriptions of study section scope available at http://cms.csr.nih.gov/PeerReviewMeetings/CSRIRGDescription/

37. Where to Get More Information Go to www.csr.nih.gov