1. 1 CSR’s Mission and Function and What’s New in Peer Review Martha M. Faraday, Ph.D. Scientific Review Officer Division of AIDS, Behavioral & Population Sciences Risk Prevention & Health Behavior IRG Psychosocial Risk & Disease Prevention Study Section Date : April 22, 2009 National Institutes of Health U.S. Department of Health and Human Services

2. National Institutes of Health National Institute on Alcohol Abuse and Alcoholism National Institute on Alcohol Abuse and Alcoholism National Institute of Arthritis and Musculoskeletal and Skin Diseases National Institute of Arthritis and Musculoskeletal and Skin Diseases National Cancer Institute National Cancer Institute National Institute on Drug Abuse National Institute on Drug Abuse National Institute of Environmental Health Sciences National Institute of Environmental Health Sciences National Institute on Aging National Institute on Aging National Institute of Child Health and Human Development National Institute of Child Health and Human Development National Institute on Deafness and Other Communication Disorders National Institute on Deafness and Other Communication Disorders National Eye Institute National Eye Institute National Human Genome Research Institute National Human Genome Research Institute National Institute of Mental Health National Institute of Mental Health National Institute of Neurological Disorders and Stroke National Institute of Neurological Disorders and Stroke National Institute of General Medical Sciences National Institute of General Medical Sciences National Institute of Nursing Research National Institute of Nursing Research National Library of Medicine National Library of Medicine Center for Scientific Review Center for Scientific Review National Center for Complementary and Alternative Medicine National Center for Complementary and Alternative Medicine National Institute of Allergy and Infectious Diseases National Institute of Allergy and Infectious Diseases Fogarty International Center Fogarty International Center National Center for Research Resources National Center for Research Resources Clinical Center Clinical Center National Center on Minority Health and Health Disparities National Center on Minority Health and Health Disparities National Institute of Biomedical Imaging and Bioengineering National Institute of Biomedical Imaging and Bioengineering Office of the Director Center for Information Technology Center for Information Technology National Heart, Lung, and Blood Institute National Heart, Lung, and Blood Institute National Institute of Dental and Craniofacial Research National Institute of Dental and Craniofacial Research National Institute of Diabetes and Digestive and Kidney Diseases National Institute of Diabetes and Digestive and Kidney Diseases

3. CSR Mission Statement To see that NIH grant applications receive fair, independent, expert, and timely reviews - free from inappropriate influences - so the Institutes and Centers within the NIH can fund the most promising research.

4. CSR Peer Review: 2008 Statistics 77,000 applications received77,000 applications received 56,000 applications reviewed56,000 applications reviewed 16,000 reviewers16,000 reviewers 240 Scientific Review Officers240 Scientific Review Officers 1,600 review meetings1,600 review meetings

5. Scientific Review Process Dual Review System for Grant Applications Second Level of Review NIH Institute/Center Council NIH Institute/Center Council First Level of Review : CSR/Institute Review Scientific Review Group (SRG) (Study Section)

6. Translational and Clinical Sciences Translational and Clinical Sciences Cardiovascular and Respiratory Sciences Cardiovascular and Respiratory Sciences Surgical Sciences, Biomedical Imaging and Bioengineering Surgical Sciences, Biomedical Imaging and Bioengineering Musculoskeletal, Oral And Skin Sciences Musculoskeletal, Oral And Skin Sciences Oncology 2 – Translational Clinical Oncology 2 – Translational Clinical Vascular and Hematology Hematology Physiological and Pathological Sciences Physiological and Pathological Sciences Endocrinology,Metabolism, Nutrition & Reproductive Sciences Reproductive Sciences Endocrinology,Metabolism, Nutrition & Reproductive Sciences Reproductive Sciences Immunology Immunology Infectious Diseases & Microbiology Infectious Diseases & Microbiology Digestive, Kidney & Urological Systems Digestive, Kidney & Urological Systems Neuroscience, Development and Aging Neuroscience, Development and Aging Brain Disorders & Clinical Neuroscience Brain Disorders & Clinical Neuroscience Molecular, Cellular & Developmental Neuroscienc Developmental Neuroscienc e Molecular, Cellular & Developmental Neuroscienc Developmental Neuroscienc e Integrative, Functional & Cognitive Neuroscience Integrative, Functional & Cognitive Neuroscience Emerging Technologies & Training in Neuroscience Emerging Technologies & Training in Neuroscience Biology of Development & Aging Biology of Development & Aging Biobehavioral & Biobehavioral & Behavioral Processes Behavioral Processes Biobehavioral & Biobehavioral & Behavioral Processes Behavioral Processes Risk, Prevention& Health Behaviors Risk, Prevention& Health Behaviors Population Sciences & Epidemiology Epidemiology Healthcare Delivery & Methodologies Healthcare Delivery & Methodologies AIDS & Related Research AIDS & Related Research AIDS, Behavioral and Population Sciences AIDS, Behavioral and Population Sciences Basic and Integrative Biological Sciences Basic and Integrative Biological Sciences Biological Chemistry & MacromolecularBiophysics MacromolecularBiophysics Bioengineering Sciences & Technologies Bioengineering Sciences & Technologies Genes, Genomes & Genetics Genes, Genomes & Genetics Oncology 1 – Basic Translational Translational Cell Biology InterdisciplinaryMolecular & Training InterdisciplinaryMolecular & Training CSR Review Divisions

7. Standing Study Sections when the subject matter of the application matches the referral guidelines for the study section Special Emphasis Panels (SEPs) when the subject matter does not fit into any study section, or when assignment of an application to the most appropriate study section would create a conflict of interest. Also used for special mechanisms (e.g., fellowships, SBIRs, AREAs) Within an IRG, applications are assigned for review to Assignment to CSR Review Groups

8. Read the instructions Never assume that reviewers “will know what you mean” Refer to the literature thoroughly State rationale of proposed investigation Include well-designed tables and figures Present an organized, lucid write-up Remember to address human subjects, vertebrate animals, potential biohazards; these could affect your score Obtain pre-review from faculty at your institution NIH Grant Writing Tips: http://grants.nih.gov/grants/grant_tips.htm http://grants.nih.gov/grants/grant_tips.htm When Preparing Your Application

9. Directing Your Application to a Specific Study Section Peruse CSR Study Section Guidelines* to Identify a Possible Home for Your Application http://csr.nih.gov/ * Recently revised; alternative study sections listed in approximate order of degree of overlap Submit a Cover Letter

10. About CSR About CSR News and Reports News and Reports Peer Review Meetings Resources for Applicants Resources for Applicants CSR Web Site: http://www.csr.nih.govhttp://www.csr.nih.gov Study Section Descriptions & Rosters

11. Role of Scientific Review Officer (SRO) Performs administrative and technical review of applications to ensure completeness and accuracy Selects reviewers based on broad input Manages study section meetings Prepares summary statements Provides any requested information about study section recommendations to Institutes/Centers and National Advisory Councils/Boards Designated Federal official with overall responsibility for the review process

12. WHOM DO I CONTACT?  Before review, contact the Scientific Review Officer in CSR  After review, contact your Program Officer in the NIH funding institute or center

13. Pre-Meeting Review Process Appropriate reviewers recruited by SRO; minimum of 3 “interactive” reviewers per application Conflicts of interest identified Applications made available to reviewers ~6 weeks prior to meeting Critiques and preliminary scores posted by assigned reviewers on NIH web site at least 2-3 days prior to meeting Critiques and preliminary scores (excluding conflicts) available to review group prior to meeting

14. Where Do We Find Reviewers? National Registry for Society- Recommended Reviewers Successful applicants Word of mouth Recommendations from study section members Recommendation from NIH IC staff CRISP (crisp.cit.nih.gov) PubMed Scientific Conferences

15. Traditional* Review Meeting Process Upper half applications discussed: Reviewers are guided by specific review criteria Protections for Humans, Vertebrate Animals, Environment (Biohazard) may affect final score Assigned reviewers recommend scores for each application in upper half; all members not in conflict vote their conscience (outlier score policy pertains) Other considerations not affecting final score are discussed (e.g., budget, foreign applicants, resource sharing plans) Lower half applications not discussed, not assigned an overall score * Aspects of this process will change in May, 2009 http://enhancing-peer-review.nih.gov

16. Post Meeting Review Process Scores are provided to investigators within 3 working days Summary Statements for discussed and scored applications include Resume & Summary of Discussion, (largely unedited) critiques, and other recommendations (e.g., Budget) Summary Statements for lower half (Not Discussed) applications receive (largely unedited) critiques and review criteria scores but no overall impact scores All Summary Statements are made available within 30 days of meeting (10 days for new investigators’ R01s)

17. What’s New in Peer Review?

18. Enhancing Peer Review “Fund the best science, by the best scientists, with the least administrative burden…” Elias Zerhouni, MD, Former Director, NIH Enhancing Peer Review “Fund the best science, by the best scientists, with the least administrative burden…” Elias Zerhouni, MD, Former Director, NIH 2008: The Year of Peer Review

19. Priority 4: Continuous Review of Peer Review Priority 1: Engage the Best Reviewers Increase Flexibility to Better Accommodate Reviewers Recruit Reviewers Acknowledge Reviewers more formally Compensate Reviewers Time and Effort Improve Review Quality with Training Priority 2: Improve the Quality & Transparency of Review Modify Rating System to Focus on Specific Review Criteria Align Summary Statement with Review Criteria Shorten and Align Application with Review Criteria Priority 3: Ensure Balanced & Fair Reviews Across Scientific Fields and Career Stages Support for Early Stage Investigators Review of Established Investigators Enhanced Review of Clinical Research Expand Awards Encouraging “Transformative Research” Reduce Need for Resubumissions Recommendations

20. Amended Applications: To speed the funding of meritorious science and minimize reviewer burden: single amendment (A1)As of January 25, 2009, all original new applications (i.e., never submitted) and competing renewal applications will be permitted only a single amendment (A1).

21. What’s New in Peer Review New Investigators/Early Stage InvestigatorsNew Investigators/Early Stage Investigators Enhanced Review CriteriaEnhanced Review Criteria Template-Based CritiquesTemplate-Based Critiques Scoring Scale (9 point scale)Scoring Scale (9 point scale) Criterion Scoring Overall Impact Score 21

22. New Investigators/Early Stage Investigators New Investigator (NI):New Investigator (NI): PD/PI who has not yet competed successfully for a substantial NIH research grant oFor multiple PD/PIs-all PD/PIs must meet requirements for NI status Early Stage Investigator (ESI):Early Stage Investigator (ESI): PD/PI who qualifies as a New Investigator AND is within 10 years of completing the terminal research degree or is within 10 years of completing medical residency (or equivalent) Applies only to R01 applicationsApplies only to R01 applications New Investigators/Early Stage Investigators will be clustered together for reviewNew Investigators/Early Stage Investigators will be clustered together for review 22

23. Enhanced Review Criteria Overall Impact:Overall Impact: exert a sustained, powerful influence on the research field(s) involved Assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved New Core Criteria Order:New Core Criteria Order: Significance Investigator(s) Innovation Approach Environment oReview criteria enhanced and expanded 23

24. Critiques Goal: To improve the quality of the critiques and to focus reviewer attention on the review criteria: Provide clear, concise, and explicit information Aid in identifying the strengths and weaknesses of each criterion Template-Based Critiques

25. Critique template contains a total of 18 boxesCritique template contains a total of 18 boxes Reviewers should provide text for only those criteria that are applicable. 25 1. Significance 6. Resubmission 13. Overall Impact 2. Investigator(s) 7. Renewal14. Budget and Period of Support 3. Innovation 8. Revision15. Select Agents 4. Approach 9. Protection of Human Subjects 16. Applications from Foreign Organization 5. Environment 10. Inclusion of Women, Minorities, and Children 17. Resource Sharing Plan 11 Vertebrate Animals 18. Additional Comments to Applicant 12. Biohazards

26. Template-Based Critiques Goal: is to write evaluative statements and to discourage summarizing the applicationGoal: is to write evaluative statements and to discourage summarizing the application Comments should be in the form of bullet points or if necessary short narratives Do not record scores Do not record scores on the critique template The entire template is uploaded to IAR to become part of the summary statement. 26 Significance 1. Significance Please limit text to ¼ page Strengths  Weaknesses 

27. Goal: To improve the transparency of the scoring process: five review criteria using a scale of 1-9Score applications on five review criteria using a scale of 1-9. overall impact score using 1-9 scalePreliminary overall impact score using 1-9 scale. Should not be the average of the criterion scores. Not Discussed Not Discussed applications will receive initial criterion scores from the three assigned reviewers Scoring – 9 Point Scale

28. Scoring Descriptions ImpactScoreDescriptorStrengths/Weaknesses High Impact 1Exceptional Weaknesses 2Outstanding 3Excellent Moderate Impact 4Very Good 5Good 6Satisfactory Low Impact 7Fair 8Marginal 9Poor Strengths

29. 29Clustering NI/ESI R01NI/ESI R01 applications will be clustered together in review. ESI applications will not be separately clustered within the NI\ESI group. ESI applications will not be separately clustered within the NI\ESI group. oNI/ESI applications will be identified for reviewers so there can be appropriate review in context of career stage. oExpectations of preliminary data and publication track record less than for established investigators.

30. Order of Review Goal: Discuss applications in order of average preliminary score. Why: Concern - variation of scores during different times of the meeting. One recommendation was to recalibrate scores at the end of the meeting. Solution: Recalibrate “dynamically” throughout meeting.

31. Order of Review For calibration purposes:For calibration purposes: Begin meeting by discussing the best scored application (any activity code) oNI/ESI R01s clustered beginning of meeting oAll other activity codes clustered if feasible (if at least 10 discussed (may include R03, R15, and R21s as a group that can be clustered )

32. Order of Review Summary Discussion order is based on the average of the impact scores from assigned reviewers Final scores of discussed applications may differ from preliminary scores as re- calibration happens dynamically

33. Discuss ~ 50-60% of applications SRO will then ask if there are any other applications that panel wishes to discuss The remaining applications will not be discussed (applications receive criterion scores only) oSame after review of ~60% of SBIR applications Not Discussed

34. Final Scores Final Scores Discussed applications will receive an overall score from each eligible (i.e., without conflicts of interest) panel member and these scores will be averaged to one decimal place, and multiplied by 10. The 81 possible priority scores will thus range from 10-90. Percentiles will be reported in whole numbers.

35. shorterSummary statement will be shorter and more focused. Discussedsummary of discussionDiscussed applications will also have a summary of the panel’s discussion at the meeting. ALL applications will be scoredALL applications will be scored. Not discussed criterion scores Not discussed applications will receive criterion scores only. Summary Statements

36. Recruiting the Best Reviewers Move a meeting a year to the West Coast Additional review platforms Develop a national registry of volunteer reviewers Searchable database with 4,000 reviewers Provide tangible rewards for reviewers No submission deadlines for chartered members of study sections (effective February 2008). 1574 chartered members used flexible deadlines during the last 6 months Provide flexible time for reviewers Choice of 3 times/year for 4 years or 2 times/year for 6 years

37. THANK YOU! This is CSR