1. Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.archivedOER Public Archive Home Page

2. Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa CSR Where we are and where we would like to be PRAC, August 2006

3. Changes in CSR Operations Our Vision for Peer Review

4. This is CSR

5. 1.Increased Communication and Transparency 2.Increase Uniformity 3.Increase Efficiency Changes in CSR Operations

6. 1. Increased Communication and Transparency Within CSR –Biweekly meetings with SRAs and other staff –New comprehensive communications plan With NIH and other Agencies With the Scientific Community –Expanded Peer Review Notes newsletter –Six Open House Meetings with leaders of scientific societies and disease groups

7. Changes in CSR Operations 2. Increase Uniformity Summary Statements –Post all within 1 month of meeting (97.3%) –Post new investigator summary statements within 1 week –Produce complete and structured resumes Unscoring –Unscore 50% (49.7%) Best Practices Committee Assessments –Appropriate Rosters –Types of Meetings –Structure of Summary Statements

8. Changes in CSR Operations 3. Increase Efficiency Electronic Submission Text Fingerprinting, Artificial Intelligence Software –Assigning applications to Integrated Review Groups or Study Sections Major pilot to directly assign applications to Integrated Review Groups and/or Study Sections will begin in October, with implementation by February 2007.

9. Scoring – Current Situation Compression of scoring “Approach is King” and “Significance” has been downgraded Reasons for scoring often not clear in summary statements Scoring of A2 applications may be inappropriately influenced by reviewers knowing A1 score

10. IRG and Study Sections Realignment Increasing concern from applicants, societies and diseases groups about the variance in the breadth of science covered by study sections.

11. Monitoring IRGs and Study Sections 1.Every IRG is being assessed by all senior CSR staff every 2 years in addition to the current 5-year assessments involving all stakeholders. 2.Emergent problems addressed by working groups with leaders from the extramural community, and senior NIH/CSR staff. 3.Substantive issues/changes reviewed by the NIH Peer Review Advisory Committee. 4.Most study sections visited by CSR’s Director and senior staff at least once a year. 5.All retiring study section chairs called by CSR’s Director to learn about problems and possible improvements. 6.All summary statements read by CSR’s Director. 7.Multiple outreach efforts now gather fresh input.

12. IRG Review Schedule Scheduled 2007 -- 9 IRGs Biology of Development and Aging (BDA) Infectious Diseases and Microbiology (IDM) Biobehavioral and Behavioral Processes (BBBP) Cell Biology (CB) Musculoskeletal, Oral and Skin Sciences (MOSS) Oncological Sciences (ONC) Surgical Sciences, Biomedical Imaging and Bioengineering (SBIB) Respiratory Sciences (RES) Renal and Urological Sciences (RUS) Scheduled 2006 --14 IRGs Biological Chemistry and Macromolecular Biophysics (BCMB) Cardiovascular Science (CVS) Bioengineering Sciences and Technologies (BST) AIDS and Related Research (AARR) Risk, Prevention, and Health Behavior (RPHB) Genes, Genomes and Genetics (GGG) Digestive Sciences (DIG) Endocrinology, Metabolism, Nutrition and Reproductive Sciences (EMNR) Brain Disorders and Clinical Neuroscience (BDCN) Integrative, Functional and Cognitive Neuroscience (IFCN) Molecular, Cellular and Developmental Neuroscience (MDCN) Hematology (HEME) Immunology (IMM) Health of the Population (HOP)

13. Monitoring IRGs and Study Sections 1.Every IRG is being assessed by all senior CSR staff every 2 years in addition to the current 5-year assessments involving all stakeholders. 2.Emergent problems addressed by working groups with leaders from the extramural community, and senior NIH/CSR staff. 3.Substantive issues/changes reviewed by the NIH Peer Review Advisory Committee. 4.Most study sections visited by CSR’s Director, Deputy Director, and senior staff at least once a year. 5.All retiring study section chairs called to learn about problems and possible improvements. 6.Multiple outreach efforts now gather fresh input. 7.Open house meetings to be held.

14. Broad Scientific Areas Biomolecular (4): Biological Chemistry and Macromolecular Biophysics (BCMB); Bioengineering Sciences and Technologies (BST); Cell Biology (CB); Genes, Genomes and Genetics (GGG) Integrated Biological (5): Immunology (IMM); Hematology (HEME); Cardiovascular Sciences (CVS); Respiratory Sciences (RES); Biology of Development and Aging (BDA) Integrated Biological (4): Digestive Sciences (DIG); Musculoskeletal, Oral and Skin Sciences (MOSS); Renal and Urological Sciences (RUS) Endocrinology, Metabolism, Nutrition and Reproductive Sciences (EMNR) Disease-based (4): AIDS and Related Research (AARR); Infectious Diseases and Microbiology (IDM); Oncological Sciences (ONC); Surgical Sciences, Biomedical Imaging and Bioengineering (SBIB) Neurological (3): Brain Disorders and Clinical Neuroscience (BDCN); Integrative, Functional and Cognitive Neuroscience (IFCN); Molecular, Cellular and Developmental Neuroscience (MDCN) Behavioral/Social (3): Biobehavioral and Behavioral Processes (BBBP); Health of the Population (HOP); Risk Prevention and Health Behavior (RPHB)

15. Our Agenda 1.Shorten the review cycle 2.Address concern that clinical research is not properly evaluated 3.Improve the assessment of innovative, high- risk/high-reward research 4.Do more to recruit and retain more high-quality reviewers

16. Our Agenda 1.Shorten the review cycle

17. Shortening the NIH Review Cycle, Initial Steps Pilot study with ~600 new investigators in 40 study sections who may revise and resubmit for the very next review cycle 4 months earlier than before (Effective Feb 06) Summary statements posted within one month after the study section meeting

18. Short Review Cycle Pilot of New Investigator R01 Applications New Investigator R01 Applications NumberPercent Reviewed in Pilot631100% Amended and Submitted for the Next Round8614%

19. R01 A1 Resubmissions in the Next Round

20. Our Agenda 1.Shorten the review cycle 2.Address concern that clinical research is not properly evaluated

21. 17.56 22.07 M. Martin, CSR/NIH/DHHS

22. Our Agenda 1.Shorten the review cycle 2.Address concern that clinical research is not properly evaluated 3.Improve the assessment of innovative, high- risk/high-reward research

23. Our Agenda 1.Shorten the review cycle 2.Address concern that clinical research is not properly evaluated 3.Improve the assessment of innovative, high- risk/high-reward research 4.Do more to recruit and retain more high-quality reviewers

24. Applications Received for all of NIH and Applications Referred for CSR Review, FY 1998 - 2006 NIH Applications Applications for CSR Review

25. Number of Research Grant Applications/Applicant

26. Growth of R01 Applications Reviewed at CSR vs. Other ICs

27. Growth of R21 Applications Reviewed at CSR vs. Other ICs

28. Institutes and Centers Use R21s Differently R21s fund ~14 types of efforts, such as— –Exploratory/developmental research –Junior investigators –Phase I/II clinical trials –International research planning –High-risk/payoff research R21s have broad parameters— –Award period: 1-3 years –Funding allowed: $100K-$450K –Research Plan: 10-20 pages

29. Average Number of Applications Per Reviewer October Council Only

30. Academic Rank of All CSR Reviewers Standing and Ad Hoc Combined

31. Reviewers – Current Situation Far too many reviewers on study sections –Broader science –Decrease in reviewer load –Unnecessary Too many ad hoc reviewers Different processes for enlisting chartered and ad hoc members Often too many telephone reviewers

32. Challenges to Recruiting Reviewers Manage the need for 20,000 reviewers Decrease the number of reviewers and increase the quality Increase the number of applications reviewed without extra workload Recruit and retain the best reviewers Have smaller study sections with fewer ad hoc reviewers

33. Possible Approaches for Recruiting and Retaining the Best Reviewers Require less travel by using electronic review modes

34. Expanding Peer Review Platforms Electronic Reviews Telephone Enhanced Discussions Video Enhanced Discussions Asynchronous Electronic Discussions Our Goal: 10% of all reviews to be electronic in 2007

35. Electronic Review Methods Number of Applications Reviewed by Year

36. Possible Approaches for Recruiting and Retaining the Best Reviewers Require less travel by using electronic review modes Have shorter meetings Use Various Review Platforms Unscore 40% of the F32s (postdoctoral fellowships) Shorten Applications

37. Size of Grant Applications RO1 Will increase number of applications reviewed by reviewers Will decrease the number of reviewers in a study section May be combined with a change in format of the application, more consonant to review criteria Strong support by councils and scientific leadership May result in better focus on significance (and innovation)

38. Trans-NIH Committee to Shorten the Application Chairs Robert Finkelstein, Ph.D. (Co-Chair), NINDS Donald Schneider, Ph.D. (Co-Chair), CSR Members Mary Custer, Ph.D. CSR Ann A. Hagan, Ph.D. NIGMS Craig A. Jordan, Ph.D. NIDCD Sherry Mills, M.D., M.P.H. OER Philip F. Smith, Ph.D. NIDDK Barbara Spalholz NCI Betsy Wilder, Ph.D. NIDDK Alan L. Willard, Ph.D. NINDS

39. Special Thanks