1. Noni Byrnes René Etcheberrigaray Richard Nakamura Review of CSR’s study sections by areas of science

2. SRG: Scientific Review Group Study Section – managed by Scientific Review Officer (SRO) IRG: Integrated Review Group Organizational Unit – group of 6-10 scientifically-related study sections, each managed by an SRO, supervised by IRG Chief Definitions

3. Genes, Genomes and Genetics(GGG) IRG Genomics, Computational Biology and Technology (GCAT) Genetics of Health and Disease (GHD) Genetic Variation and Evolution (GVE) Molecular Genetics A (MGA) Molecular Genetics B (MGB) Prokaryotic Cell and Molecular Biology (PCMB) Therapeutic Approaches to Genetic Diseases Study Section (TAG) Bioengineering Sciences and Technologies (BST) IRG Biodata Management and Analysis (BDMA) Biomaterials and Biointerfaces (BMBI) Gene and Drug Delivery Systems (GDD) Instrumentation Systems and Development (ISD) Modeling and Analysis of Biological Systems (MABS) Nanotechnology (NANO) High Throughput Screening Assays for Probe Discovery (ZRG1-BST-U (55) Example: study sections have scientific overlap other study sections both within and outside their IRGs

4. “As recommended by the PSBR Report and the CSR Advisory Committee, CSR will examine the functioning of the new IRGs and study sections…...” Panel on Scientific Boundaries of Review (PSBR) 1999-2003

5. By IRG (examination of scientific guidelines of SRGs, rosters, summary statements, metrics, etc.) – Presentation by Chief, Q/A and internal review by CSR management By IRG (examination of scientific guidelines of study sections, rosters, summary statements, metrics, etc.) - Presentations by Chief and SROs, Q/A and internal review by CSR management By IRG – email input from selected reviewers from all SRGs, teleconference with study section Chairs, white paper by Chief with summary of external input from reviewers/Chair teleconference, presentations by Chief and SROs, Q/A and internal review by CSR management By IRG - external scientific working group in-person meeting, provided them bibliometric/scientific overlap, roster, metrics, study section guidelines, asked questions about quality, review process, white paper by Chief, presentation by Chief to external working group Evolution of the IRG Review Process at CSR

6. By IRG – external working group commented on operations in IRG, rosters, balance between regular/adhoc members, diversity, etc. Issues discussed includes – use of surveys, exit interviews, ranking, H-indices, bibliometrics, should Chairs be used to recruit new members, NIH A2 policy, new investigators, percent “not discussed”, IRG function (via discussion with SROs) The main scientific changes made to study sections were endorsement of proposals made by the IRG Chief in the white paper and presentation Latest IRG Evaluation Results – process vs. science

7. 1) Review by scientific clusters, not IRGs – each cluster with about 12-20 study sections (SRGs) 2) Assemble working group of scientifically broad, senior scientists, preferably with interest in more than one study section (SRG) 3) Provide enough information, time and guidance for meaningful evaluation and recommendation by working group 4) Ask 1-2 questions designed to focus discussion on science, not process Proposed Plan

8. 1) Review by scientific clusters, not IRGs – each cluster with about 12-20 study sections (SRGs) 2) Assemble working group of scientifically broad, senior scientists, preferably with interest in more than one study section (SRG) 3) Provide enough information, time and guidance for meaningful evaluation and recommendation by working group 4) Ask 1-2 questions designed to focus discussion on science, not process and provide specific guidance Proposed Plan

9. Boyack KW, Chen MC, Chacko G (2014) Characterization of the Peer Review Network at the Center for Scientific Review, National Institutes of Health. PLoS ONE 9(8): e104244. doi:10.1371/journal.pone.0104244 Figure 3. Consensus study section network.

10. CSR committee (senior management, plus IRG Chiefs representing each scientific division at CSR) Consensus network of study sections Scientific guidelines for study sections, list of overlapping study sections listed on the web Data on review of study section member applications in related study sections Close examination of scientific content of applications by committee members – abstracts/specific aims for content Input from Senior Staff, CSR IRG Chiefs – adjust scientific clusters 1. Form Scientific Clusters

11. 1) Review by scientific clusters, not IRGs – each cluster with about 12-20 study sections (SRGs) 2) Assemble working group of scientifically broad, senior scientists, preferably with interest in more than one study section (SRG) within a cluster 3) Provide enough information, time and guidance for meaningful evaluation and recommendation by working group 4) Ask 1-2 questions designed to focus discussion on science, not process Proposed Plan

12. 1) Review by scientific clusters, not IRGs – each cluster with about 12-20 study sections (SRGs) 2) Assemble working group of scientifically broad, senior scientists, preferably with interest in more than one study section (SRG) 3) Provide enough information, time and guidance for meaningful evaluation and recommendation by working group 4) Ask 1-2 questions designed to focus discussion on science, not process, and provide specific guidance Proposed Plan

13. Current scientific guidelines for each study section in cluster Titles, abstracts, specific aims for applications randomly selected from 3 rounds Workload trends for up to 9 rounds 3. Information Provided to Working Group

14. 1) Review by scientific clusters, not IRGs – each cluster with about 12-20 study sections (SRGs) 2) Assemble working group of scientifically broad, senior scientists, preferably with interest in more than one study section (SRG) 3) Provide enough information, time and guidance for meaningful evaluation and recommendation by working group 4) Ask 1-2 questions designed to focus discussion on science, not process, and provide specific guidance Proposed Plan

15. How well does the scientific scope of the study sections align with the current state of the science? Recommend modifications if: – a study section is currently over- or under-subscribed – the scope of the science reviewed by a study section is not adequately representative of the scope of the field and emerging fields 4. Questions for the Working Group

16. Read the study section guidelines for all 16 panels, look at the workload (including workload trends over the last 9 rounds), and then look over the sample abstracts/aims for each panel to get a better idea of what type of science is reviewed there. Does this study section: –Appear to be in the optimal size range (approximately 60-80 applications, +/- 10 ok) to encourage scientific breadth/competition – if not, recommend a change –Appear to map onto the current state of the scientific field well – if not, recommend a change –Appear to have cohesion – i.e. the scientific areas are broad but interrelated, not mismatched, if not, recommend a change “Change” can be: change in guidelines, merge study sections, create new study sections, move an area of science from one study section to another or several others, add areas of science that appear to be missing The only caveats in recommending a change – –there should be some scientific overlap between study sections – enough to ensure competition and a choice of review venue for investigators, but not so much that the science/expertise is scattered in too many places –any new study section should have enough applications in a broad, cohesive field to be sustainable, i.e. no small, boutique, insular panels. 4. (cont’d) Specific Guidance for Working Group

17. Non-hypothesis driven areas: Biomaterials Basic biomedical imaging Instrumentation/tools design Computational biology, modeling, informatics Proof of Concept: Cluster #1 (“bioengineering”)

18. Healthcare Delivery and Methodologies IRG (HDM) Biomedical Computing and Healthcare Informatics (BCHI) Biostatistical Methods and Research Design (BMRD) Community Influences on Health Behavior (CIHB) Community-Level Health Promotion (CLHP) Dissemination and Implementation Research in Health (DIRH) Health Disparities and Equity Promotion (HDEP) Understanding and Promoting Health Literacy SEP (HDM-X) Health Services Organization and Delivery (HSOD) Nursing and Related Clinical Sciences (NRCS) Surgical Sciences, Biomedical Imaging & Bioengineering IRG (SBIB) Biomedical Imaging Technology A (BMIT-A) Biomedical Imaging Technology B (BMIT-B) Bioengineering, Technology, and Surgical Sciences (BTSS) Clinical Molecular Imaging and Probe Development (CMIP) Clinical and Translational Imaging Applications (DTCS 81) Medical Imaging (MEDI) Surgery, Anesthesiology, and Trauma (SAT) Bioinformatics in Surgical Sciences, Biomedical Imaging and Bioengineering (ZRG1-SBIB-Q (80) Genes, Genomes and Genetics IRG (GGG) Genomics, Computational Biology and Technology (GCAT) Genetics of Health and Disease (GHD) Genetic Variation and Evolution (GVE) Molecular Genetics A (MGA) Molecular Genetics B (MGB) Prokaryotic Cell and Molecular Biology (PCMB) Therapeutic Approaches to Genetic Diseases Study Section (TAG) Bioengineering Sciences and Technologies IRG (BST) Biodata Management and Analysis (BDMA) Biomaterials and Biointerfaces (BMBI) Gene and Drug Delivery Systems (GDD) Instrumentation Systems and Development (ISD) Modeling and Analysis of Biological Systems (MABS) Nanotechnology (NANO) High Throughput Screening Assays for Probe Discovery (ZRG1-BST-U (55) Musculoskeletal, Oral and Skin Sciences IRG (MOSS) Arthritis, Connective Tissue and Skin (ACTS) Musculoskeletal Rehabilitation Sciences (MRS) Musculoskeletal Tissue Engineering (MTE) Oral, Dental and Craniofacial Sciences (ODCS) Skeletal Biology Development and Disease (SBDD) Skeletal Biology Structure and Regeneration (SBSR) Skeletal Muscle and Exercise Physiology (SMEP) Interdisciplinary Molecular Sciences and Training IRG (IMST) Enabling Bioanalytical and Imaging Technologies (EBIT) Biological Chemistry and Macromolecular Biophysics IRG (BCMB) Biochemistry and Biophysics of Membranes (BBM) Macromolecular Structure and Function A (MSFA) Macromolecular Structure and Function B (MSFB) Macromolecular Structure and Function C (MSFC) Macromolecular Structure and Function D (MSFD) Synthetic and Biological Chemistry A (SBCA) Synthetic and Biological Chemistry B (SBCB) Emerging Technologies and Training in Neurosciences IRG (ETTN) Bioengineering of Neuroscience, Vision and Low Vision Technologies (BNVT) Molecular Neurogenetics Study Section (MNG) Neuroscience and Ophthalmic Imaging Technologies (NOIT)

19. Healthcare Delivery and Methodologies IRG (HDM) Biomedical Computing and Healthcare Informatics (BCHI) Biostatistical Methods and Research Design (BMRD) Community Influences on Health Behavior (CIHB) Community-Level Health Promotion (CLHP) Dissemination and Implementation Research in Health (DIRH) Health Disparities and Equity Promotion (HDEP) Understanding and Promoting Health Literacy SEP (HDM-X) Health Services Organization and Delivery (HSOD) Nursing and Related Clinical Sciences (NRCS) Surgical Sciences, Biomedical Imaging & Bioengineering IRG (SBIB) Biomedical Imaging Technology A (BMIT-A) Biomedical Imaging Technology B (BMIT-B) Bioengineering, Technology, and Surgical Sciences (BTSS) Clinical Molecular Imaging and Probe Development (CMIP) Medical Imaging (MEDI) Surgery, Anesthesiology, and Trauma (SAT) Bioinformatics in Surgical Sciences, Biomedical Imaging and Bioengineering ZRG1-SBIB-Q (80) Genes, Genomes and Genetics IRG (GGG) Genomics, Computational Biology and Technology (GCAT) Genetics of Health and Disease (GHD) Genetic Variation and Evolution (GVE) Molecular Genetics A (MGA) Molecular Genetics B (MGB) Prokaryotic Cell and Molecular Biology (PCMB) Therapeutic Approaches to Genetic Diseases Study Section (TAG) Bioengineering Sciences and Technologies IRG (BST) Biodata Management and Analysis (BDMA) Biomaterials and Biointerfaces (BMBI) Gene and Drug Delivery Systems (GDD) Instrumentation Systems and Development (ISD) Modeling and Analysis of Biological Systems (MABS) Nanotechnology (NANO) High Throughput Screening Assays for Probe Discovery (ZRG1-BST-U (55) Musculoskeletal, Oral and Skin Sciences IRG (MOSS) Arthritis, Connective Tissue and Skin (ACTS) Musculoskeletal Rehabilitation Sciences (MRS) Musculoskeletal Tissue Engineering (MTE) Oral, Dental and Craniofacial Sciences (ODCS) Skeletal Biology Development and Disease (SBDD) Skeletal Biology Structure and Regeneration (SBSR) Skeletal Muscle and Exercise Physiology (SMEP) Interdisciplinary Molecular Sciences and Training IRG (IMST) Enabling Bioanalytical and Imaging Technologies (EBIT) Biological Chemistry and Macromolecular Biophysics IRG (BCMB) Biochemistry and Biophysics of Membranes (BBM) Macromolecular Structure and Function A (MSFA) Macromolecular Structure and Function B (MSFB) Macromolecular Structure and Function C (MSFC) Macromolecular Structure and Function D (MSFD) Synthetic and Biological Chemistry A (SBCA) Synthetic and Biological Chemistry B (SBCB) Emerging Technologies and Training in Neurosciences IRG (ETTN) Bioengineering of Neuroscience, Vision and Low Vision Technologies (BNVT) Molecular Neurogenetics Study Section (MNG) Neuroscience and Ophthalmic Imaging Technologies (NOIT)

20. Creation of new study sections in emerging areas (e.g. Microphysiological Systems, Clinical Informatics) Dissolution/redistribution of study sections that are undersubscribed or need to merge/redesign into new panels Modification of study section scope and guidelines to better map onto the scientific field (e.g. imaging committees) Evaluate with another cluster (e.g. MSFD) No change (e.g. MABS, ISD) Recommendations from Working Group Meeting (April 29, 2015)

21. Discussion with relevant CSR and IC staff about practical considerations Develop an implementation plan (when appropriate/feasible) Council advice – continue SRG evaluations with this type of cross-IRG process? Next Steps

22. Committee: Advisor: Richard Nakamura, Director, CSR Members: John Bowers, Acting Director, Division of Clinical & Translational Sciences (DTCS) Noni Byrnes, Director, Division of Basic & Integrative Biological Sciences (DBIB) Valerie Durrant, Acting Director, Division of Neuroscience, Development & Aging (DNDA) Rene Etcheberrigaray, Deputy Director Jonathan Ivins, Chief, Digestive, Kidney and Urological Sciences (DKUS) IRG Richard Panniers, Chief, Genes, Genomes & Genetics (GGG ) IRG Syed Quadri, Chief, Oncology 2 Translational Clinical (OTC) IRG Ray Jacobson, SRO, Bioengineering Science & Technologies (BST) IRG CSR Staff

23. QUACKENBUSH, JOHN, PHD PROFESSOR OF COMPUTATIONAL BIOLOGY ANDBIOINFORMATICS DEPARTMENT OF BIOSTATISTICS HARVARD SCHOOL OF PUBLIC HEALTH, DANA-FARBER CANCER INSTITUTE SNYDER, MICHAEL P, PHD PROFESSOR AND CHAIR OF GENETICS DEPARTMENT OF GENETICS STANFORD UNIVERSITY TOGA, ARTHUR W, PHD PROVOST PROFESSOR LABORATORY OF NEURO IMAGING THE INSTITUTE FOR NEUROIMAGING AND INFORMATICS KECK SCHOOL OF MEDICINE OF USC UNIVERSITY OF SOUTHERN CALIFORNIA VUNJAK-NOVAKOVIC, GORDANA, PHD MIKATI FOUNDATION PROFESSOR OF BIOMEDICAL ENGINEERING DEPARTMENT OF BIOMEDICAL ENGINEERING AND MEDICAL SCIENCE COLUMBIA UNIVERSITY YASZEMSKI, MICHAEL J, MD, PHD PROFESSOR OF ORTHOPEDICS DEPARTMENTS OF ORTHOPEDIC SURGERY AND BIOMEDICAL ENGINEERING MAYO CLINIC COLLEGE OF MEDICINE, ROCHESTER CSR AC ADVISORS: BRINTON, ROBERTA DIAZ, PHD R. PETE VANDERVEEN CHAIR IN THERAPEUTIC DISCOVERY AND DEVELOPMENT BIOMEDICAL ENGINEERING AND NEUROLOGY NORRIS FOUNDATION LABORATORY FOR NEUROSCIENCE RESEARCH UNIVERSITY OF SOUTHERN CALIFORNIA MAYO, STEPHEN L, PHD BREN PROFESSOR OF BIOLOGY AND CHEMISTRY WILLIAM K. BOWES JR. LEADERSHIP CHAIR DIVISION OF BIOLOGY AND BIOLOGICAL ENGINEERING CALIFORNIA INSTITUTE OF TECHNOLOGY CHAIR: GRAINGER, DAVID W, PHD UNIVERSITY DISTINGUISHED PROFESSOR DEPARTMENT OF PHARMACEUTICS AND PHARMACEUTICAL CHEMISTRY UNIVERSITY OF UTAH MEMBERS: ABECASIS, GONCALO, DPHIL FELIX E. MOORE COLLEGIATE PROFESSOR OF BIOSTATISTICS DEPARTMENT OF BIOSTATISTICS UNIVERSITY OF MICHIGAN ALLBRITTON, NANCY L, MD, PHD DEBRECZENY DISTINGUISHED PROFESSOR DEPARTMENT OF BIOMEDICAL ENGINEERING UNIVERSITY OF NORTH CAROLINA CHUTE, CHRISTOPHER G, MD, DRPH BLOOMBERG DISTINGUISHED PROFESSOR OF HEALTH INFORMATICS DIVISION OF GENERAL INTERNAL MEDICINE JOHNS HOPKINS UNIVERSITY GORE, JOHN C PHD, PHD HERTHA RAMSEY CRESS UNIVERSITY PROFESSOR VANDERBILT UNIVERSITY MEDICAL CENTER INSTITUTE OF IMAGING SCIENCE GRILL, WARREN M. PHD, PHD ADDY PROFESSOR OF BIOMEDICAL ENGINEERING DEPARTMENT OF BIOMEDICAL ENGINEERING DUKE UNIVERSITY HYLTON-WATSON, NOLA M., PHD PROFESSOR OF RADIOLOGY DEPARTMENT OF RADIOLOGYAND BIOMEDICAL IMAGING UNIVERSITY OF CALIFORNIA, SAN FRANCISCO LATHROP, RICHARD H, PHD PROFESSOR COMPUTER SCIENCE DEPARTMENT DONALD BREN SCHOOL OF INFORMATION ANDCOMPUTER SCIENCE UNIVERSITY OF CALIFORNIA, IRVINE External Working Group