1. NIH Peer Review in the Population and Social Sciences Center for Scientific Review National Institutes of Health U.S. Department of Health and Human Services

2. Scientific Review Administrator and Referral Officer Center for Scientific Review National Institutes of Health Charles N. Rafferty, Ph.D. Study Sections   Health Services Organization and Delivery (HSOD) – reviews R01, R21, and R03 applications   Occupational Health and Safety SBIR/STTR Referral IRGs  HOP, RPHB, BST, and DIG

3. Outline  Overview of NIH and Peer Review  Application Receipt and Referral  Initial Peer Review Process, The Study Section  Grantsmanship

4. Premise Understanding the peer review process will help you prepare a successful grant application. Success = Award

5. Dr. Brent Stanfield “ Our work is critical because we know the result of peer review is the primary factor determining which research NIH funds." “ Our work is critical because we know the result of peer review is the primary factor determining which research NIH funds."

6. National Institutes of Health Much of the biomedical research in the United States is supported by the Federal Government, primarily the National Institutes of Health (NIH) Much of the biomedical research in the United States is supported by the Federal Government, primarily the National Institutes of Health (NIH)

7. NIH Extramural Awarding Components  National Cancer Institute (NCI)  National Heart, Lung, and Blood Institute (NHLBI)  National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK)  National Library of Medicine (NLM)  National Institute of Child Health and Human Development (NICHD)  National Institute of Deafness and Other Communication Disorders (NIDCD)  National Institute of Environmental Health Sciences (NIEHS)  National Institute of Neurological Disorders and Stroke (NINDS)  National Institute on Aging (NIA)  National Institute of Allergy and Infectious Diseases (NIAID)  National Institute of Arthritis and Musculoskeletal and Skin Diseases (NIAMS)  National Institute of Dental and Craniofacial Research (NIDCR)  National Eye Institute (NEI)  National Institute of General Medical Sciences (NIGMS)  National Institute of Mental Health (NIMH)  National Institute for Nursing Research (NINR)  National Institute on Alcohol Abuse and Alcoholism (NIAAA)  National Institute on Drug Abuse (NIDA)  National Center for Complementary and Alternative Medicine (NCCAM)  National Center for Research Resources (NCRR)  National Human Genome Research Institute (NHGRI)  National Institute of Biomedical Imaging and Bioengineering (NIBIB)  Fogarty International Center (FIC)  National Center on Minority Health and Health Disparities (NCMHD)

8. A Typical Institute/Center Office of the IC DirectorNationalAdvisoryCouncil Board of ScientificCounselors Extramural ScientificPrograms GrantsContracts Intramural LaboratoryStudiesClinicalStudies

9. NIH Extramural Program GrantPatron (assistance, encouragement) (assistance, encouragement) CooperativePartner Agreement(assistance but substantial program involvement) program involvement) ContractPurchaser (procurement) (procurement)

10. NIH Funding in FY 2004: By Mechanism Total = $27B Over 80% of NIH funds support extramural research.

11. NIH Peer Review Process of evaluation of NIH grant applications for scientific and program merit NIH uses dual review system Scientific Review Group or Study Section Institute/Center Program Review

12. Dual Review System for Grant Applications Second Level of Review Second Level of ReviewCouncil  Assesses Quality of SRG  Review of Grant Applications  Makes Recommendation to  Institute Staff on Funding  Evaluates Program Priorities  and Relevance  Advises on Policy First Level of Review Scientific Review Group (SRG)  Provides Initial Scientific Merit  Review of Grant Applications  Rates Applications and Makes Recommendations for Appropriate Level of Support and Duration of Award

13. Review Process for a Research Grant National Institutes of Health Center for Scientific Review Study Section Institute Advisory Councils and Boards Institute Director School or Other Research Center School or Other Research Center Research Grant Application Research Grant Application Submits Application Allocates Funds Initiates Research Idea Conducts Research Assigns to IC & IRG/ Study Section Reviews for Scientific Merit Evaluates for Relevance Recommends Action Takes final action

14. GROUPS CSR IRGs Study Sections Special Emphasis Panels INSTITUTES INSTITUTES Scientific Review Groups Contract Review Committees APPLICATIONS REVIEWED Research Projects Academic Research Academic Research Enhancement Awards Postdoctoral Fellowships Small Business Innovation Research Shared Instrumentation Program Projects Centers Institutional Training Grants Conference Grants Career Awards Types of Scientific Review Groups Where are Applications Reviewed? Small Grants RFAs Contracts

15. Center for Scientific Review (CSR) Focal Point for Initial Review at NIH Focal Point for Initial Review at NIH Central receipt point for PHS applications Referral to Institutes and to IRGs and Study Sections Review of most research and research training applications..for scientific merit

16. Center for Scientific Review  Referral  Central Receipt Point for most PHS Grant Applications  Institute Assignment (Potential Funding Component)  Assignment to Scientific Review Group in CSR or in an Institute  Scientific Review  More than 200 chartered study sections and regularly recurring special emphasis panels that review: u Research Grant Applications u Postdoctoral Fellowship Application u Academic Research Enhancement Award Applications u Small Business Innovation Research Applications

17. Overall Timeframe from Submission to Award There are three overlapping cycles per year : There are three overlapping cycles per year : JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN JUL ReviewCouncil Cycle 1 Receipt Referral Award Review Council Cycle 3 Receipt Referral Award Review Council Cycle 2 Receipt Award Referral

18. Grant Application Receipt and Assignment

19. Applications Submitted to NIH  Over 60,000 grant applications are submitted to NIH each year, of which 25-30% are funded  Competing grant applications are received for three review cycles per year

20. CSR Receipt and Referral: Central Receipt Point for Applications submitted to the Public Health Service National Institutes of Health Substance Abuse and Mental Health Services Administration CSR Receipt & Referral Centers for Disease Control Food & Drug Agency for Health Care Policy & Research Office of Assistant Secretary for Health

21. Applications are Assigned by Referral Officers: Referral Officers: Professional scientists, most of whom also serve as scientific review administrators of CSR study sections

22. Applications are Assigned to:  Scientific review groups based on: – Specific review guidelines for each scientific review group  Institutes based on: – Overall mission of the Institute – Specific programmatic mandates and interests of the Institute

23. Assignment to Institutes Applications are referred to an Institute or Center as the potential funding component: Applications are referred to an Institute or Center as the potential funding component:  This assignment is based on a match between the research proposed and the overall mission of the Institute or Center  Where applications are appropriate for more than one Institute or Center, multiple assignments are made

24. Sample Application Number Individual Serial Amended Individual Serial Amended Research Number Research Number Grant Grant 1 R01 CA12345601A1 1 R01 CA12345601A1 New National Grant New National Grant Application Cancer Support Institute Year Institute Year

25. Special Referral Issues  Investigators should write a cover letter for their applications!  Referral Officers almost always honor investigator requests for Institute assignments (funding) and CSR study section assignments (review)  NIMH, NIAA, and NIDA review all health services and treatment research applications assigned to them for funding  All other investigator-initiated health services and treatment research applications are reviewed by CSR

26. Initial Review in CSR

27. CSR Study Sections  Each CSR standing study section has 12-24 members who are primarily from academia  CSR standing study sections convene face-to- face meetings  As many as 60-100 applications are reviewed by each study section  Each study section is managed by a Scientific Review Administrator

28. CSR Review Divisions Division of Biologic Basis of Disease Elliot Postow, Ph.D. Immunology IRG (IMM) Calbert Laing, Ph.D. AIDS and Related Research IRG (AARR) Ranga V. Srinivas, Ph.D. Oncological Sciences IRG (ONC) Syed Quadri, Ph.D. Endocrinology, Metabolism, Nutrition and Reproductive Sciences IRG (EMNR) Sooja Kim, Ph.D. Infectious Diseases and Microbiology IRG (IDM) Alex Politis, Ph.D. Division of Physiology and Pathology Michael Martin, Ph.D. Cardiovascular Sciences IRG (CVS) Joyce Gibson, D.Sc. Integrative, Functional and Cognitive Neuroscience IRG (IFCN) Christine Melchior, Ph.D. Renal and Urological Sciences IRG (RUS) Daniel McDonald, Ph.D. Hematology IRG (HEME) Joyce Gibson, D.Sc. Digestive Sciences IRG (DIG) Mushtaq Khan, Ph.D. Division of Clinical and Population-Based Studies Anita Miller Sostek, Ph.D Surgical Sciences, Biomedical Imaging and Bioengineering IRG (SBIB) Eileen Bradley, D. Sc. Health of the Population IRG (HOP) Robert Weller, Ph.D. Risk, Prevention, and Health Behavior IRG (RPHB) Michael Micklin, Ph.D. Brain Disorders and Clinical Neuroscience IRG (BDCN) David Armstrong, Ph.D. Behavioral & Biobehavioral Processes IRG (BBBP) Karen Sirocco, Ph.D Division of Molecular and Cellular Mechanisms Donald Schneider, Ph.D. Biochemical Sciences IRG (BCS) Zakir Bengali, Ph.D. Bioengineering Sciences and Technologies IRG (BST) Sally Amero, Ph.D. Biology of Development and and Aging IRG (BDA) Sherry Dupere, Ph.D. Biophysical and Chemical Sciences IRG (BPC) John Bowers, Ph.D. Cell Development and Function IRG (CDF) Marcia Steinberg, Ph.D. Genetic Sciences IRG (GNS) Richard Panniers, Ph.D. Molecular, Cellular and Developmental Neuroscience IRG (MDCN) Carole Jelsema, Ph.D. Respiratory Sciences IRG (RES) Mushtaq Khan, Ph.D. Musculoskeletal, Oral, and Skin Sciences IRG (MOSS) Daniel McDonald, Ph.D.

29. Biobehavioral and Behavioral Processes IRG (BBBP) Study Sections  BRLE – Biobehavioral Regulation, Learning and Ethology  MESH – Biobehavioral Mechanisms of Emotion, Stress and Health  LCOM – Language and Communication  CP – Cognition and Perception  APDA – Adult Psychopathology and Disorders of Aging  CPDD – Child Psychopathology and Developmental Disabilities  MFSR – Motor Function, Speech, and Rehabilitation  BBBP Small Business Activities

30. Health of the Population IRG (HOP) Study Sections  CLHP – Community-Level Health Promotion  BGES – Behavioral Genetics and Epidemiology  SSPS - Social Sciences and Population Studies  HSOD – Health Services Organization and Delivery  BMRD - Biostatistical Methods and Research  ECD - Epidemiology of Chronic Disease  EPIC – Epidemiology of Cancer  ECDA – Epidemiology of Clinical Disorders and Aging  NSCF – Nursing Science: Children and Families  NSAA – Nursing Science: Adults and Older Adults  HOP Small Business Activities

31. Risk Prevention and Health Behavior IRG (RPHB) Study Sections  PDRP – Psychosocial Development, Risk and Prevention  PRDP – Psychosocial Risk and Disease Prevention  BMIO – Behavioral Medicine Interventions and Outcomes  SPIP - Social Psychology, Personality and Interpersonal Processes  RPHB Small Business Activities

32. Scientific Review Administrator   Designated Federal Official   Performs administrative and technical review of applications   Selects reviewers   Manages study sections   Prepares summary statements   Provides requested information about study section recommendations to Institutes and National Advisory Councils/Boards

33. Selection of Peer Reviewers Non-Doctoral Scientific Community Non-Research Research Capability Active and Productive Researchers

34. Criteria For Selection of Peer Reviewers  Demonstrated Scientific Expertise  Doctoral Degree or Equivalent  Mature Judgment  Work Effectively in a Group Context  Breadth of Perspective  Impartiality  Interest in Serving  Adequate Representation of Women and Minority Scientists

35. Review of Research Grants REVIEW CRITERIA:  Significance  Approach  Innovation  Investigator  Environment _________  Protection of Human Subjects  Inclusion of Women, Minorities, and Children  Animal Welfare/Biohazards  Overall Evaluation & Score Reflects Impact on Field

36. Review Criteria (continued)  Significance: Does the study address an important problem? How will scientific knowledge be advanced? What are the societal benefits?  Approach: Are design and methods well-developed and appropriate? Are problem areas addressed?  Innovation: Are there novel concepts or approaches? Are the aims original and innovative?  Investigator: Is the investigator appropriately trained?  Environment: Does the scientific environment contribute to the probability of success? Are there unique features of the scientific environment?

37. Additional Review Criteria – Amended Applications (Most Mechanisms)  Adequacy of response to the previous review  Degree of overall improvement of the revised application

38. Scientific Review Group or Study Section Actions  Scored, Scientific Merit Rating (priority scores and percentiles)  Unscored (lower half)  Deferral  Not recommended for further consideration

39. Priority Scores/Percentile Rank  For each study section, applications in the upper half are scored from 1.0-3.0, with 1.0 the best score  Individual scores are averaged and multiplied by 100 to give the final priority score Percentile ranking is calculated based on results of current plus past two meetings

40. Summary Statement Once applications are reviewed, the results are documented by the SRA in a summary statement and forwarded to the Institute (and the PI) where a funding decision is made: Once applications are reviewed, the results are documented by the SRA in a summary statement and forwarded to the Institute (and the PI) where a funding decision is made: The summary statement contains: The summary statement contains:  Overall Resume and Summary of Review Discussion  Essentially Unedited Critiques  Priority Score and Percentile Ranking  Budget Recommendations  Administrative Notes

41. What Determines Which Awards Are Made?  Scientific merit  Program considerations  Availability of funds

42. Grantsmanship Steps in preparing a successful grant application

43. There is no grantsmanship that will turn a bad idea into a good one, but…….. There are many ways to disguise a good one. William Raub, Past Deputy Director, NIH

44. The NIH Grant Culture  Bio-Medical Model  Randomized Clinical Trial as gold standard  Evidence based  Underlying Conceptual Model  Emphasis on Outcomes  Need for Measurement

45. Step One – Scoping  Identify possible research projects  Use web-based NIH data-bases and resources  Identify candidate NIH Institutes/Centers  Identify candidate NIH grant initiatives  Program announcement (PA)  Request for applications (RFA)  Investigator initiated application  Review NIH grant application procedures – PHS 398 Instructions

46. Step Two – Make NIH Contacts  Confer with NIH Program Directors  Assess the “fit” to the Institute/Center  Find out what’s new – PAs and RFAs  Decide on mechanism – e.g., RO1, R03, R21  Find collaborators  Identify review issues – Dos and Don’ts  Define product and focus application

47. Types of Grants  R01 – Research Projects  R03 – Small Research Grants  R21 – Exploratory/Developmental Grants  R13 – Conference Grants  R41/R42 – Small Business Technology Transfer Grants Phase I/II  K-Awards – Career Development  F-Awards - Fellowships  P01 – Research Program Projects http://deainfo.nci.nih.gov/flash/awards.htm http://deainfo.nci.nih.gov/flash/awards.htmhttp://deainfo.nci.nih.gov/flash/awards.htm

48. NIH Grant Mechanisms  R01 Traditional investigator-initiated grant < $500K/yr, 3-5 yrs. Need approval if more than $500K for any year of the grant  R03 Small Grant < $100K for 2 yrs  R21 (NCI) Exploratory/Developmental Grant < $275K for 2 yrs  R13 Conference Grants amount dependent on score, timeliness, budget, NIH interest

49. NIH Opportunities for Young Investigators  National Research Service Individual Fellowship (F32)  Mentored Research Scientist Development Award (K01)  Independent Scientist Award (K02)  Mentored Clinical Scientist Development Award (K08)  Small Grant (R03)  Academic Research Enhancement Award (R15)  Exploratory/Developmental Grant (R21)

50. Step 3 Develop Your Idea Step 3 Develop Your Idea  Review literature  Generate preliminary data  Enlist collaborators, include letters of commitment  Review successful grant applications of other colleagues

51. The “Top Ten” List 1.Read and re-read the program announcement 2.Assemble a strong research team 3.Use the strongest study design possible 4.If you have not been on a study section, confer with someone who has 5.Be sure to document the innovations(s) 6.Document strong access to the study population 7.Make sure the writing, organization, & grammar are as tight as possible (write, re-write…read, re-read) 8.Seek reviews from experienced peers before submission 9.Make careful use of the summary statement 10.Persevere and don’t take rejection personally (Source: Ross Brownson 1/13/2004)

52. Step 4 Writing the Application  Clear, concise writing style  Be focused  Don’t rush  Critique, critique, and critique again  Follow up with NIH program directors before and after review

53. Step 5 Preparing the Application  Follow instructions – PHS 398  Never assume that reviewers “know what you mean”  Refer to literature thoroughly  Present a clear rationale for the proposed work  Make sure that the experimental approach is thorough and detailed  Include well-designed tables and figures  Anticipate human subject issues

54. General Design Issues  Will it work?  Supporting preliminary data  Valid Instruments  Pilot data  Reality check – subject burden  Will compliance rate(s) be adequate

55. Methodological Issues  Sampling Methods  Power Calculations  Theoretical-based Intervention  Compliances  Data Acquisition and Management  Participant Training and Monitoring  Data Analysis

56. Human Subjects Issues Four criteria  Risks*  Protections  Benefits to subjects and others  Importance of knowledge  Data Safety and Monitoring Plan for clinical trials  Exemptions applicable  Inclusion plans  Minorities, women, children, *Risks include the possibility of physical, psychological, or social injury resulting from research.

57. More Human Subject Issues  Recruitment and informed consent  Vulnerable populations  Incentives  Informed Consent u Participation u Use of information u Future analysis http://ohrp.osophs.dhhs.gov/humansubjects/guidance/45cfr46.htm

58. Step 6 Submit the application   Include cover letter   Request Institute assignment for funding   Request study section assignment for review   Indicate potential conflicts   Suggest expertise but not reviewers by name   Multiple Institute assignments acceptable   Institute or CSR review predetermined (you can’t choose)   Meet submission deadlines   If late, ask for exception and provide reason u u Weather, health, study section activities u u Exceptions never granted prior to submission

59. Step 7 Monitor Review Process  Contact Scientific Review Administrator for information and to express any concerns  Timing  Institute assignments  Study section assignment  Provide input about needed expertise - Do Not Suggest Reviewers by Name!  Identify possible conflicts of study section reviewers – e.g., professional, personal, financial, institutional  Be mindful that NIH review administrators are typically managing multiple meetings involving about 100 applications per round

60. Step 8 Post Review Followup  Contact Program Director for information and guidance  Discuss outcome of merit peer review  Review summary statement  What the scores mean (Institute ranking)  Strengths and weaknesses  Recommendations for improvement  Discuss Institute program priorities  Likelihood of funding  Next steps

61. Most Common Problems  Lack of new or original ideas  Absence of an acceptable scientific rationale  Lack of experience in the essential methodology  Questionable reasoning in experimental approach  Uncritical approach  Diffuse, superficial, or unfocused research plan  Lack of sufficient methodological detail  Lack of knowledge of published relevant work  Unrealistically large amount of work  Uncertainty concerning future directions

62. Summary

63. Keys To Success  Find collaborators and mentors who are experienced in writing and winning NIH grants  Make contact with NIH scientific staff at appropriate stages of the review/award cycle  Institute/Center Program Administrators  Scientific Review Administrators  Recognize that NIH peer review has a special culture based on standing study sections composed of senior academic researchers with long histories of service and expectations of style, academic rigor, and hypothesis- based research

64. Good Luck! If at first you don’t succeed – Revise and resubmit

65. Contact Information Charles N. Rafferty, Ph.D. Scientific Review Administrator Health of the Population IRG Center for Scientific Review National Institutes of Health 6701 Rockledge Drive, MSC 7770, Room 3172 Bethesda, MD 20892 (20817 for overnight mail) 301-435-3562 301-480-3962 fax raffertc@csr.nih.govhttp://www.csr.nih.gov

66. NIH Information Sources

67. Information on the World Wide Web Selected Sites of Interest  National Institutes of Health (http://www.nih.gov)  Office of Extramural Research (http://www.nih.gov/grants/oer.htm)  Grants Policy (http://www.nih.gov/grants/policy/policy.htm)  Center for Scientific Review (http://www.csr.nih.gov)  Referral and Review (http://www.csr.nih.gov/refrev.htm)  CSR Study Section Rosters (http://www.csr.nih.gov/committees/rosterindex.asp)  Review Group Meeting Dates (http://www. csr.nih.gov/committees/meetings/ssmeet1.asp)  CSR Reorganization News (http://www.csr.nih.gov/review/reorgact.asp)

68. CSR Web Site: http://www.csr.nih.gov  News and Events  Resources for Applicants Study Section Information  Study Section Information  Employment Opportunities  Contact Information

69. END