1. Demystifying the Federal Grant Review Process for NIH Career Development Awards The Ks

2. The Panel Diane Martin, Univ. of Washington Diane Martin, Univ. of Washington Frances Chesley, AHRQ Frances Chesley, AHRQ Lester Gorelic, NIH Lester Gorelic, NIH Willard Manning, University of Chicago Willard Manning, University of Chicago

3. Overview Writing a successful K proposal for young investigator (D) Writing a successful K proposal for young investigator (D) Review criteria and reviewer expectations (D) Review criteria and reviewer expectations (D) Responding to the Pink Sheet (W) Responding to the Pink Sheet (W) Views from AHRQ & E-Submission (F) Views from AHRQ & E-Submission (F) Views from NCI, differences across NIH (L) Views from NCI, differences across NIH (L) Questions and Answers Questions and Answers

4. Focus for Panel Assist transition from clinical or research doctoral work to become independent researcher Assist transition from clinical or research doctoral work to become independent researcher Varies by clinical versus research doctoral background Varies by clinical versus research doctoral background Involves skill acquisition and Involves skill acquisition and research development

5. Focus (contd) Preparatory to submitting an R01 at end of the grant Preparatory to submitting an R01 at end of the grant Provides organizing principle for your Ks proposed skill acquisition and research. Career Development Awards Career Development Awards – Mentored Clinician Scientist – Mentored Research Scientist – Independent Scientist More details on range of alternatives available to be discussed by Drs. Chesley and Gorelic.

7. Writing a Successful K Application Diane Martin University of Washington Academy Health ARM 08

8. Select an Important Topic Consider important problems in healthcare Consider important problems in healthcare Gap in knowledge/research Gap in knowledge/research Rationale for your project Rationale for your project Begin by writing 2-3 page synopsis of specific aims, significance, and overview of methods Begin by writing 2-3 page synopsis of specific aims, significance, and overview of methods

9. K with Legs Sequence of studies contributing answers to a topic Sequence of studies contributing answers to a topic Eventually leading to an RO1 Eventually leading to an RO1

10. Plan 6 Months in Advance Consider type of K and length of funding Consider type of K and length of funding Consider appropriate funding branch and study section Consider appropriate funding branch and study section Understand funding mechanism (KOx); read K solicitation carefully Understand funding mechanism (KOx); read K solicitation carefully Construct application submission calendar, working back from due date Construct application submission calendar, working back from due date Nice if conduct pilot, obtain dataset Nice if conduct pilot, obtain dataset

11. Obtain Information Innovation and non-duplication Innovation and non-duplication – NIH Guide for grants and contracts – CRISP (Computer retrieval of information on Scientific Projects) – HSRProj – AcademyHealth database of HSR projects Obtain copies of successful Ks, funded by target branch Obtain copies of successful Ks, funded by target branch Know study section members research Know study section members research

12. Discuss with Many People Faculty at home and in other institutions Faculty at home and in other institutions Federal officials in target branch Federal officials in target branch Successful K awardees Successful K awardees Two external reviewers for later draft Two external reviewers for later draft Home institution grants administrators (VIP) Home institution grants administrators (VIP) – Budget, budget justification, IRB, environment – Internal procedures and deadlines – Add to application submission calendar

13. Planning a Mentored K Conduct personal SWOT analysis Conduct personal SWOT analysis Coherent, integrated application Coherent, integrated application Tie career development plan to research plan Tie career development plan to research plan Include strong, specific roles for mentors Include strong, specific roles for mentors The above lead to increased knowledge & skills to be an independent researcher The above lead to increased knowledge & skills to be an independent researcher

14. Develop a Team: Mentor(s), Collaborators, Consultants Primary mentor is key; should not be someone you have worked with extensively in the past Primary mentor is key; should not be someone you have worked with extensively in the past Identify expertise of each person Identify expertise of each person Specify time each will give you Specify time each will give you Meet as a group to discuss K, review drafts Meet as a group to discuss K, review drafts Ask for letters of support early Ask for letters of support early

15. Read K Solicitation Read instructions Read instructions Follow page limits, font, use of appendices Follow page limits, font, use of appendices Create K timeline that incorporates career development plan & research plan Create K timeline that incorporates career development plan & research plan Address each review criterion in your application Address each review criterion in your application Make it easy for reviewers to understand; give big picture, then explain details Make it easy for reviewers to understand; give big picture, then explain details

16. Review Criteria for Ks Candidate Candidate Career development plan Career development plan Research plan Research plan (Mentor, co-mentor) (Mentor, co-mentor) Environment and instl commitment Environment and instl commitment Budget Budget Human subjects protection Human subjects protection Inclusion of women and minority subjects Inclusion of women and minority subjects (Inclusion of AHRQ priority pops.) (Inclusion of AHRQ priority pops.)

17. Candidate Describe your background in research, formal coursework and experiences Describe your background in research, formal coursework and experiences Persuade the reviewers you have potential to be an independent productive researcher & how K will help Persuade the reviewers you have potential to be an independent productive researcher & how K will help Tell a story, let your passion show Tell a story, let your passion show Chance to give big picture, describe integration Chance to give big picture, describe integration

18. Career Development Plan Applies to various types of Ks Applies to various types of Ks Coherent rationale for your plan Coherent rationale for your plan Provide details of past experience and coursework subject matter & level Provide details of past experience and coursework subject matter & level Gap and what you need to know to have a successful research project and career Gap and what you need to know to have a successful research project and career Propose a specific career development plan explaining depth of training and how it will contribute to your research Propose a specific career development plan explaining depth of training and how it will contribute to your research

19. Research Plan Discuss how research will build your skills & knowledge, and aid you in becoming an independent researcher Discuss how research will build your skills & knowledge, and aid you in becoming an independent researcher The research plan should be well integrated across all sections The research plan should be well integrated across all sections Use parallel construction throughout application: aim 1, aim 2, aim 3 Use parallel construction throughout application: aim 1, aim 2, aim 3

20. Specific Aims Most important part of application Most important part of application Narrow the topic Narrow the topic Aims testable, stated clearly, in unambiguous language Aims testable, stated clearly, in unambiguous language Aims integrated across hypotheses, methods, expected results and interpretation of alternative findings Aims integrated across hypotheses, methods, expected results and interpretation of alternative findings

21. Background Synthesize and integrate the previous research in this area Synthesize and integrate the previous research in this area Briefly summarize major findings, strengths, weaknesses Briefly summarize major findings, strengths, weaknesses Identify gap your research will fill Identify gap your research will fill Discuss how your research will build on past strengths and overcome limitations Discuss how your research will build on past strengths and overcome limitations Describe the conceptual framework you will use Describe the conceptual framework you will use

22. Significance Why is the problem important?? Why is the problem important?? How will your results advance scientific knowledge? How will your results advance scientific knowledge? How will your study contribute to improved methods? How will your study contribute to improved methods? How will your results likely be used to change practice and policy? How will your results likely be used to change practice and policy?

23. Methods In my experience, shift over time to more detailed research plan In my experience, shift over time to more detailed research plan Specify methods for each aim Specify methods for each aim Research study design and target population/sample Research study design and target population/sample Quantitative, qualitative, mixed methods Quantitative, qualitative, mixed methods Describe interventions, comparisons Describe interventions, comparisons Clearly define measures, choice of variables Clearly define measures, choice of variables

24. Analysis Plan Move from univariate, to bivariate and then multivariable analyses (mock tables) Move from univariate, to bivariate and then multivariable analyses (mock tables) Include an analytic model that describes in words the dependent variable, major independent variables, confounders, and interaction terms Include an analytic model that describes in words the dependent variable, major independent variables, confounders, and interaction terms A well developed plan includes the statistical techniques to be used, the assumptions necessary, and the likely way the results will be interpreted A well developed plan includes the statistical techniques to be used, the assumptions necessary, and the likely way the results will be interpreted

25. General Methods Are framework, design, methods and analysis sufficiently developed? Are framework, design, methods and analysis sufficiently developed? Are they integrated and appropriate to project aims? Are they integrated and appropriate to project aims? Point out any novel concepts and methods Point out any novel concepts and methods Discuss generalizability of results Discuss generalizability of results Acknowledge potential problems & weaknesses and consider alternative tactics Acknowledge potential problems & weaknesses and consider alternative tactics

26. Writing and Editing Clear concise writing: keep related ideas together, shorten long sentences, eliminate redundancy Clear concise writing: keep related ideas together, shorten long sentences, eliminate redundancy Dont assume reviewers will know jargon, methods Dont assume reviewers will know jargon, methods Allow time for administrative parts of appl. Allow time for administrative parts of appl. Allow critique by 2 external reviewers Allow critique by 2 external reviewers Allow time for thorough editing Allow time for thorough editing

27. Other Essentials Environment and institutional commitment Environment and institutional commitment Budget Budget Human subjects protection Human subjects protection Inclusion of women and minority subjects Inclusion of women and minority subjects Inclusion of AHRQ priority pops. Inclusion of AHRQ priority pops. Know the study section review process Know the study section review process

28. Most Common Problems Career development plan does not include sufficient detail, not high level Career development plan does not include sufficient detail, not high level Career development plan is not related to research plan Career development plan is not related to research plan Role and interaction with mentors is not specified Role and interaction with mentors is not specified Aims are not integrated across the research plan Aims are not integrated across the research plan Methods are underdeveloped Methods are underdeveloped

30. Responding to Study Section's Concerns in the Pink Sheets Willard Manning Harris School of Public Policy Studies University of Chicago

31. Responding to Study Section Expect making a second submission of your proposal. Expect making a second submission of your proposal. Very few proposals funded first time unless one-shot announcement. Very few proposals funded first time unless one-shot announcement. Prepare for a critical review. Language will be: Prepare for a critical review. Language will be: – Frank, and – Not overly enthusiastic

32. Facts about Review and Reviewers Reviewers may not be from same discipline or specialty. Reviewers may not be from same discipline or specialty. Check the section roster, then PubMed/Google Check the section roster, then PubMed/Google Many reviewers will have trouble with jargon Many reviewers will have trouble with jargon – Your technical terminology may be unintelligible jargon to me unless explained "Collective efficacy" means what? "Collective efficacy" means what? "Diff-n-diff" means what? "Diff-n-diff" means what? HSR is multidisciplinary and reviews reflect this! ! ! HSR is multidisciplinary and reviews reflect this! ! ! Unless the study section has many from your discipline, you must talk to a wider audience Unless the study section has many from your discipline, you must talk to a wider audience

33. Facts about Review (contd) Remember how reviews are done: – Reviewers doing the review "cold." – Reviewers work under pressure of short deadline while still teaching, seeing patients, etc. – Reviewers do not have benefit of lengthy discussions with research team. – Reviewers do not have time to read the proposal over and over again.

34. Facts about Review (contd) Basic Reviewing Principles Burden of proof of idea and approach is on the proposer. Burden of proof of idea and approach is on the proposer. If they cannot find what they need easily, they often will treat it as missing or inadequately described. If they cannot find what they need easily, they often will treat it as missing or inadequately described. But remember reviewers are: But remember reviewers are: – Looking for problems. – Trying to help, esp. if they see some merit in proposal.

35. Initial Response to Pink Sheets Sulking is normal. Sulking is normal. Do not take it personally. Do not take it personally. Get advice from agency staff ASAP. Get advice from agency staff ASAP. GET SENIOR MENTOR/COLLEAGUES HELP TO INTERPRET COMMENTS AND FRAME RESPONSE. GET SENIOR MENTOR/COLLEAGUES HELP TO INTERPRET COMMENTS AND FRAME RESPONSE. Reserve a "cold reviewer" to react to draft resubmission Reserve a "cold reviewer" to react to draft resubmission – Preferably with study section or area experience Plan to resubmit unless "fatally flawed" Plan to resubmit unless "fatally flawed"

36. Responding to "Pink" Sheets Leave plenty of time to Leave plenty of time to – Overhaul in response to study section and agency staff comments – Solicit reaction to revised submission from cold reviewer, preferably with study section or area experience Respond accordingly to internal cold reviewers comments. Respond accordingly to internal cold reviewers comments. Do not rush to resubmit (e.g., by July 1st after receiving pink sheets on June 6th). Do not rush to resubmit (e.g., by July 1st after receiving pink sheets on June 6th).

37. Revising the Proposal Outrage is OK for a personal reaction but never in a response! Outrage is OK for a personal reaction but never in a response! Take the feedback seriously as indicative of Take the feedback seriously as indicative of – Gaps in exposition or logic Underdeveloped educational plan. Underdeveloped educational plan. Too sketchy on some research aspect. Too sketchy on some research aspect. – Overly terse in key areas. – Organizational issues. Identify common themes across reviewers. Identify common themes across reviewers. Respond to all concerns in "Response," as well as text. Respond to all concerns in "Response," as well as text. Thank the reviewers for their valuable comments. Thank the reviewers for their valuable comments. Apologize for inadequacy of … Apologize for inadequacy of …

38. Revising the Proposal (cont'd) Even if you are "right," clean up or expand the exposition to make the logic more transparent. Even if you are "right," clean up or expand the exposition to make the logic more transparent. Revise the whole proposal if needed. Revise the whole proposal if needed. Ask co-researchers, mentors / senior colleagues, and a cold reviewer for reaction. Ask co-researchers, mentors / senior colleagues, and a cold reviewer for reaction. Revise again Revise again – For substance. – For ease of reviewers to evaluate.

39. Common problems Ks are not about – 5 years of support – 75 percent buyout – $$$ – Those matter to your Dept. Chairman, but not to AHRQ or NIH. but not to AHRQ or NIH.

40. It Is About Mentored Mentored Clinical or Research Scientist Clinical or Research Scientist Development Development

41. It Is About (contd) It does require real mentoring. It does require real mentoring. It is about career development for researchers: It is about career development for researchers: – Not just about more education. – Not just about doing preliminary studies. – Must have both! Dont confuse Ks with series of R03s Dont confuse Ks with series of R03s

42. Disconnected Mentor Mentors letter not closely tied to content of proposal. Mentors letter not closely tied to content of proposal. Mentors letter written by proposer and it looks like it. Mentors letter written by proposer and it looks like it. Mentor approached with proposal with only week left before due date. Mentor approached with proposal with only week left before due date. – Little impact on proposal. – Worse if proposal is naïve.

43. Distant Mentor Always very hard to sell. Always very hard to sell. Study section distrusts supposed level of commitment by mentor. Study section distrusts supposed level of commitment by mentor. Esp. if plans for linkage, visiting vague. Esp. if plans for linkage, visiting vague.

44. Who is in charge? PI + mentor must be primary PI + mentor must be primary Instead, proposal has: Instead, proposal has: – Too many other mentors – No strong primary mentor – Nobody appears to have oversight responsibility

45. Career Development Plan Weak Lacked compelling case for a K, instead of series of R03s, R21s Lacked compelling case for a K, instead of series of R03s, R21s Did not explain why additional training and mentored research work necessary for successful R01 and subsequent career Did not explain why additional training and mentored research work necessary for successful R01 and subsequent career Educational elements too vague Educational elements too vague – Visiting Prof. Jones T times per year, where T is small – Will to him/her at annual meeting Lack of specificity. Needed concrete details on courses and experiences necessary to conduct future work. Lack of specificity. Needed concrete details on courses and experiences necessary to conduct future work. Not clear depth of training or courses. Needs to be at research training level, preferably at PhD or MS level, not MPH level. Not clear depth of training or courses. Needs to be at research training level, preferably at PhD or MS level, not MPH level. Formal course work preferred Formal course work preferred – if a good match for needs – if level appropriate

47. Demystifying the Federal Grant Process: K Awards at AHRQ Francis D. Chesley, Jr., M.D. Director, Office of Extramural Research, Education and Priority Populations June 8, 2008

48. AHRQs Mission Improve the quality, safety, efficiency and effectiveness of health care for all Americans

49. Training Opportunities Pre and Postdoctoral Training Pre and Postdoctoral Training – NRSA Institutional Training Programs (T32) – NRSA Predoctoral Fellowships for Underrepresented Minority Students (F31) – NRSA Postdoctoral Fellowships (F32) – Dissertation Grants (R36) Career Development Awards Career Development Awards – Mentored Clinical Scientist Awards (K08) – Mentored Research Scientist Development Award (K01) – Independent Scientist Awards (K02)

50. Mentored Research Scientist Awards (K01) Audience – research trained doctorates (e.g., Ph.D., Sc.D., Dr.P.H.) who require mentoring and have potential to develop into independent investigators Audience – research trained doctorates (e.g., Ph.D., Sc.D., Dr.P.H.) who require mentoring and have potential to develop into independent investigators Duration -- 3 to 5 years, nonrenewable Duration -- 3 to 5 years, nonrenewable Level of Support -- $90,000 annually, plus fringe benefits and research development support up to $25,000 Level of Support -- $90,000 annually, plus fringe benefits and research development support up to $25,000

51. Mentored Clinical Scientist Awards (K08) Audience -- clinical doctorates ( including those in patient-oriented research ) who require mentoring and have potential to develop into independent investigators Audience -- clinical doctorates ( including those in patient-oriented research ) who require mentoring and have potential to develop into independent investigators Duration -- 3 to 5 years, nonrenewable Duration -- 3 to 5 years, nonrenewable Level of Support -- $90,000 annually, plus fringe benefits and research development support up to $25,000 Level of Support -- $90,000 annually, plus fringe benefits and research development support up to $25,000

52. Independent Investigator Awards (K02) Audience -- promising new clinical and nonclinical investigators who are out of training 5 years or less, with demonstrated need of intensive research focus Audience -- promising new clinical and nonclinical investigators who are out of training 5 years or less, with demonstrated need of intensive research focus Duration -- 3 to 5 years, nonrenewable Duration -- 3 to 5 years, nonrenewable Level of Support -- $90,000 annually, plus fringe benefits, travel, justified educational expenses Level of Support -- $90,000 annually, plus fringe benefits, travel, justified educational expenses

53. 2008 Priorities for K Award Funding Prevention and Healthcare Management of Complex Patients Prevention and Healthcare Management of Complex Patients – http://grants.nih.gov/grants/guide/notice-files/NOT- HS-08-004.html http://grants.nih.gov/grants/guide/notice-files/NOT- HS-08-004.html http://grants.nih.gov/grants/guide/notice-files/NOT- HS-08-004.html Comparative Effectiveness Research Comparative Effectiveness Research – http://grants.nih.gov/grants/guide/notice-files/NOT- HS-08-003.html http://grants.nih.gov/grants/guide/notice-files/NOT- HS-08-003.html http://grants.nih.gov/grants/guide/notice-files/NOT- HS-08-003.html Diagnostic Errors in Ambulatory Care Settings Diagnostic Errors in Ambulatory Care Settings – http://grants.nih.gov/grants/guide/notice-files/NOT- HS-08-002.html http://grants.nih.gov/grants/guide/notice-files/NOT- HS-08-002.html http://grants.nih.gov/grants/guide/notice-files/NOT- HS-08-002.html

54. Important Considerations Research and program priorities matter Research and program priorities matter Application process must be understood Application process must be understood – Electronic Grant Application has arrived! Concept papers are an important option Concept papers are an important option Peer review is a step away from funding Peer review is a step away from funding Avoid common pitfalls Avoid common pitfalls – Mentorship!

55. Applicant Responsibilities Know PHS Form 398 and 424 R&R Know PHS Form 398 and 424 R&R Know the Funding Agency and Staff Know the Funding Agency and Staff Understand Agency Budget Understand Agency Budget Know Agency Research Priorities Know Agency Research Priorities Know the Grant Mechanisms Know the Grant Mechanisms Know the Grant Process and Key Changes Know the Grant Process and Key Changes

56. AHRQ Transitions to Electronic Grant Application Submission AHRQ transitioning to electronic grant submission through Grants.gov AHRQ transitioning to electronic grant submission through Grants.gov – Grants.gov - Web portal that serves as the single access point for all Federal grant programs. – Grants.gov provides the interface for 26 agencies to announce $350 billion in annual grant awards and for all grant applicants to find and submit applications to those funding announcements. www.grants.gov www.grants.gov Transition for K awards in February 2009 Transition for K awards in February 2009

57. Getting Started One time registrations for Grants.gov (http://grants.gov) and eRA Commons (era.nih.gov/commons) systems must be completed before application submission. One time registrations for Grants.gov (http://grants.gov) and eRA Commons (era.nih.gov/commons) systems must be completed before application submission.http://grants.gov (era.nih.gov/commons)http://grants.gov (era.nih.gov/commons) For up to date general information on electronic submission, the SF 424 (R&R), and Grants.gov, visit the AHRQ Electronic Submission of Grant Applications Web Site: http://www.ahrq.gov/path/egrants.htm For up to date general information on electronic submission, the SF 424 (R&R), and Grants.gov, visit the AHRQ Electronic Submission of Grant Applications Web Site: http://www.ahrq.gov/path/egrants.htm http://www.ahrq.gov/path/egrants.htm

58. Contact Information AHRQ WEBSITE www.ahrq.gov Francis D. Chesley, Jr., M.D. (301) 427-1521 Francis.Chesley@ahrq.hhs.gov

59. Questions ?

61. LESTER GORELIC SLIDES HERE LESTER GORELIC SLIDES HERE

63. Questions & Answers Panel

64. Resources Information on different K grant mechanisms Information on different K grant mechanisms http://grants.nih.gov/training/careerdevelopmentawards.htm Instructions on how to prepare your application Instructions on how to prepare your application http://grants.nih.gov/grants/funding/phs398/section_1.html Contact information: – Diane: dianemar@u.washington.edu dianemar@u.washington.edu – Francis: Francis.Chesley@ahrq.hhs.gov Francis.Chesley@ahrq.hhs.gov – Les:gorelicl@mail.nih.gov gorelicl@mail.nih.gov – Will:w-manning@uchicago.edu w-manning@uchicago.edu