1 Writing Proposals, Getting Reviews, and Persevering Ming Tai-Seale, PhD, MPH School of Rural Public Health

2 Acknowledgement Enola Proctor, PhD Enola Proctor, PhD Washington Univ at S.L. Washington Univ at S.L. Kenneth Wells, MD Kenneth Wells, MD UCLA UCLA

3 The Agenda Life of a proposal Life of a proposal Scientific review: who, where, how Scientific review: who, where, how Critical areas for improvement Critical areas for improvement Summary statement Summary statement Finding your agent: working with program staff Finding your agent: working with program staff

4

5 3 Overall Strategies Think First, Second, and Third Think First, Second, and Third Get Feedback at all Stages (3 times) Get Feedback at all Stages (3 times) Develop and Follow Timeline for Submit Develop and Follow Timeline for Submit Plan on 3 Major Rewrites Plan on 3 Major Rewrites Talk to Agency Staff Early, 3 Times Talk to Agency Staff Early, 3 Times After a Good Idea and Opportunity, It’s Methods, Methods, Methods After a Good Idea and Opportunity, It’s Methods, Methods, Methods -From Ken Wells

6 Make a list of 5 project ideas that you would want to work on for years Make a list of 5 project ideas that you would want to work on for years Prioritize and sketch out 1-3 ideas Prioritize and sketch out 1-3 ideas Get feedback from senior researchers, clinicians, or patients Get feedback from senior researchers, clinicians, or patients and community members and community members Finding Ideas

7 Have a Conceptual Framework

8 Make a list of 5 project ideas that you would want to work on for years Make a list of 5 project ideas that you would want to work on for years Prioritize and sketch out 1-3 ideas Prioritize and sketch out 1-3 ideas Get feedback from senior researchers, clinicians, or patients and community members Get feedback from senior researchers, clinicians, or patients and community members Having Ideas

9 Shaping the Idea Based on literature, agency priorities, identify next steps suitable to your stage of development and institution capabilities Based on literature, agency priorities, identify next steps suitable to your stage of development and institution capabilities Identify data sources needed Identify data sources needed Identify special opportunities Identify special opportunities SENIOR CONSULT SENIOR CONSULT

10 Clarify settings and subjects Clarify settings and subjects Develop Partnership – What model of collaboration? Develop Partnership – What model of collaboration? Consider diversity goals Consider diversity goals CONSULT with a statistician CONSULT with a statistician Develop the Design –Hypotheses –Sampling –Design

11 Developing the Design (cont.) Develop rough ballpark for budget Develop rough ballpark for budget SENIOR HELP NEEDED SENIOR HELP NEEDED Discuss concept with funding agency, based on aim, opportunity, Discuss concept with funding agency, based on aim, opportunity, design, and likely budget design, and likely budget

12 Develop Conceptual Framework Develop Conceptual Framework Identify Main Variables Identify Main Variables Draft Interventions, if applicable Draft Interventions, if applicable Develop Pilot Data Develop Pilot Data GET SENIOR CONSULT AND BEG FOR $ GET SENIOR CONSULT AND BEG FOR $ Write Aims, Background, Design (Pretend you’re almost done!) Write Aims, Background, Design (Pretend you’re almost done!) First Draft

13 First Draft Statistician Consult to help outline main analysis and develop Power Calculations to determine: Statistician Consult to help outline main analysis and develop Power Calculations to determine: Is the Study Affordable? Is the Study Affordable? If Yes, PROCEED TO PARK PLACE If Yes, PROCEED TO PARK PLACE If Not, Go Back to Start (JAIL) If Not, Go Back to Start (JAIL)

14 Second Draft Develop Operations plans Develop Operations plans team organization, data collection, timeline team organization, data collection, timeline Detailed Budget--GET HELP Detailed Budget--GET HELP Statistical Consult: Detailed Analysis Plan Statistical Consult: Detailed Analysis Plan Review assumptions Review assumptions Don’t delegate blindly Don’t delegate blindly They are YOUR hypotheses; They are YOUR hypotheses; Modify design, scope, budget Modify design, scope, budget as needed as needed Human Subjects Section: Human Subjects Section: Consult with IRB, Mentors Consult with IRB, Mentors Plop revised draft together Plop revised draft together

15 Third Draft Meet all agency requirements Meet all agency requirements Develop budget justification Develop budget justification Highlight “value added,” pilot data, fit of aims, method, and analysis to model Highlight “value added,” pilot data, fit of aims, method, and analysis to model AGENCY & SENIOR REVIEW AGENCY & SENIOR REVIEW

16 It’s Not Over Yet Take the feedback seriously Take the feedback seriously Revise the whole proposal if needed Revise the whole proposal if needed Common problems: Not feasible (budget too large or scope too broad ); No pilot data; Aims not specific; Background has literature but no synthesis, framework, value-added unclear; Methods are under-developed (alternatives not considered; analysis not tied to hypotheses; design flaws: wrong sample for aim, Common problems: Not feasible (budget too large or scope too broad ); No pilot data; Aims not specific; Background has literature but no synthesis, framework, value-added unclear; Methods are under-developed (alternatives not considered; analysis not tied to hypotheses; design flaws: wrong sample for aim, causal inference poor; limitations unaddressed causal inference poor; limitations unaddressed

17

18 Disciplines Disciplines Anthropology Anthropology Biostatistics Biostatistics Economics Economics Epidemiology Epidemiology Health services research Health services research Medicine Medicine Nursing Nursing Organizational Theory Organizational Theory Sociology Sociology Methodological Orientations Quantitative Qualitative Mixed Stages in Their Own Careers Senior Scholars Emergent scientists Who Serve on Study Sections?

19 The Review Source: Enola Proctor Source: Enola Proctor

20 Reviewer charge Assess likelihood that proposed research will have substantial impact on pursuit of NIH research goals: Advance understanding of Biological systems Biological systems Improve control of disease Improve control of disease Enhance health Enhance health

21 Review Criteria Significance Significance Approach Approach Innovation Innovation Investigator Investigator Environment Environment Priority Populations Budget Human Subject Protections Summary Major strengths and weaknesses Recommendation for or against funding

22 Significance Is the dependent variable important to NIH objectives? Does the study extend boundaries of current knowledge? Does the study address funding agency priorities? Bridging Science and Services Surgeon General Surgeon General’s report IOM: quality Agency partnership: “The Road Ahead” Can the findings inform policy or practice?

23 Approach: Aims Is the research question fully specified? Aims are manageable, but reflects long term agenda Incorporates relevant concepts, variables

24 Approach: Methods Setting and context Collaborative arrangements feasible data collection plan (sufficient subjects, reasonable procedures, plan (sufficient subjects, reasonable procedures, good measures) Manageable: Sufficient time for methodology and products? Person power: enough, and the right people Analytic model = fully specified rival explanations controlled through design or measurement of relevant variables Procedures protect data quality Appropriate methods of analysis Sufficient power

25 Methods: Critical Areas for Improvement Design problem Design problem Measurement Measurement Choice of variables Choice of variables Intervention/comparison Intervention/comparison Analysis problem Analysis problem Choice of approach Choice of approach Technique Technique Test Test Theoretical or conceptual model or framework Theoretical or conceptual model or framework Missing, deficient, or erroneous Missing, deficient, or erroneous

26 Innovation Original and innovative aims Novel concepts, methods, approaches Challenges existing thinking or approaches Advances new methods or technologies

27 Investigator Sufficient experience to direct the project Well trained Productive Proposed project builds on background work investigator publications preliminary studies Proposed work will make “next logical next logical” contribution to knowledge Is surrounded by a team who can ensure success Range of substantive and methodological expertise Multidisciplinary Critical areas for improvement Investigator expertise deficient Needs consultants or collaborators

28 Environment Environment contributes to project success Project capitalizes on environment, its people, and resources and resources Collaborations Evidence of organizational support

29 Ks Review Guideline Candidate Candidate Career development plan Career development plan Research plan Research plan Mentor/co-mentor Mentor/co-mentor Environment and institutional commitment Environment and institutional commitment Budget Budget Human subjects Human subjects Women/minorities/children Women/minorities/children SUMMARY SUMMARY major strengths and weaknesses major strengths and weaknesses Recommendation for or against funding Recommendation for or against funding

30 Overall Review All criteria are considered when assigning overall score Application does not have to be equally strong in all categories in all categories Major issue: will proposed research have major scientific impact?

31 Rating Scale in NIH Review 1.0 virtually flawless, with negligible weaknesses 1.5 extremely strong, with a few minor weakness 2.0 very strong, but with moderate weaknesses 2.5 strong, but with some major weaknesses that must be addressed 3.0 fair, neutral balance of strengths and weaknesses 3.5 weak, but with some major strengths

32 Priority Score How is the summary priority score calculated? How is the summary priority score calculated? Group average Group average Average Score times 100 Average Score times 100 Equal weight Equal weight Some are “un-scored” Some are “un-scored” What is the fundable range? What is the fundable range? Study sections can have different norms Study sections can have different norms When in doubt, ask the project officer When in doubt, ask the project officer

33 The Physical Setting

34 Critical Areas for Improvement in Rs Design problem Design problem Measurement Measurement Choice of variables Choice of variables Intervention/comparison Intervention/comparison Analysis problem Analysis problem Choice of approach Choice of approach Technique Technique Test Test

35 Critical Areas for Improvement in Rs Weak justification for study Weak justification for study Background and significance unconvincing Background and significance unconvincing Literature review incomplete Literature review incomplete Investigator expertise deficient Investigator expertise deficient Needs consultants or collaborators Needs consultants or collaborators Theoretical or conceptual model or framework Theoretical or conceptual model or framework Missing, deficient, or erroneous Missing, deficient, or erroneous

36 Summary Statement

37 How to Read the Pink Sheet Expect the language to be Expect the language to be Frank, and Frank, and Not overly enthusiastic Not overly enthusiastic Be emotionally detached, after the initial… Be emotionally detached, after the initial… Talk to an experienced grant-maker Talk to an experienced grant-maker Resubmit unless you see “fatally flawed” Resubmit unless you see “fatally flawed” Do NOT resubmit right away Do NOT resubmit right away Recruit a “cold reviewer” Recruit a “cold reviewer”

38 Take a Vacation …

39 Program Staff Project officer – Your Agent Project officer – Your Agent Read your concept paper and draft Read your concept paper and draft Send it in EARLY!, and Send it in EARLY!, and Often Often Interpret the fundability of your priority score Interpret the fundability of your priority score

40 Resources Instructions on how to prepare your application Instructions on how to prepare your application Follow it VERY closely Follow it VERY closely Panel presentation at AcademyHealth 2006 Panel presentation at AcademyHealth pdf 06.pdf 06.pdf 06.pdf