Securing External Federal Funding Janice F. Almasi, Ph.D. Carol Lee Robertson Endowed Professor of Literacy University of Kentucky

Institute of Education Sciences 2 Grant Competitions per year Janice F. Almasi, Ph.D., University of Kentucky

Current Funding Opportunities 14 Long-term Programs of Research

Be Informed Subscribe to Newsflash at ies.ed.gov/newsflash

IES Research Goals  Goal 1: Identification  Identifying programs and practices associated with better educational outcomes (secondary data analysis)  Goal 2: Development Projects  Developing educational interventions  Goal 3: Efficacy and Replication Projects  Determine if fully-developed interventions are effective  Goal 4: Scale-Up  Goal 5: Measurement Projects Janice F. Almasi, Ph.D., University of Kentucky

Prior to Peer Review Meeting Triage identifies top 25 applications Reviewers read, rate about 8 applications Reviewers check for COIs Each application assigned to at least 2 reviewers Janice F. Almasi, Ph.D., University of Kentucky

Criteria  Significance  Research Plan  Personnel  Resources Janice F. Almasi, Ph.D., University of Kentucky

Review Criterion Ratings (More Weaknesses than Strengths) (Balance of Strengths and Weaknesses) (More Strengths than Weaknesses) Janice F. Almasi, Ph.D., University of Kentucky 1 PoorExcellent

What Reviewers Look For Where Applications Tend to be Weak Janice F. Almasi, Ph.D., University of Kentucky

Significance Goal 1 Theoretical and empirical rationale for study and practical importance of the intervention (e.g., program, practice) that will be examined Goals 2 and 3 Describe (a) the intervention (e.g., features, components) and the logic model for the intervention, (b) theoretical and empirical support for intervention, and (c) practical importance of the intervention Janice F. Almasi, Ph.D., University of Kentucky

Significance in Goals 2 and 3  Context for Proposed Interventions  Provide context for the proposed intervention by including data on, or reviewing research describing, the attributes of typical existing practices.  Identify shortcomings of current practice and how they contribute to the rationale for the proposed intervention.  Provide context for understanding how much of a change the proposed intervention is intended to achieve. Janice F. Almasi, Ph.D., University of Kentucky

Significance in Goals 2 and 3  Intervention, Theory of Change, Empirical/Theoretical Rationale  Clearly describe the intervention  Clearly describe the theory of change for the intervention  How do the features or components of the intervention relate to each other temporally (or operationally), pedagogically, and theoretically (e.g., why A leads to B)?  Provide a strong theoretical and empirical justification for the design and sequencing of the features or components of the intervention. Enables evaluation of:  Relation between the intervention and its theoretical and empirical foundation (e.g., is the proposed intervention a reasonable operationalization of the theory?)  Relation between the intervention and the outcome measures (e.g., do the proposed measures tap the constructs that the intervention is intended to address?) Janice F. Almasi, Ph.D., University of Kentucky Include a Logic Model

Significance in Goals 2 and 3  Practical Importance of Intervention  When the proposed intervention is fully developed will it have the potential to improve student outcomes in educationally meaningful increments, if it were implemented over the course of a semester or school year?  Would the proposed intervention be both affordable for and easily implemented by schools (e.g., not involve major adjustments to normal school schedules)? Janice F. Almasi, Ph.D., University of Kentucky

Research Plan:Goal 2 Janice F. Almasi, Ph.D., University of Kentucky Sample Samples and settings used to assess feasibility of intervention and for pilot data assessing promise of intervention Iterative Development Process Revision Implementation Observation Revision How do you define “operating as intended?” How do you define “operating as intended?” What data will be gathered to determine how intervention is operating? What data will be gathered to determine how intervention is operating? How will the data gathered be used to revise the intervention? How will the data gathered be used to revise the intervention? What criteria will be used to determine if the intervention operates as intended? What criteria will be used to determine if the intervention operates as intended?

Research Plan:Goal 2 Janice F. Almasi, Ph.D., University of Kentucky Feasibility of Implementation Goal is a fully developed intervention Data that addresses feasibility of implementing in small sample of authentic education settings Promise of intervention in terms of outcomes Pilot Study Pilot data on outcome measures progressing in right direction Pilot data demonstrates implementation of intervention is associated with behaviors consistent with theory of change No more than 30% of funds Data should not be a test of efficacy

Research Plan:Goal 2 Janice F. Almasi, Ph.D., University of Kentucky Measures Clearly describe procedures for gathering data to refine and revise the intervention and provide insight into feasibility and usability of proposed intervention What needs to be observed? How will observations be gathered? Clearly describe measures that will be used (and reliability and validity if appropriate)

Reearch Plan:Goal 3 Janice F. Almasi, Ph.D., University of Kentucky Research Questions Pose clear, concise hypotheses or research questions Sample Define sample to be selected Define sampling procedures (including justification for inclusion and exclusion) Strategies to be used to reduce attrition

Reearch Plan:Goal 3 Janice F. Almasi, Ph.D., University of Kentucky Research Design Provide detail! How will threats to internal/external validity be addressed? Studies using random assignment are preferred where feasible What is unit of randomization and what procedures will be used to make assignments to conditions? Power What power is needed to detect a reasonably expected and minimally important effect? How was effect size calculated? If clusters are randomly assigned to treatment conditions be sure to include intraclass correlation and anticipated effect size in power analysis

Research Plan:Goal 3 Janice F. Almasi, Ph.D., University of Kentucky Measures Justify appropriateness of measures Are measures of practical interest to educators and not overly aligned with intervention? Include reliability and validity information Fidelity of Implementation How will implementation be documented and measured? How will factors associated with fidelity be identified and assessed? How will fidelity data be incorporated into analyses of impact? How do conditions in the school setting affect fidelity of implentation?

Research Plan:Goal 3 Janice F. Almasi, Ph.D., University of Kentucky Comparison Group How does comparison group compare to intervention on critical features of intervention? Using a “business-as-usual” comparison is acceptable but explain why using it is acceptable How will contamination be avoided? Mediating and Moderating Variables Observational, survey, or qualitative methods are encouraged to help identify factors that may explain the effect or lack of effect of intervention Data Analysis Quantitative: Specify statistical procedures and include formulas where appropriate Qualitative: Specific methods used to index, summarize, and interpret data should be identified Relation between hypotheses, measures, and independent and dependent variables should be clear

Personnel  What role will each individual have in the project?  What qualifications, training, and experience do key personnel possess?  How will qualifications be used on the research?  Are key personnel dedicating sufficient time to competently implement proposed research? Janice F. Almasi, Ph.D., University of Kentucky

Resources  Are the resources adequate to support the proposed activities in terms of:  Facilities  Equipment  Supplies  Institutional Support for Managing/Directing Grants and Supporting Scholarship  Have partners shown support for implementation and support of the project? Janice F. Almasi, Ph.D., University of Kentucky

Scientist Reviewer Critiques  A brief description of the overall application.  Identify each application’s key strengths and weaknesses in each of the evaluation areas and prepare critical, evaluative comments.  Integrated summary of the overall assessment of the application, including the main strengths and weaknesses of the application. Janice F. Almasi, Ph.D., University of Kentucky

Overall Score Janice F. Almasi, Ph.D., University of Kentucky

Peer ReviewMeeting Process Janice F. Almasi, Ph.D., University of Kentucky 3. Notetaker Summarizes the discussion orally and in writing 2. Full Panel Discusses the application, asks questions, and offers additional critique Discusses the budget 1. Assigned Scientist Reviewers Share Overall Scores Reviewer 1 Summarizes the application and its strengths and weaknesses in each Reviewer 2 elaborates on areas of agreement or disagreement

Peer ReviewMeeting Process Janice F. Almasi, Ph.D., University of Kentucky 6. Assigned Scientist Reviewers May edit/revise their original written critiques based on panel discussion 5. Full Panel Privately assigns criteria scores and overall scores 4. Assigned Scientist Reviewers Adjust initial recommended criteria scores and overall scores