Moving from Development to Efficacy & Intervention Fidelity Topics National Center for Special Education Research Grantee Meeting: June 28, 2010

Objectives Development Fully develop intervention Gather data on the feasibility of implementing the intervention Provide pilot data on the promise of the intervention for generating desired outcomes Efficacy Estimate the strength or potency of the impact of the intervention Inform the degree to which the intervention can be feasibly or practically implemented Assess implementation fidelity

Grant application components Significance Research Plan Personnel Resources

Significance Development Context for the Intervention Detailed Description of the Intervention to be developed Theory of change Theoretical and empirical support Practical Importance Rationale for why this grant should be funded Efficacy Detailed description of developed intervention Theory of Change Justification for evaluating the intervention - Practical Importance - Rationale for grant funding

Research Plan Development 1.Sample and setting 2.Iterative development process 3.Feasibility of implementation 4.Pilot study 5.Measures Efficacy 1.Research questions 2.Sample and setting 3.Detailed research design 4.Power Analysis 5.Measures 6.Fidelity of Implementation 7.Description of Comparison Group 8.Mediating and Moderation Variables 9.Data Analysis

Efficacy: Research Questions Clear and concise Stem from Significance section Relate to design and methodology Map onto analysis section (e.g., if subgroups are of interest)

Efficacy: Sample Issues Show that sample size is large enough to rule out that a finding of no impact is due to lack of power Provide support from education settings for what you are planning to do (e.g., RCTs) Provide sufficient assurance that you can recruit the necessary number of participants

Efficacy: Rigorous Research Design RCT favored - Unit of randomization and justification - Procedures for assignment Strong quasi-experiment - why RCT not possible - How it reduces or models selection bias - RDD, IV, strong matching - Discuss threats to internal validity that remain and how this will effect conclusions drawn

Efficacy: Power Analysis Provide and justify all values used in calculating power Provide method used to calculate power –Address clustering/nesting of data Reviewers should be able to reproduce your results

Efficacy: Measures Assess feasibility Assess impact - include measures of broad educational interest (aligned and general) - sensitive to change in performance caused by intervention - assess proximal and distal outcomes - include mediators and moderators Assess fidelity

Efficacy: Fidelity of Implementation Description of the intervention should include how it will be implemented Documenting and measuring implementation –Discuss how capture core components –Reliability and validity of measures; coding of qualitative measures –How fidelity study will be implemented Mediators and moderators of fidelity Use of fidelity data in impact analysis

Efficacy: Describing the Comparison Group Determine if comparison group received similar treatment –Overall approach and/or key components A fidelity study for the comparison group –Measures, reliability & validity –Often “business as usual” (define) or alternative treatment Explain impacts Address contamination if this is a possibility

Efficacy: Mediating and Moderating Variables Described in your theory of change Discuss how to measure –Describe method and process to collect and code Measure in both treatment and comparison group

Efficacy: Detailed Data Analysis Thorough description of data analysis –Main analysis, –Sub-analyses including mediators and moderators (fidelity levels, subgroups) –Show models –Address common statistical issues: clustering of units, missing data, attrition bias –Qualitative data analyses – often not described Show that analysis fits with design and addresses research questions How conclusions drawn from results

Personnel Development Developer plays primary role Methodological and statistical expertise represented Expertise needed in working with schools (or other study settings) and with population Content domain experts Efficacy Developer role is prominent Methodological and statistical expertise prominent Expertise needed in working with schools (or other study settings) and with population Content domain experts

Resources In both Development and Efficacy grant applications, there need to be resources sufficient to support research activities, and provide access to settings in which research will be conducted