1. Assessing Intervention Fidelity in RCTs: Models, Methods and Modes of Analysis David S. Cordray & Chris Hulleman Vanderbilt University Presentation for the IES Research Conference Washington, DC June 9, 2009

2. Overview Fidelity and Achieved Relative Strength: –Definitions, distinctions and illustrations –Conceptual foundation for assessing fidelity in RCTs –Achieved relative strength, a special case in RCTs Modes of analysis: –Approaches and challenges Chris Hulleman -- Assessing implementation fidelity and achieved relative strength indices: The single core component case Questions and discussion

3. Distinguishing Implementation Assessment from the Assessment of Implementation Fidelity Two ends on a continuum of intervention implementation/fidelity: A purely descriptive model: –Answering the question “What transpired as the intervention was put in place (implemented). Based on a priori intervention model, with explicit expectations about implementation of program components: –Fidelity is the extent to which the realized intervention (t Tx ) is faithful to the pre-stated intervention model (T Tx ) –Infidelity = T Tx – t Tx Most implementation fidelity assessments involve descriptive and model-based approaches.

4. Dimensions Intervention Fidelity Aside from agreement at the extremes, little consensus on what is meant by the term “intervention fidelity”. Most frequent definitions: –True Fidelity = Adherence or compliance: Program components are delivered/used/received, as prescribed With a stated criteria for success or full adherence The specification of these criteria is relatively rare –Intervention Exposure: Amount of program content, processes, activities delivered/received by all participants (aka, receipt, responsiveness) This notion is most prevalent –Intervention Differentiation: The unique features of the intervention are distinguishable from other programs, including the control condition A unique application within RCTs

5. Linking Intervention Fidelity Assessment to Contemporary Models of Causality Rubin’s Causal Model: –True causal effect of X is (Y i Tx – Y i C ) –RCT methodology is the best approximation to this true effect –In RCTs, the difference between conditions, on average, is the causal effect Fidelity assessment within RCTs entails examining the difference between causal components in the intervention and control conditions. Differencing causal conditions can be characterized as achieved relative strength of the contrast. –Achieved Relative Strength (ARS) = t Tx – t C –ARS is a default index of fidelity

6. Achieved Relative Strength =.15 Infidelity “Infidelity” (85)-(70) = 15 t C t tx T Tx TCTC.45.40.35.30.25.20.15.10.05.00 Treatment Strength Expected Relative Strength = (0.40-0.15) = 0.25 100 90 85 80 75 70 65 60 55 50 Outcome

7. Why is this Important? Statistical Conclusion validity –Unreliability of Treatment Implementation: Variations across participants in the delivery receipt of the causal variable (e.g., treatment). Increases error and reduces the size of the effect; decreases chances of detecting covariation. Resulting in a reduction in statistical power or the need for a larger study….

8. The Effects Structural Infidelity on Power.60.801.0Fidelity

9. Influence of Infidelity on Study-size 1.0.80.60Fidelity

10. If That Isn’t Enough…. Construct Validity: –Which is the cause? (T Tx - T C ) or (t Tx – t C ) Poor implementation: essential elements of the treatment are incompletely implemented. Contamination: The essential elements of the treatment group are found in the control condition (to varying degrees). Pre-existing similarities between T and C on intervention components. External validity – generalization is about (t Tx - t C ) –This difference needs to be known for proper generalization and future specification of the intervention components

11. So what is the cause? …The achieved relative difference in conditions across components Augmentation of Control Infidelity PD= Professional Development Asmt=Formative Assessment Diff Inst= Differentiated Instruction

12. Achieved Relative Strength =.15 Infidelity “Infidelity” t C t tx T TX’ T Tx TCTC.45.40.35.30.25.20.15.10.05.00 Treatment Strength 100 90 85 80 75 70 65 60 55 50 Outcome Intervention Exposure True Fidelity Tx Contamination Augmentation of C Intervention Exposure Positive Infidelity Intervention Differentiation Review: Concepts and Definitions

13. Some Sources and Types of Infidelity If delivery or receipt could be dichotomized (yes or no): –Simple fidelity involves compliers; –Simple infidelity involves “No shows” and cross- overs. Structural flaws in implementing the intervention: –Missing or incomplete resources, processes –External constraints (e.g. snow days) Incomplete delivery of core intervention components –Implementer failures or incomplete delivery

14. A Tutoring Program: Variation in Exposure 4-5 tutoring sessions per week, 25 minutes each, 11weeks Expectations: 44-55 sessions Cycle 1 47.7 16-56 Cycle 2 33.1 12-42 Cycle 3 31.6 16-44 Average Sessions Delivered Range Random Assignment of Students Time 

15. Variation in Exposure: Tutor Effects Individual Tutors Average Number of Tutoring Sessions per Tutor The other fidelity question: How faithful to the tutoring model is each tutor?

16. In Practice…. Identify core components in the intervention group –e.g., via a Model of Change Establish bench marks (if possible) for T TX and T C Measure core components to derive t Tx and t C –e.g., via a “Logic model” based on Model of Change Measurement (deriving indicators) Converted to Achieved Relative Strength and implementation fidelity scales Incorporated into the analysis of effects

17. What do we measure? What are the options? (1) Essential or core components (activities, processes); (2) Necessary, but not unique, activities, processes and structures (supporting the essential components of T); and (3) Ordinary features of the setting (shared with the control group) Focus on 1 and 2.

18. Fidelity Assessment Starts With a Model or Framework for the Intervention From: Gamse et al. 2008

19. Core Reading Components for Local Reading First Programs Use of research-based reading programs, instructional materials, and assessment, as articulated in the LEA/school application Teacher professional development in the use of materials and instructional approaches 1)Teacher use of instructional strategies and content based on five essential components of reading instruction 2) Use of assessments to diagnose student needs and measure progress 3) Classroom organization and supplemental services and materials that support five essential components Design and Implementation of Research-Based Reading Programs After Gamse et al. 2008

20. From Major Components to Indicators… Professional Development Reading Instruction Support for Struggling Readers Assessment Instructional Time Instructional Material Instructional Activities/Strategies Block Actual Time Scheduled block? Reported time Major Components Sub- components Facets Indicators

21. Reading First Implementation: Specifying Components and Operationalization ComponentsSub-componentsFacetsIndicators (I/F) Reading Instruction Instructional Time 2 2 (1) Instructional Materials 4 12 (3) Instructional Activities /Strategies 8 28 (3.5) Support for Struggling Readers (SR) Intervention Services 3 12 (4) Supports for Struggling Readers 2 16 (8) Supports for ELL/SPED 2 5 (2.5) AssessmentSelection/Interpretation 5 12 (2.4) Types of Assessment 3 9 (3) Use by Teachers 1 7 (7) Professional development Improved Reading Instruction 11 67 (6.1) 4 10 41170 (4) Adapted from Moss et al. 2008

22. Reading First Implementation: Some Results ComponentsSub- components Performance LevelsARSI (U3) RFNon-RF Reading Instruction Instructional Time (minutes) 101780.33 (63%) Support79%58%0.50 (69%) Struggling Readers More Tx, Time, Supplemental Service 83%74%0.20 (58%) Professional Development Hours of PD41.517.60.42 (66%) Five reading dimensions 86%62%0.55 (71%) AssessmentGrouping, progress, needs 84%71%0.32 (63%) 0.39 (65%) Adapted from Moss et al. 2008

23. So What Do I Do With All This Data? Start with: –Scale construction, aggregation over facets, sub- components, components Use as: –Descriptive analyses –Explanatory (AKA exploratory) analyses –There are a lot of options In this section we describe a hierarchy of analyses, higher to lower levels of causal inference Caveat: Except for descriptive analyses, most approaches are relative new and not fully tested.

24. Hierarchy of Approaches to Analysis: –ITT (Intent-to-treat) estimates (e.g., ES) plus: an index of true fidelity: –ES=.50 Fidelity = 96% an index of Achieved Relative Strength (ARS). –Hulleman’s initial analysis: ES=0.45, ARS=0.92. –LATE (Local Average Treatment Effect): If treatment receipt/delivery can be meaningfully dichotomized and there is experimentally induced receipt or non-receipt of treatment: – adjust ITT estimate by T and C treatment receipt rates. Simple model can be extended to an Instrumental Variable Analysis (see Bloom’s 2005 book). –ITT retains causal status; LATE can approximate causal statements.

25. More on Fidelity to Outcome Linkages –TOT (Treatment-on-Treated) Simple: ITT estimate adjusted for compliance rate in Tx, no randomization. Two-level linear production function, modeling the effects of implementation factors in Tx and modeling factors affecting C in separate Level 2 equations. Regression-based model, exchanging implementation fidelity scales for treatment exposure variable.

26. Descriptive Analyses Fidelity is often examined in the intervention group, only. –Dose-response relationship –Partition intervention sites into “high” and “low” implementation fidelity: My review of some ATOD prevention studies, the ES HIGH =0.13 to 0.18 ES LOW =0.00 to 0.03

27. Some Challenges Interventions are rarely clear; Measurement involves novel constructs; How should components be weighted? If at all. Fidelity assessment occurs at multiple levels; Fidelity indicators are used in 2nd and 3 rd levels of HLM models, few degrees of freedom; There is uncertainty about the psychometric properties of fidelity indicators; and Functional form of fidelity and outcome measures is not always known. But, despite these challenges, Chris Hulleman has a dandy example…

28. Assessing Implementation Fidelity in the Lab and in Classrooms: The Case of a Motivation Intervention

29. PERCEIVED UTILITY VALUE INTEREST PERFORMANCE MANIPULATED RELEVANCE Model Adapted from: Eccles et al. (1983); Hulleman et al. (2009) The Theory of Change

30. Methods (Hulleman & Cordray, 2009) LaboratoryClassroom SampleN = 107 undergraduatesN = 182 ninth-graders 13 classes 8 teachers 3 high schools TaskMental Multiplication Technique Biology, Physical Science, Physics Treatment manipulationWrite about how the mental math technique is relevant to your life. Pick a topic from science class and write about how it relates to your life. Control manipulationWrite a description of a picture from the learning notebook. Pick a topic from science class and write a summary of what you have learned. Number of manipulations12 – 8 Length of Study1 hour1 semester Dependent VariablePerceived Utility Value

31. Motivational Outcome g = 0.05 (p =.67) ?

32. Fidelity Measurement and Achieved Relative Strength Simple intervention – one core component Intervention fidelity: –Exposure: “quality of participant responsiveness” –Rated on scale from 0 (none) to 3 (high) –2 independent raters, 88% agreement

33. Exposure LaboratoryClassroom CTxC Quality of Responsiveness N%N%N%N% 04710071186963841 1001524444043 2002946001415 30012190000 Total47100631009010092100 Mean0.001.730.040.74 SD0.000.900.210.71

34. Indexing Fidelity Absolute –Compare observed fidelity (t Tx ) to absolute or maximum level of fidelity (T Tx ) Average –Mean levels of observed fidelity (t Tx ) Binary –Yes/No treatment receipt based on fidelity scores –Requires selection of cut-off value

35. Fidelity Indices ConceptualLaboratoryClassroom AbsoluteTx C AverageTx1.730.74 C0.000.04 BinaryTx C

36. Indexing Fidelity as Achieved Relative Strength Intervention Strength = Treatment – Control Achieved Relative Strength (ARS) Index Standardized difference in fidelity index across Tx and C Based on Hedges’ g (Hedges, 2007) Corrected for clustering in the classroom (ICC’s from.01 to.08) See Hulleman & Cordray (2009)

37. Average ARS Index Where, = mean for group 1 (t Tx ) = mean for group 2 (t C ) S T = pooled within groups standard deviation n Tx = treatment sample size n C = control sample size n = average cluster size p = Intra-class correlation (ICC) N = total sample size Group DifferenceSample Size Adjustment Clustering Adjustment

38. Absolute and Binary ARS Indices Where, p Tx = proportion for the treatment group (t Tx ) p C = proportion for the control group (t C ) n Tx = treatment sample size n C = control sample size n = average cluster size p = Intra-class correlation (ICC) N = total sample size Group DifferenceSample Size Adjustment Clustering Adjustment

39. Achieved Relative Strength = 1.32 Fidelity Infidelity T Tx TCTC 100 66 33 0 Treatment Strength tCtC t tx 32103210 Average ARS Index (0.74)-(0.04) = 0.70

40. Achieved Relative Strength Indices Observed Fidelity Lab vs. Class Contrasts LabClassLab - Class AbsoluteTx0.580.25 C0.000.01 g1.720.80 0.92 AverageTx 1.730.74 C 0.000.04 g 2.521.32 1.20 BinaryTx0.650.15 C0.00 g1.880.801.08

41. Linking Achieved Relative Strength to Outcomes

42. Sources of Infidelity in the Classroom Student behaviors were nested within teacher behaviors Teacher dosage Frequency of student exposure Student and teacher behaviors were used to predict treatment fidelity (i.e., quality of responsiveness/exposure).

43. Sources of Infidelity: Multi-level Analyses Part I: Baseline Analyses Identified the amount of residual variability in fidelity due to students and teachers. –Du to missing data, we estimated a 2-level model (153 students, 6 teachers) Student:Y ij = b 0j + b 1j (TREATMENT) ij + r ij, Teacher:b 0j = γ 00 + u 0j, b 1j = γ 10 + u 10j

44. Sources of Infidelity: Multi-level Analyses Part II: Explanatory Analyses Predicted residual variability in fidelity (quality of responsiveness) with frequency of responsiveness and teacher dosage Student:Y ij = b 0j + b 1 (TREATMENT) ij + b 2 (RESPONSE FREQUENCY) ij + r ij Teacher:b 0j = γ 00 + u 0j b 1j = γ 10 + b 10 (TEACHER DOSAGE) j + u 10j b 2j = γ 20 + b 20 (TEACHER DOSAGE) j + u 20j

45. Sources of Infidelity: Multi-level Analyses Baseline ModelExplanatory Model Variance Component Residual Variance % of TotalVariance % Reduction Level 1 (Student) 0.15437*520.15346* < 1 Level 2 (Teacher) 0.13971*480.04924 65* Total 0.294080.20270 * p <.001.

46. Case Summary The motivational intervention was more effective in the lab (g = 0.45) than field (g = 0.05). Using 3 indices of fidelity and, in turn, achieved relative treatment strength, revealed that: –Classroom fidelity < Lab fidelity –Achieved relative strength was about 1 SD less in the classroom than the laboratory Differences in achieved relative strength = differences motivational outcome, especially in the lab. Sources of fidelity: teacher (not student) factors

47. Key Points and Issues Identifying and measuring, at a minimum, should include model-based core and necessary components Collaborations among researchers and practitioners (e.g., developers and implementers) is essential for specifying: –Intervention models –Core and essential components –Benchmarks for T Tx (e.g., an educationally meaningful dose; what level of X is needed to instigate change) –Tolerable adaptation

48. Key Points and Issues Fidelity assessment serves two roles: –Average causal difference between conditions; and –Using fidelity measures to assess the effects of variation in implementation on outcomes. Post-experimental (re)specification of the intervention

49. Thank You Questions and Discussion