1. Assessing Interventions and Control Conditions in RCTs: Concepts and Methods
David S. Cordray, PhD
Vanderbilt University
Presentation for the IES/NCER Summer Research Training Institute: Cluster-Randomized Trials
Northwestern University
Evanston, Illinois
July 7, 2008

2. Overview
Fidelity of Intervention Implementation: Definitions and distinctions
Conceptual foundation for assessing fidelity in RCTs, a special case.
Model-based assessment of implementation fidelity
Models of change
“Logic model”
Program model
Research model
Indexing Fidelity
Methods of Assessment
Sample-based fidelity assessment
Regression-based fidelity assessment
Questions and discussion

3. Intervention Fidelity: Definitions, Distinctions, and Some Examples

4. Dimensions Intervention Fidelity
Little consensus on what is meant by the term “intervention fidelity”.
But Dane & Schneider (1998) identify 5 aspects:
Adherence/compliance– program components are delivered/used/received, as prescribed;
Exposure – amount of program content delivered/received by participants;
Quality of the delivery – theory-based ideal in terms of processes and content;
Participant responsiveness – engagement of the participants; and
Program differentiation – unique features of the intervention are distinguishable from other programs (including the counterfactual)

5. Distinguishing Implementation Assessment from Implementation Fidelity Assessment
Two models of intervention implementation, based on:
A purely descriptive model
Answering the question “What transpired as the intervention was put in place (implemented).
An a priori intervention model, with explicit expectations about implementation of core program components.
Fidelity is the extent to which the realized intervention (tTx) is “faithful” to the pre-stated intervention model (TTx)
Fidelity = TTx – tTx
We emphasize this model, but both are important

6. Some Examples
The following examples are from an 8-year, NSF-supported project involving biomedical engineering education at Vanderbilt, Northwestern, Texas, Harvard/MIT (VaNTH, Thomas Harris, MD, PhD, Director)
The goal was to change the curriculum to incorporate principles of “How People Learn” (Bransford et al. and the National Academy of Sciences, 1999).
We’ll start with a descriptive question, move to model-based examples.

7. Descriptive Assessment: Expectations about Organizational Change
From: Cordray, Pion & Harris, 2008

8. Macro-Implementation
From: Cordray, Pion & Harris, 2008

9. Changes in Learning Orientation
From: Cordray, Pion & Harris, 2008

10. Model Based Fidelity Assessment: What to Measure?
Adherence to the intervention model:
(1) Essential or core components (activities, processes);
(2) Necessary, but not unique to the theory/model, activities, processes and structures (supporting the essential components of T); and
(3) Ordinary features of the setting (shared with the counterfactual groups (C)
Essential/core and Necessary components are priority parts of fidelity assessment.

11. An Example of Core Components” Bransford’s HPL Model of Learning and Instruction
John Bransford et al. (1999) postulate that a strong learning environment entails a combination of:
Knowledge-centered;
Learner-centered;
Assessment-centered; and
Community-centered components.
Alene Harris developed an observation system (the VOS) that registered novel (components above) and traditional pedagogy in classes.
The next slide focuses on the prevalence of Bransford’s recommended pedagogy.

12. Challenge-based Instruction in HPL-based Intervention Courses: The VaNTH Observation System (VOS)
Percentage of Course Time Using Challenge-based Instructional Strategies
Adapted from Cox & Cordray, in press

13. Challenge-based Instruction in “Treatment” and Control Courses: The VaNTH Observation System (VOS)
Percentage of Course Time Using Challenge-based Instructional Strategies
Adapted from Cox & Cordray, in press 13

14. Student-based Ratings of HPL Instruction in HPL and non-HPL Courses
We also examined the same question from the students point of view through surveys (n=1441):
Scale
BME Program
HPL Courses
Non-HPL Courses
Effect Size
Mean Sd
N (Courses) N
(Courses)
Knowledge, Learner, Assessment (KLA) VU
58.2
8.74 34
53.0
7.96 16
0.60 NU
51.1
8.28 17
41.0
12.11 22
0.93 UT
52.7
2.51 2
44.9
13.00 10
0.69
Community
13.6
2.82
14.0
2.90
- 0.15
13.9
4.58
10.6
3.93
0.80
17.3
4.57
9.9
5.50
1.40
From: Cordray, Pion & Harris, 2008

15. Implications Descriptive assessments involve:
Expectations
Multiple data sources
Can assist in explaining outcomes
Model-based assessments involve:
Benchmarks for success (e.g., the optimal fraction of time devoted to HPL-based instruction)
With comparative evidence, fidelity can be assessed even when there is no known benchmark (e.g., 10 Commandments)
In practice interventions can be a mixture of components with strong, weak or no benchmarks
Control conditions can include core intervention components due to:
Contamination
Business as usual (BAU) contains shared components, different levels
Similar theories, models of action
To index fidelity, we need to measure, at a minimum, intervention components within the control condition.

16. Conceptual Foundations for Fidelity Assessment within Cluster Randomized Controlled Trials

17. Linking Intervention Fidelity Assessment to Contemporary Models of Causality
Rubin’s Causal Model:
True causal effect of X is (YiTx – YiC)
RCT methodology is the best approximation to the true effect
Fidelity assessment within RCT-based causal analysis entails examining the difference between causal components in the intervention and counterfactual condition.
Differencing causal conditions can be characterized as “achieved relative strength” of the contrast.
Achieved Relative Strength (ARS) = tTx – tC
ARS is a default index of fidelity

18. Infidelity and Relevant Threats to Validity
Statistical Conclusion validity
Unreliability of Treatment Implementation (TTX-tTx) : Variations across participants in the delivery receipt of the causal variable (e.g., treatment). Increases error and reduces the size of the effect; decreases chances of detecting covariation.
Construct Validity – cause [(TTx – tTx) –(TC-tC)]
Forms of Contamination:
Compensatory Rivalry: Members of the control condition attempt to out-perform the participants in the intervention condition (The classic example is the “John Henry Effect”).
Treatment Diffusion: The essential elements of the treatment group are found in the other conditions (to varying degrees).
External validity – generalization is about (tTx-tC)
Variation across settings, cohort by treatment interactions

19. .45
.40
.35
.30
.25
.20
.15
.10
.05
.00
Treatment Strength
100 90 85 80 75 70 65 60 55 50
Outcome
TTx TC
Infidelity
“Infidelity”
(85)-(70) = 15
txC
t tx
Achieved Relative Strength =.15
Expected Relative Strength =.25

20. In Practice…. Identify core components in both groups
e.g., via a Model of Change
Establish bench marks for TTX and TC;
Measure core components to derive tTx and tC
e.g., via a “Logic model” based on Model of Change
Research methods
With multiple components and multiple methods of assessment; achieved relative strength needs to be:
Standardized indices of fidelity
Absolute
Average
Binary
Converted to Achieved Relative Strength, and
Combined across:
Multiple indicators
Multiple components
Multiple levels (HLM-wise)

21. Indexing Fidelity Absolute Average Binary
Compare observed fidelity (tTx) to absolute or maximum level of fidelity (TTx)
Average
Mean levels of observed fidelity (tTx and tC)
Binary
Yes/No treatment receipt based on fidelity scores (both groups)
Requires selection of cut-off value
Somewhere in this section I may want to refer back to the original figure that Dave presents on Achieved Relative Strength and show each of these indices (absolute, average, and binary) as they map onto that figure. 21

22. Indexing Fidelity as Achieved Relative Strength
Intervention Strength = Treatment – Control
Achieved Relative Strength (ARS) Index
Standardized difference in fidelity index across Tx and C
Based on Hedges’ g (Hedges, 2007)
Corrected for clustering in the classroom
By itself, treatment fidelity does not help us distinguish the strength of the intervention – it is only in comparison to the control condition that we can determine the achieved relative strength of the intervention.
Achieved Relative Strength (ARS) Index = standardized difference in the fidelity index across treatment and control conditions (expressed in SD units). Based on Hedges’ g (Hedges, 2007).
Intra-class correlations (ICC’s) were actually quite small:
ICC for quality of responsiveness = 0.08
ICC for perceived utility value = 0.01 22 22

23. Average ARS Index Group Difference Sample Size Adjustment
Clustering Adjustment
Where, = mean for group 1 (tTx ) = mean for group 2 (tC) ST = pooled within groups standard deviation nTx = treatment sample size nC = control sample size n = average cluster size p = Intra-class correlation (ICC) N = total sample size 23 23

24. Example –The Measuring Academic Progress (MAP) RCT
The Northwest Evaluation Association (NWEA) developed the Measures of Academic Progress (MAP) program to enhance student achievement
Used in school districts, 17,500 schools
No evidence of efficacy or effectiveness
The upcoming example presents heuristics for translating conceptual variables into operational form.

25. MAP’s Simple Model of Change
Feedback
Professional Development
Achievement
Differentiated Instruction

26. Conceptual Model for the Measuring Academic Progress (MAP) Program

27. Operational Intervention Model: MAP
Academic Schedule
Fall Semester
Spring Semester
Aug
Sept
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
MAP Activity
PD1
PD2
PD3
PD4
Data Sys
Use Data
Diff Instr
Change
State Testing
Full Implementation Interval

28. Final RCT Design: 2-Year Wait Control

29. Translating Model of Change into Activities: the “Logic Model”
From: W.T. Kellogg Foundation, 2004

30. Moving from Logic Model Components to Measurement
The MAP Model:
Feedback
Achievement
Professional Development
Differentiated Instruction
Resources:
3 Computer Adaptive Testing
DesCarte system
Activities:
Grouping of students
Continuous assessment
Resources:
Four training sessions
On-line resources
Outcomes & Measures
Attendance
Knowledge Acquisition
Outcomes & Measures
Testing completed
Access DesCarte
Outcomes & Measures
Changes in pedagogy
Outcomes & Measures
State tests
MAP assessments

31. Fidelity Assessment Plan for the MAP Program

32. Measuring Resources, Activities and Outputs
Observations
Structured
Unstructured
Interviews
Surveys
Existing scales/instruments
Teacher Logs
Administrative Records

33. Sampling Strategies Census Sampling Probabilistic Non-probability
Persons (units)
Institutions
Time
Non-probability
Modal instance
Heterogeneity
Key events

34. Key Points and Future Issues
Identifying and measuring, at a minimum, should include model-based core and necessary components;
Collaborations among researchers, developers and implementers is essential for specifying:
Intervention models;
Core and essential components;
Benchmarks for TTx (e.g., an educationally meaningful dose; what level of X is needed to instigate change); and
Tolerable adaptation

35. Points and Issues Fidelity assessment serves two roles:
Average causal difference between conditions; and
Using fidelity measures to assess the effects of variation in implementation on outcomes.
Should minimize “infidelity” and weak ARS:
Pre-experimental assessment of TTx in the counterfactual condition…Is TTx > TC?
Build operational models with positive implementation drivers
Post-experimental (re)specification of the intervention: For example …..

36. Intervention and Control Components
Infidelity
Augmentation of Control
PD= Professional Development
Asmt=Formative Assessment
Diff Inst= Differentiated Instruction

37. Questions and Discussion

38. Small Group Projects

39. Overview Logistics: Parameters for the group project
Rationale for the group project
Group assignments
Resources
ExpERT (Experimental Education Research Training) Fellows
Parameters for the group project
Small group discussions

40. Rationale for the Project
Rationale for the group projects:
Purpose of this training is to enhance skills in planning, executing and reporting cluster RCTs.
Various components of RCTs are, by necessity, presented serially.
The ultimate design for an RCT is the product of:
Tailoring of design, measurement, and analytic strategies to a given problem.
Successive iterations as we attempt to optimize all features of the design.
The project will provide a chance to engage in these practices, with guidance from your colleagues.

41. About Group Assignments….
We are assuming that RCTs need to be grounded in specific topical areas.
There is a diversity of topical interests represented.
The group assignments may not be optimal.
To manage the guidance and reporting functions we need to have a small number of groups.

42. Resources ExpERT Fellows:
Laura Williams – Quantitative Methods and Evaluation
Chuck Munter – Teaching and Learning
David Stuit – Leadership and Policy

43. Parameters of the Proposal
IES goal is to support research that contributes to the solution of education problems.
RFA IES-NCER provides extensive information about the proposal application and review process.
Proposals are reviewed in 4 areas:
Significance
Research Plan
Personnel
Resources
For our purposes, we’ll focus on Significance and the Research Plan.

44. Significance

45. Research Plan

46. Awards/Duration

47. Group Project Report
Each group will present its proposal on Thursday (60 minutes each
45 minutes for the proposal
15 minutes for discussion
Ideally, each report will contain:
Problem statement, intervention description, rationale for why it should work (10-15 minutes)
Overview of the research plan
Samples
Groups and Assignment
Power
Fidelity Assessment
Outcomes
Impact Analysis Plan
Use tables, figures and bullet points in your presentation

48. Expectations You will produce a rough plan
Some details will be guesses
The planning processes is often iterative, with the need to revisit earlier steps and specifications.
Flexibility helps….

49. Initial Group Interactions
Meet with your assigned group (45 minutes) to assess “common ground”
Group discussion of “common issues”