1. Implementation of Randomized Trials
David Myers
American Institutes for Research
Washington, DC
Prepared for IES/NCER Summer Research Training Institute, 2008

2. Context Conducting effectiveness trials and not efficacy trials
Design and implementation of random assignment should not distort the program/intervention or the population served
Programs should run at capacity

3. Illustrative Studies: Personal Involvement
Upward Bound
Nationally representative study
67 sites and more than 3,000 students
Student RA within site
Workplace Literacy
3 sites and about 500 adult learners
Learner RA within site
"Addressing Literacy Needs at Work: Implementation and Impact of Workplace Literacy Programs. Final Report." Washington, DC: Mathematica Policy Research, Inc., October 1998

4. Illustrative Studies: Personal Involvement
21st Century Community Learning Centers
12 grantees (elementary school study)
Student RA within site
NYC Voucher Experiment
2000 students
Family RA

5. Illustrative Studies: Personal Involvement
Reading Comprehension
10 districts and 40 schools
School RA
Closing the Reading Gap
32 schools (4 interventions)
Student RA to reading groups within schools

6. Illustrative Studies: Personal Involvement
Quantum Opportunities
7 sites and about 1000 students
Student RA within site

7. Practical Problems in Implementation
Technical
Political and Ethical Challenges
Recruitment (not independent of P and E Challenges)

8. Further Exploration: Technical
Cross-overs
Members of C get into T or something like T (use UB example and James Comer example)
Into T -- straight forward adjustment (AIR, 1996)
Something like T -- ?; affects interpretation
Attrition
Post baseline -- “straight forward” adjustment (e.g., MI)
Before baseline -- problematic, no information
How to fix?????

9. Further Exploration: Technical
Unbalanced designs
Don’t demand additional recruitment
Large variance in selection probabilities (weights)
“over subscription” didn’t meet expectations (UB example)
Dishonest assignment -- post randomization
Sites don’t tell all students they have been selected for the program (UB example)
Sites ignore RA and move controls into T

10. Further Exploration: Ethical and Political Challenges
Random assignment isn’t fair
Programs afraid of denying services to students
Is it fair to never give a student a chance? (UB example)
Random assignment will force a program/teacher to serve a different population
Role of stratification to serve the desired mix
Some students (units) must be served
Role of the “wild card” before randomization
Wild cards excluded from analysis

11. Further Exploration: Ethical and Political Challenges
All “seats” must be filled
Role of the waiting list
Random selection within strata, if needed and desired by program operators
T and C groups analyzed as implemented at initial randomization
When programs believe recruitment will distort the population
Identify “most likely” and “least likely” to serve
Prior to randomization and stratify in analysis

12. Further Exploration: Ethical and Political Challenges
New treatments/interventions are hard to sell
Core programs vs. supplemental programs
Does the core program align with other curriculum and state assessments?
Concerns about making AYP
Will a supplemental program reduce hours of instruction in a core area such as ELA?

13. Strategies for Recruitment: Schools
Start with assistant superintendent for instruction (reading, math, science) or someone of similar stature
Quickly develop a relationship with an office and not just an individual -- staff come and go
Determine who needs to approve and to buy into participation
Superintendent -- they may need to go to the school board
Principals
Teachers
Parents and community groups

14. Strategies for Recruitment: Schools
Be prepared to meet with them
Have a recruitment team
Technical expertise in the design
Be prepared to tell audience why RA is valuable and not in a technical sense!
Deep knowledge about the intervention -- most important
Example of a meeting (next slide)

15. Irrefutable Evidence

16. Strategies for Recruitment: Schools
Establish expectations for the researchers and the schools
Minimize legal talk
Will bring in their general counsel and … .
Responsibilities and timelines
Obtaining consent
Data collection
Administering instruments and tests
Answering questions from parents and others