1. Rerandomization to Improve Baseline Balance in Educational Experiments
Kari Lock Morgan
Department of Statistics
Pennsylvania State University
with Anna Saavedra and Amie Rapaport
SREE
March 1st, 2018

2. Motivation RCTs are the “gold standard” for estimating causal effects
WHY?
They eliminate confounding variables (balance covariates)
They yield unbiased estimates
… on average!
For any particular experiment, covariate imbalance is possible (and likely!), and conditional bias exists

3. Typical RCT
Randomize units to treatment groups
Why not check balance before conducting the experiment, when you can still fix it?
Conduct experiment
Check baseline balance
Analyze results

4. Rerandomization Collect covariate data
Specify objective criteria for acceptable balance
(Re)randomize units to treatment groups
(Re)randomize units to treatment groups
Randomize units to treatment groups
Check balance
unacceptable
acceptable
Conduct experiment
Analyze results

5. Context
Students learn Advanced Placement (AP) content through the Knowledge in Action (KIA) project-based learning approach designed to develop students’ deeper learning of skills and content
RCT evaluation of KIA impact on student outcomes
Recruited teachers across five districts, teachers in 76 schools enrolled
Randomized at the school level within districts

6. KIA Covariates
Only previous cohort data available at the time of randomization
Covariates varied by district, but included
Standardized test scores (PSAT/AP/8th grade)
Socio-economic status
% Nonwhite (some districts)
Course (APES or APGOV) (some districts)

7. KIA Criteria Standardized difference in means:
| 𝑋 1, 𝑇 − 𝑋 1, 𝐶 | 𝑠 1 , | 𝑋 2, 𝑇 − 𝑋 2, 𝐶 | 𝑠 2 , ...
Thresholds varied by district: 0.05 – 0.25
Another option is Mahalanobis distance:
𝑿 𝑇 − 𝑿 𝑐 ′ cov 𝒙 −1 𝑿 𝑇 − 𝑿 𝑐

8. Covariate Balance: One District
Percent reduction in variance: 𝑃𝑅𝐼𝑉= 𝑣𝑎𝑟 𝑥 𝑗, 𝑇 − 𝑥 𝑗, 𝐶 −𝑣𝑎𝑟 𝑥 𝑗, 𝑇 − 𝑥 𝑗, 𝐶 | 𝑟𝑒𝑟𝑎𝑛𝑑. 𝑣𝑎𝑟 𝑥 𝑗, 𝑇 − 𝑥 𝑗, 𝐶

9. Covariate Balance

10. Outcome Precision
If PRIV is equal for all covariates, then PRIV for the outcome difference in means is
𝑃𝑅𝐼𝑉 𝑌 = 𝑅 2 × 𝑃𝑅𝐼𝑉 𝑋
Here, 𝑅 2 ≈0.75 and 𝑃𝑅𝐼𝑉 𝑋 ≈90%, so
𝑃𝑅𝐼𝑉 𝑌 ≈0.75×0.90=67.5%
Precision increases by a factor of 1 1− = 3.1
Equivalent to more than tripling n!!! (Effective sample size goes from 76 to 234!)
76 to 293

11. More Power! Significance for smaller effect sizes!
Use randomization test to take advantage of this; otherwise inference will be conservative

12. Regression
Rerandomization reduces the need to account for covariates via modeling
But if you do still choose to model…
𝑌 𝑖 =𝛼+𝜷 𝑿 𝑖 +𝜏 𝑊 𝑖 + 𝜀 𝑖
Better covariate balance: estimation of 𝜏 depends less on estimation of 𝜷…
increases precision/power
reduces reliance on modeling assumptions

13. Why Rerandomize? Avoid bad/unlucky randomizations
Improve covariate balance
Increase power
Reduce reliance on assumptions

14.
Morgan, K.L., and Rubin, D.B. (2012). “Rerandomization to Improve Covariate Balance in Experiments,” Annals of Statistics, 40(2):
Morgan, K.L. and Rubin, D.B. (2015). “Rerandomization to Balance Tiers of Covariates,” JASA, 110(512): 1412 – 1421.