1. Sample Design for Group- Randomized Trials Howard S. Bloom Chief Social Scientist MDRC Prepared for the IES/NCER Summer Research Training Institute held at Northwestern University on July 27, 2010.

2. Today we will examine Sample size determinants Sample size determinants Precision requirements Precision requirements Sample allocation Sample allocation Covariate adjustments Covariate adjustments Matching and blocking Matching and blocking Subgroup analyses Subgroup analyses Generalizing findings for sites and blocks Generalizing findings for sites and blocks Using two-level data for three-level situations Using two-level data for three-level situations

3. Part I: The Basics

4. Statistical properties of group- randomized impact estimators Unbiased estimates Y ij =  +B 0 T j +e j +  ij E(b 0 ) = B 0 Less precise estimates VAR(  ij ) =  2 VAR(e j ) =  2  =  2 /(  2 +  2 )

5. Design Effect (for a given total number of individuals)______________________________________ Intraclass Individuals per Group (n) Correlation (         _____________________________________

6. Sample design parameters Number of randomized groups (J) Number of randomized groups (J) Number of individuals per randomized group (n) Number of individuals per randomized group (n) Proportion of groups randomized to program status (P) Proportion of groups randomized to program status (P)

7. Reporting precision A minimum detectable effect (MDE) is the smallest true effect that has a “good chance” of being found to be statistically significant. A minimum detectable effect (MDE) is the smallest true effect that has a “good chance” of being found to be statistically significant. We typically define an MDE as the smallest true effect that has 80 percent power for a two-tailed test of statistical significance at the 0.05 level. We typically define an MDE as the smallest true effect that has 80 percent power for a two-tailed test of statistical significance at the 0.05 level. An MDE is reported in natural units whereas a minimum detectable effect size (MDES) is reported in units of standard deviations An MDE is reported in natural units whereas a minimum detectable effect size (MDES) is reported in units of standard deviations

8. Minimum Detectable Effect Sizes For a Group-Randomized Design with  = 0.05 and no Covariates ___________________________________ Randomized Individuals per Group (n) Groups (J) 10 50 500 10 0.77 0.53 0.46 10 0.77 0.53 0.46 20 0.50 0.35 0.30 20 0.50 0.35 0.30 40 0.35 0.24 0.21 40 0.35 0.24 0.21 120 0.20 0.14 0.12 120 0.20 0.14 0.12___________________________________

9. Implications for sample design It is extremely important to randomize an adequate number of groups. It is extremely important to randomize an adequate number of groups. It is often far less important how many individuals per group you have. It is often far less important how many individuals per group you have.

10. Part II Determining required precision

11. When assessing how much precision is needed: Always ask “relative to what?”  Program benefits  Program costs  Existing outcome differences  Past program performance

12. Effect Size Gospel According to Cohen and Lipsey Cohen Lipsey Cohen Lipsey (speculative) (empirical) (speculative) (empirical)_______________________________________________ Small = 0.2  Small = 0.15  Small = 0.2  Small = 0.15  Medium = 0.5  Medium = 0.45  Large = 0.8  Large = 0.90  Large = 0.8  Large = 0.90 

13. Five-year impacts of the Tennessee class-size experiment Treatment: 13-17 versus 22-26 students per class Effect sizes: 0.11  to 0.22  for reading and math Findings are summarized from Nye, Barbara, Larry V. Hedges and Spyros Konstantopoulos (1999) “The Long-Term Effects of Small Classes: A Five- Year Follow-up of the Tennessee Class Size Experiment,” Educational Evaluation and Policy Analysis, Vol. 21, No. 2: 127-142.

14. Annual reading and math growth ReadingMath Grade Growth Growth Transition Effect Size Effect Size ---------------------------------------------------------------- K - 1 1.52 1.14 K - 1 1.52 1.14 1 - 2 0.97 1.03 1 - 2 0.97 1.03 2 - 3 0.60 0.89 2 - 3 0.60 0.89 3 - 4 0.36 0.52 3 - 4 0.36 0.52 4 - 5 0.40 0.56 4 - 5 0.40 0.56 5 - 6 0.32 0.41 5 - 6 0.32 0.41 6 - 7 0.23 0.30 6 - 7 0.23 0.30 7 - 8 0.26 0.32 7 - 8 0.26 0.32 8 - 9 0.24 0.22 8 - 9 0.24 0.22 9 - 10 0.19 0.25 9 - 10 0.19 0.25 10 - 11 0.19 0.14 10 - 11 0.19 0.14 11 - 12 0.06 0.01 11 - 12 0.06 0.01------------------------------------------------------------------------------------------------- Based on work in progress using documentation on the national norming samples for the CAT5, SAT9, Terra Nova CTBS, Gates MacGinitie (for reading only), MAT8, Terra Nova CAT, and SAT10. 95% confidence intervals range in reading from +/-.03 to.15 and in math from +/-.03 to.22

15. Performance gap between “average” (50 th percentile) and “weak” (10 th percentile) schools Subject and gradeDistrict IDistrict IIDistrict IIIDistrict IV Reading Grade 30.310.180.160.43 Grade 50.410.180.350.31 Grade 7.0250.110.30NA Grade 100.070.11NA Math Grade 30.290.250.190.41 Grade 50.270.230.360.26 Grade 70.200.150.23NA Grade 100.140.17NA Source: District I outcomes are based on ITBS scaled scores, District II on SAT 9 scaled scores, District III on MAT NCE scores, and District IV on SAT 8 NCE scores.

16. Demographic performance gap in reading and math: Main NAEP scores Subject and grade Black- White Hispanic- White Male- Female Eligible- Ineligible for free/reduced price lunch Reading Grade 4-0.83-0.77-0.18-0.74 Grade 8-0.80-0.76-0.28-0.66 Grade 12-0.67-0.53-0.44-0.45 Math Grade 4-0.99-0.850.08-0.85 Grade 8-1.04-0.820.04-0.80 Grade 12-0.94-0.680.09-0.72 Source: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2002 Reading Assessment and 2000 Mathematics Assessment.

17. ES Results from Randomized Studies Achievement MeasurenMean Elementary School3890.33 Standardized test (Broad)210.07 Standardized test (Narrow)1810.23 Specialized Topic/Test1800.44 Middle Schools360.51 High Schools430.27

18. Part III The ABCs of Sample Allocation

19. Sample allocation alternatives Balanced allocation  maximizes precision for a given sample size;  maximizes robustness to distributional assumptions. Unbalanced allocation  precision erodes slowly with imbalance for a given sample size  imbalance can facilitate a larger sample  Imbalance can facilitate randomization

20. Variance relationships for the program and control groups Equal variances: when the program does not affect the outcome variance. Equal variances: when the program does not affect the outcome variance. Unequal variances: when the program does affect the outcome variance. Unequal variances: when the program does affect the outcome variance.

21. MDES for equal variances without covariates

22. How allocation affects MDES

23. Minimum Detectable Effect Size For Sample Allocations Given Equal Variances Allocation Example * Ratio to Balanced Allocation Example * Ratio to Balanced Allocation Allocation 0.5/0.5 0.54  1.00 0.5/0.5 0.54  1.00 0.6/0.4 0.55  1.02 0.6/0.4 0.55  1.02 0.7/0.3 0.59  1.09 0.7/0.3 0.59  1.09 0.8/0.2 0.68  1.25 0.8/0.2 0.68  1.25 0.9/0.1 0.91  1.67 0.9/0.1 0.91  1.67________________________________________ * Example is for n = 20, J = 10,  = 0.05, a one-tail hypothesis test and no covariates.

24. Implications of unbalanced allocations with unequal variances

25. Implications Continued The estimated standard error is unbiased  When the allocation is balanced  When the variances are equal The estimated standard error is biased upward  When the larger sample has the larger variance The estimated standard error is biased downward  When the larger sample has the smaller variance

26. Interim Conclusions Don’t use the equal variance assumption for an unbalanced allocation with many degrees of freedom. Don’t use the equal variance assumption for an unbalanced allocation with many degrees of freedom. Use a balanced allocation when there are few degrees of freedom. Use a balanced allocation when there are few degrees of freedom.

27. References Gail, Mitchell H., Steven D. Mark, Raymond J. Carroll, Sylvan B. Green and David Pee (1996) “On Design Considerations and Randomization-Based Inferences for Community Intervention Trials,” Statistics in Medicine 15: 1069 – 1092. Bryk, Anthony S. and Stephen W. Raudenbush (1988) “Heterogeneity of Variance in Experimental Studies: A Challenge to Conventional Interpretations,” Psychological Bulletin, 104(3): 396 – 404.

28. Part IV Using Covariates to Reduce Sample Size

29. Basic ideas Goal: Reduce the number of clusters randomized Goal: Reduce the number of clusters randomized Approach: Reduce the standard error of the impact estimator by controlling for baseline covariates Approach: Reduce the standard error of the impact estimator by controlling for baseline covariates Alternative Covariates Alternative Covariates  Individual-level  Cluster-level  Pretests  Other characteristics

30. Impact Estimation with a Covariate y ij = the outcome for student i from school j T j = 1 for treatment schools and 0 for control schools X j = a covariate for school j x ij = a covariate for student i from school j x ij = a covariate for student i from school j e j = a random error term for school j  ij = a random error term for student i from school j

31. Minimum Detectable Effect Size with a Covariate MDES = minimum detectable effect size M J-K = a degrees-of-freedom multiplier 1 J = the total number of schools randomized n = the number of students in a grade per school P = the proportion of schools randomized to treatment  = the unconditional intraclass correlation (without a covariate) R 1 2 = the proportion of variance across individuals within schools (at level 1) predicted by the covariate R 2 2 = the proportion of variance across schools (at level 2) predicted by the covariate 1 For 20 or more degrees of freedom M J-K equals 2.8 for a two-tail test and 2.5 for a one-tail test with statistical power of 0.80 and statistical significance of 0.05

32. Questions Addressed Empirically about the Predictive Power of Covariates School-level vs. student-level pretests School-level vs. student-level pretests Earlier vs. later follow-up years Earlier vs. later follow-up years Reading vs. math Reading vs. math Elementary vs. middle vs. high school Elementary vs. middle vs. high school All schools vs. low-income schools vs. low-performing schools All schools vs. low-income schools vs. low-performing schools

33. Empirical Analysis Estimate , R 2 2 and R 1 2 from data on thousands of students from hundreds of schools, during multiple years at five urban school districts Estimate , R 2 2 and R 1 2 from data on thousands of students from hundreds of schools, during multiple years at five urban school districts Summarize these estimates for reading and math in grades 3, 5, 8 and 10 Summarize these estimates for reading and math in grades 3, 5, 8 and 10 Compute implications for minimum detectable effect sizes Compute implications for minimum detectable effect sizes

34. Estimated Parameters for Reading with a School-level Pretest Lagged One Year___________________________________________________________________ School District ___________________________________________________________ ___________________________________________________________ A B C D E A B C D E___________________________________________________________________ Grade 3  0.20 0.15 0.19 0.22 0.16  0.20 0.15 0.19 0.22 0.16 R 2 2 0.31 0.77 0.74 0.51 0.75 R 2 2 0.31 0.77 0.74 0.51 0.75 Grade 5  0.25 0.15 0.20 NA 0.12  0.25 0.15 0.20 NA 0.12 R 2 2 0.33 0.50 0.81 NA 0.70 R 2 2 0.33 0.50 0.81 NA 0.70 Grade 8  0.18 NA 0.23 NA NA  0.18 NA 0.23 NA NA R 2 2 0.77 NA 0.91 NA NA R 2 2 0.77 NA 0.91 NA NA Grade 10  0.15 NA 0.29 NA NA  0.15 NA 0.29 NA NA R 2 2 0.93 NA 0.95 NA NA R 2 2 0.93 NA 0.95 NA NA____________________________________________________________________

35. Minimum Detectable Effect Sizes for Reading with a School-Level Pretest (Y -1 ) or a Student-Level Pretest (y -1 ) Lagged One Year ________________________________________________________ Grade 3 Grade 5 Grade 8 Grade 10 Grade 3 Grade 5 Grade 8 Grade 10________________________________________________________ 20 schools randomized No covariate 0.57 0.56 0.61 0.62 No covariate 0.57 0.56 0.61 0.62 Y -1 0.37 0.38 0.24 0.16 Y -1 0.37 0.38 0.24 0.16 y -1 0.38 0.40 0.28 0.15 y -1 0.38 0.40 0.28 0.15 40 schools randomized No covariate 0.39 0.38 0.42 0.42 No covariate 0.39 0.38 0.42 0.42 Y -1 0.26 0.26 0.17 0.11 Y -1 0.26 0.26 0.17 0.11 y -1 0.26 0.27 0.19 0.10 y -1 0.26 0.27 0.19 0.10 60 schools randomized No covariate 0.32 0.31 0.34 0.34 No covariate 0.32 0.31 0.34 0.34 Y -1 0.21 0.21 0.13 0.09 Y -1 0.21 0.21 0.13 0.09 y -1 0.21 0.22 0.15 0.08 y -1 0.21 0.22 0.15 0.08________________________________________________________

36. Key Findings Using a pretest improves precision dramatically. Using a pretest improves precision dramatically. This improvement increases appreciably from elementary school to middle school to high school because R 2 2 increases. This improvement increases appreciably from elementary school to middle school to high school because R 2 2 increases. School-level pretests produce as much precision as do student-level pretests. School-level pretests produce as much precision as do student-level pretests. The effect of a pretest declines somewhat as the time between it and the post-test increases. The effect of a pretest declines somewhat as the time between it and the post-test increases. Adding a second pretest increases precision slightly. Adding a second pretest increases precision slightly. Using a pretest for a different subject increases precision substantially. Using a pretest for a different subject increases precision substantially. Narrowing the sample to schools that are similar to each other does not improve precision beyond that achieved by a pretest. Narrowing the sample to schools that are similar to each other does not improve precision beyond that achieved by a pretest.

37. Source Bloom, Howard S., Lashawn Richburg-Hayes and Alison Rebeck Black (2007) “Using Covariates to Improve Precision for Studies that Randomize Schools to Evaluate Educational Interventions” Educational Evaluation and Policy Analysis, 29(1): 30 – 59.

38. Part V The Putative Power of Pairing A Tail of Two Tradeoffs (“It was the best of techniques. It was the worst of techniques.” Who the dickens said that?)

39. Pairing Why match pairs? Why match pairs?  for face validity  for precision How to match pairs? How to match pairs?  rank order clusters by covariate  pair clusters in rank-ordered list  randomize clusters in each pair

40. When to pair? When the gain in predictive power outweighs the loss of degrees of freedom When the gain in predictive power outweighs the loss of degrees of freedom Degrees of freedom Degrees of freedom  J - 2 without pairing  J/2 - 1 with pairing

41. Deriving the Minimum Required Predictive Power of Pairing Without pairing With pairing Breakeven R 2

42. The Minimum Required Predictive Power of Pairing Randomized Required Predictive Clusters (J) Power (R min 2 ) * Clusters (J) Power (R min 2 ) * 60.52 60.52 80.35 80.35 100.26 100.26 200.11 200.11 30 0.07 30 0.07 * For a two-tail test.

43. A few key points about blocking Blocking for face validity vs. blocking for precision Blocking for face validity vs. blocking for precision Treating blocks as fixed effects vs.random effects Treating blocks as fixed effects vs.random effects Defining blocks using baseline information Defining blocks using baseline information

44. Part VI Subgroup Analyses #1: When to Emphasize Them

45. Confirmatory vs. Exploratory Findings Confirmatory: Draw conclusions about the program’s effectiveness if results are Confirmatory: Draw conclusions about the program’s effectiveness if results are  Consistent with theory and contextual factors  Statistically significant and large  And subgroup was pre-specified Exploratory: Develop hypotheses for further study Exploratory: Develop hypotheses for further study 45

46. Pre-specification Before the analysis, state that conclusions about the program will be based in part on findings for this set of subgroups Before the analysis, state that conclusions about the program will be based in part on findings for this set of subgroups Pre-specification can be based on Pre-specification can be based on  Theory  Prior evidence  Policy relevance 46

47. Statistical significance When should we discuss subgroup findings? When should we discuss subgroup findings? Depends on Depends on  Whether significant differences in impacts across subgroups  Might depend on whether impacts for the full sample are statistically significant 47

48. Part VII Subgroup Analyses #2: Creating Subgroups

49. Defining Features Creating subgroups in terms of: Creating subgroups in terms of:  Program characteristics  Randomized group characteristics  Individual characteristics

50. Defining Subgroups by Program Characteristics Based only on program features that were randomized Based only on program features that were randomized Thus one cannot use implementation quality Thus one cannot use implementation quality

51. Defining Subgroups by Characteristics Of Randomized Groups Types of impacts Types of impacts  Net impacts  Differential impacts Internal validity Internal validity  only use pre-existing characteristics Precision Precision  Net impact estimates are limited by reduced number of randomized groups  Differential impact estimates are triply limited (and often need four times as many randomized groups)

52. Defining Subgroups by Characteristics of Individuals Types of impacts Types of impacts  Net impacts  Differential impacts Internal validity Internal validity  Only use pre-existing characteristics  Only use subgroups with sample members from all randomized groups Precision Precision  For net impacts: can be almost as good as for full sample  For differential impacts: can be even better than for full sample

54. Part VIII Generalizing Results from Multiple Sites and Blocks

55. Fixed vs. Random Effects Inference: A Vexing Issue  Known vs. unknown populations  Broader vs. narrower inferences  Weaker vs. stronger precision  Few vs. many sites or blocks

56. Weighting Sites and Blocks Implicitly through a pooled regression Implicitly through a pooled regression Explicitly based on Explicitly based on  Number of schools  Number of students Explicitly based on precision Explicitly based on precision  Fixed effects  Random effects Bottom line: the question addressed is what counts Bottom line: the question addressed is what counts

57. Part IX Using Two-Level Data for Three- Level Situations

58. The Issue General Question: What happens when you design a study with randomized groups that comprise three levels based on data which do not account explicitly for the middle level? General Question: What happens when you design a study with randomized groups that comprise three levels based on data which do not account explicitly for the middle level? Specific Example: What happens when you design a study that randomizes schools (with students clustered in classrooms in schools) based on data for students clustered in schools? Specific Example: What happens when you design a study that randomizes schools (with students clustered in classrooms in schools) based on data for students clustered in schools?

59. 3-level vs. 2-level Variance Components

60. 3-level vs. 2-level MDES for Original Sample

61. Further References Bloom, Howard S. (2005) “Randomizing Groups to Evaluate Place-Based Programs,” in Howard S. Bloom, editor, Learning More From Social Experiments: Evolving Analytic Approaches (New York: Russell Sage Foundation). Bloom, Howard S., Lashawn Richburg-Hayes and Alison Rebeck Black (2005) “Using Covariates to Improve Precision: Empirical Guidance for Studies that Randomize Schools to Measure the Impacts of Educational Interventions” (New York: MDRC). Donner, Allan and Neil Klar (2000) Cluster Randomization Trials in Health Research (London: Arnold). Hedges, Larry V. and Eric C. Hedberg (2006) “Intraclass Correlation Values for Planning Group Randomized Trials in Education” (Chicago: Northwestern University). Murray, David M. (1998) Design and Analysis of Group-Randomized Trials (New York: Oxford University Press). Raudenbush, Stephen W., Andres Martinez and Jessaca Spybrook (2005) “Strategies for Improving Precision in Group-Randomized Experiments” (University of Chicago). Raudenbush, Stephen W. (1997) “Statistical Analysis and Optimal Design for Cluster Randomized Trials” Psychological Methods, 2(2): 173 – 185. Schochet, Peter Z. (2005) “Statistical Power for Random Assignment Evaluations of Education Programs,” (Princeton, NJ: Mathematica Policy Research).