1. Agenda: Block Watch: Random Assignment, Outcomes, and indicators Issues in Impact and Random Assignment: Youth Transition Demonstration –Who is randomized? –Sample size, power, and effect size –Who’s in the average?

2. Block Watch: Random Assignment, Outcomes, and Indicators What random assignment protocol would you use to assess the impacts of Block Watch? What are the strengths and weaknesses of your approach? What are the key outcomes you want to assess? What are indicators for those?

3. Youth Transition Demonstration Evaluation Plan: Background on YTD evaluation plan The basics of Impact size and significance Power and sample size No Shows/ Intent to Treat vs. Treatment on the Treated Multiple Comparisons Regression adjusted comparisons

4. Youth Transition Demonstration: Targets Youth receiving disability payments to help in transition to adult life and employment Goals: increase earnings, decrease costs, facilitate transition to self-sufficiency Six program sites with variation in programs Services: –Waiver of benefit decrease with earnings –Education, Job training, work placements –Case management, counseling, referral to services

7. YTD Evaluation: Selected 6 sites for demonstration and evaluation Intervention built on research from past programs and evaluations Randomly assigned youth to treatment or control Large sample sizes to allow identification of smaller effects and sub-group effects Process and Impact Evaluation Data collected from administrative files, surveys before and after program Advisory group of experts

9. Sampling: Why did they divide the list of potential participants (sampling frame) into groups of 10 for contact? Why did they randomize 55 percent to the treatment? Why get pre-intervention characteristics if they are randomly assigning groups?

10. Comparisons may be: -over time -across intervention groups with and without program; levels of intervention (“dosage”) Impact here!

11. Statistical significance: When can we rule out having an impact IF there is no impact? Compare 2 means from independent samples: Means: Proportions: Pooled sample variance:

12. Compare 2 means from independent samples: Means: Proportions: Pooled sample variance:

13. Compare 2 means from independent samples: Means: Proportions: Pooled sample variance:

14. Compare 2 means from independent samples: Means: Proportions: Pooled sample variance:

15. So, it’s easier to say impact is “real” (not just randomness) if: –Size of impact is larger –Variation in outcomes is small (S) –Sample sizes are larger Same factors figure into deciding how big a sample we need to find the effect if it’s there! [Power, sample size, minimally detectable effects]

16. Power and sample size: Given randomness, what % of time will you be able to rule out the null, IF it is NOT true (there IS an impact)? How big a sample size do you need to rule out NO effect if the program DOES have an impact? (Rossi et al p.312)

17. Online Calculators for Sample size and Power: Sample size: –http://www.dssresearch.com/toolkit/sscalc/size_a2.asphttp://www.dssresearch.com/toolkit/sscalc/size_a2.asp –http://www.dssresearch.com/toolkit/sscalc/size_p2.asphttp://www.dssresearch.com/toolkit/sscalc/size_p2.asp Power: –http://www.dssresearch.com/toolkit/spcalc/power_a2.asphttp://www.dssresearch.com/toolkit/spcalc/power_a2.asp –http://statpages.org/proppowr.htmlhttp://statpages.org/proppowr.html

18. Minimum Detectable Impacts: What are the smallest effects you will be able to detect given n and predicted S?

19. Adjustments to impact assessment: Regression adjusted impacts decrease S and increase power by controlling for “noise” using baseline characteristics Multiple Comparisons are a problem because randomness happens if you look long enough! –MDRC picked “primary outcomes” –Use adjustments to account for multiple comparisons

20. Showing estimated impacts over time in program

21. Who’s in the average? “No shows” in treatment group didn’t get any services –Unlikely to be similar to “shows” –If drop, then may overstate potential impacts “Intent to Treat” outcomes include outcomes for no-shows “Treatment on the Treated” outcomes do not include no-shows Non-response to follow-up surveys could bias impact assessments –Use administrative data available for all for key outcomes –Put resources into follow up to minimize non-response –Construct weights to make survey sample estimates comparable to baseline sample

22. Lessons from Summary: Randomization is hard Need to use power analysis to choose target sample sizes Even randomization may not give comparable baseline characteristics Regression may increase comparability and precision Worry about who we have outcome information for (both control and treatment)