Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact Study Presenters: Alan Werner, co-Principal Investigator Laura Peck, co-Principal Investigator Project Director: Gretchen Locke APPAM Conference November, 2013

Abt Associates | pg 2 Presentation Overview  What we know and don’t know about what works  The HPOG Impact Study  Strategies to “Get Inside the Black Box”  Q&A

Abt Associates | pg 3 Effects of Training: Progress to Date  It is well-established that vocational training and employment support programs for low-income individuals work –Strong experimental impact research supports claims of effectiveness of specific approaches –Best when program model tested in multiple sites and/or when program models tested head-to-head (e.g., HCD vs. LFA in NEWWS) –But, new program models within Career Pathways framework-based programs, and…

Abt Associates | pg 4 Challenges in Evaluating Training  Challenge has been to get inside the “black box” of program to develop findings more useful for policy and program design, for example: –What “parts” of a tested program are most responsible for impacts? –What accounts for variation in impacts across multiple program realizations? –What design and implementation strategies work better than others? –What works best for whom?

Abt Associates | pg 5 HPOG and Its Impact Evaluation  Career Pathways framework-based training for TANF and low-income individuals to pursue healthcare sector careers  HPOG-Impact is part of a rich research “portfolio” at ACF  Impact Evaluation involves an experimental design, with randomization of eligibles to control and treatment groups, with randomization to enhanced treatment in some locations

Abt Associates | pg 6 Study Sample and Data Collection  Sample size –Individuals: about 10,500 overall: 7,000 T; 3,500 C –Study sites: 38 study sites programs across 20 grantees –Planned variation sample (TBD) Peer support Emergency financial assistance Non-cash incentives  Data collection –At baseline (before RA), from PRS & supplement –Quarterly wage data (NDNH) –Follow-up surveys at 15 months post-randomization –Implementation study site visits –Grantee and other surveys from NIE

Abt Associates | pg 7 Research Questions HPOG-Impact will address the following questions: 1.What impacts do the HPOG programs as a group have on the outcomes of participants and their families? 2.To what extent do those impacts vary across selected subpopulations? 3. Which locally-adopted program components influence average impacts? 4. To what extent does participation in a particular component (or components) change the impacts experienced by individual trainees?

Abt Associates | pg 8 Research Questions HPOG-Impact will address the following questions: 1.What impacts do the HPOG programs as a group have on the outcomes of participants and their families? 2.To what extent do those impacts vary across selected subpopulations? 3. Which locally-adopted program components influence average impacts? 4. To what extent does participation in a particular component (or components) change the impacts experienced by individual trainees?

Abt Associates | pg 9 Sources of Variation Generating new evidence on the role of program components in generating impacts from HPOG sites will encompass:  Natural variation in program features across sites  Planned variation in certain sites where subsets of participants are randomized to gain access to a specific program enhancement in addition to the basic program  Person-to-person variation in components of the offered intervention actually received by individuals

Abt Associates | pg 10 Sources of Variation Generating new evidence on the role of program components in generating impacts from HPOG sites will encompass:  Natural variation in program features across sites  Planned variation in certain sites where subsets of participants are randomized to gain access to a specific program enhancement in addition to the basic program  Person-to-person variation in components of the offered intervention actually received by individuals

Abt Associates | pg 11 Intervention Components by Site and RA Sample: An Illustration Site PQRSN Type of Program Component: Naturally occurring, universalFFFFF Naturally occurring, varied–GEH– Randomized enhancement –E–– E Components Provided for Each Random Assignment Sample: C group sampleABCDL T1 sampleA, FB, F, GC, F, ED, F, HL, F T2 sample–B, F, G, E–--L, F, E

Abt Associates | pg 12 Sources of Variation Extracting the best information on the role of program components in generating impacts from HPOG sites will encompass:  Natural variation in program features across sites  Planned variation in certain sites where subsets of participants are randomized to gain access to a specific program enhancement in addition to the basic program  Person-to-person variation in components of the offered intervention actually received by individuals

Abt Associates | pg 13 Conceptually… When exposed to treatment… used program component (e.g., emergency assistance) achieved interim outcome (e.g., recognized credential) If exposed to treatment, would have… used program component (e.g., emergency assistance) achieved interim outcome (e.g., recognized credential)

Abt Associates | pg 14 Practically… Step 1  Step 1: Use baseline (exogenous) characteristics to predict subgroup membership –To capitalize on the internal validity of the experimental design, use symmetric identification of T and C subgroups  HPOG application –Use baseline variables… demographics, supplemental baseline Qs on efficacy, work preferences, barriers/needs –…to predict participation in various program components: used emergency assistance, supports, used child care support/services, accessed majority of available supports –…or to predict selected achieved/short-term outcomes (mediators): earned recognized credential, found healthcare job

Abt Associates | pg 15 ASPES: Steps 2 & 3  Step 2: Estimate impacts on predicted subgroups –Impact estimates are unbiased  Step 3: Convert estimated impacts for predicted subgroups to represent actual subgroups –Conversion rests (1) on an assumption of the homogeneity of impacts among those predicted to be in a subgroup; and (2) on the foundation of an experimental impact estimate

Further Information Molly Irwin Federal Project Officer, HPOG HHS/ACF/OPRE Alan Werner & Laura Peck Co-Principal Investigators Abt Associates Inc.