1. Carolyn J. Heinrich LaFollette School of Public Affairs, University of Wisconsin Peter R. Mueser University of Missouri, IMPAQ International, LLC, and IZA Kenneth R. Troske University of Kentucky and IZA Kyung-Seong Jeon University of Missouri Daver C. Kahvecioglu IMPAQ International, LLC November 2009 New Estimates of Public Employment and Training Program Net Impacts: A Nonexperimental Evaluation of the Workforce Investment Act Program

2. 1998 Workforce Investment Act (WIA)  Largest job-training program in the U.S. Implemented in 2000 in most states—replaced Job Training Partnership Act (JTPA)  Current annual federal budget: $3 billion Significantly lower relative to past public expenditures  Program implementation differs by state and local area Work-first emphasis, service sequencing, referrals, access to training

3. 3 WIA Adult Programs  Two primary adult programs serving: Disadvantaged workers, both unemployed and those in low-paying and unstable jobs Dislocated workers who have lost jobs or are slated to be laid off  Voluntary: participants recruited by local agency staff or referred by training providers– may have specific number of slots to fill Locally-determined eligibility standards

4. 4 WIA Service Sequencing  Core services - outreach, job search, placement aid, and labor market information  Intensive services - comprehensive assessments, individual employment plans, counseling and career planning  Training services – mostly occupational/vocational training, some on-the-job training Most training provided with voucher Similar services available to anyone in Employment Service offices (“Wagner- Peyser” services) Training time varies from a few months up to 2 years

5. Impact Analysis  Impact estimates for Adult program Dislocated Worker program Incremental impact of Training services vs. Intensive/Core services for these programs  Using state administrative data from 12 states: Connecticut, Indiana, Kentucky, Maryland, Mississippi, Missouri, Minnesota, Montana, New Mexico, Tennessee, Utah, Wisconsin

6. 6

7. 7 Comparison Groups  Comparison groups: UI claimants (9 states), Wagner- Peyser (3 states) Programs have substantial overlap WIA recipients receive the most meaningful services WIA Wagner-Peyser Services UI Claim Both are plausible comparison groups because they contain individuals with employment problems & those seeking assistance

8. 8 Measures Outcomes: Earnings & employment 16 qtrs Control variables:  Calendar quarter of program entry  Demographics: gender (exact match), age, education, race/ethnicity  Disability, veteran  Local labor market  Employment and earnings over the two years prior to program entry  Industry of most recent job  Prior program participation (WIA, UI, ES)

9. 9 Overview of Analysis WIA Program GroupSample Group AdultDWTreatmentComparison (a)(b)(c)(d) 1. XXWIAUI Claim or Wagner-Peyser 2. XXWIA Receiving UIUI Recipients 3. XXWIA TrainingWIA Core/Intensive

10. 10 Estimates  Estimates for states are combined  Weighted by number of WIA participants WIA participants entering in PY2003 and PY2004  Estimate of average program impact across 12 states Effect of the treatment (WIA) on the treated Comparison group defined by program contact in quarter of entry (UI claim or benefit, Wagner- Peyser service receipt, WIA participants who did not receive training services for training impact estimate)

11. Matching Procedures  Matching is within state, within gender, and (usually) within quarter of participation  Logit specification predicts propensity score Details of variable coding differ by state; approx. 100 individual characteristics, labor market experience and prior program participation, geographic area within state  Matching is many-to-one, using a caliper, with replacement (“radius matching”)  Standard errors use formula recommended by Imbens (2008) based on a conditional variance estimate

12. Basic Data: Demographics 12 WIA Adult WIA Dislocated Worker Comparison Group Sample size Unique individuals95,58063,5152,929,496 Units available for matching97,55264,0896,161,510 Demographic Mean Male0.4200.4820.585 Black0.4450.3300.171 Hispanic0.0310.0220.064 Age 32.7040.2439.59 Years of education 12.2712.5512.42 Summary Statistics for WIA Participants and Comparison Group: 12 States

13. Basic Data: Adult Program Earnings Comparison 13 Dip in earnings is small for Adult Program participants Dip in earnings is large for comparison program participants

14. Impact Estimates: Adult Program All Services, Females All Services Earnings Females 14 Quarterly earnings increase by $600 in 16 quarters after program entry (Mean quarterly earnings are $2000- $4000) Estimates in quarters 1-3 are probably upwardly biased Other estimates are realistic

15. Impact Estimates: Adult Program All Services, Males All Services Earnings Males 15 Again, quarter 1-3 estimates upwardly biased Other estimates are realistic Quarterly earnings increase by $400 for males

16. Impact Estimates: Adult Program Training, Females Training Earnings Females 16 Comparison group is WIA participants who don’t enter training

17. Impact Estimates: Adult Program Training, Males Training Earnings Males 17 Comparison group is WIA participants who don’t enter training

18. Impact Estimates: Adult Program All Services, High Training States All Services Earnings Females 7 High- Training States 18 Initial increase greater

19. Impact Estimates: Adult Program All Services, High Training States All Services Earnings Males 7 High- Training States 19 No initial decline

20. Summary of Impact Estimates: Adult Program  At face value, results imply strong immediate impact Aggressive initial counseling (plausible?) Selection into program may cause positive initial impact estimates Self-selection, counselor selection of those with good prospects  Training appears to be of some value, but there may be selection bias in results  Results for states with greater investments (“high training” states) differ No initial decline, greater growth over time in impacts

21. Basic Data: Displaced Worker Program Earnings Comparison 21 Dip in earnings is large for Dislocated Worker Program participants

22. Impact Estimates: Dislocated Worker Program All Services, Females All Services Earnings Females 22 Difference-in-difference estimates is much lower WIA entrants are advantaged relative to the comparison group: Causal impact is uncertain

23. Impact Estimates: Dislocated Worker Program All Services, Males All Services Earnings Males 23 Difference-in-difference estimates is much lower for males, too

24. Impact Estimates: Dislocated Worker Program Training, Females Training Earnings Females 24

25. Impact Estimates: Dislocated Worker Program Training, Males Training Earnings Males 25

26. Summary of Impact Estimates: Dislocated Worker Program  Pattern of results consistent with expectations that WIA services require time to produce impacts for workers facing serious difficulties in obtaining reemployment DW participant earnings do not reach earnings of comparable nonparticipants until more than two years after participation  Estimates also imply little incremental impact of training for DW participants Unlikely initial costs of training could be recouped  Results for 7 High-Training states show less evidence of bias (especially for women) but impacts on earnings even after training is complete appear minimal There does appear to be some impact on employment 26

27. Conclusion  Adult Program We observe long-term positive impacts of the WIA program Training also appears to be valuable  Dislocated Worker Program Selection on stable unobserved factors may induce positive bias in impact estimates Program long-run impacts difficult to gauge Long-term impacts appear minimal Training appears to have little long-run effect 27

28. Conclusion (continued)  Analysis shows both the potential benefits and the limitations of nonexperimental estimates based on administrative data Selection clearly affects results Patterns of estimates provide an indication of where impacts may be greatest Specification tests are important Comparison across control types and programs is useful 28