1. Program Evaluation It’s Not Just for OMB Anymore…

2. 2 Questions Think about your state What has your public investment been in workforce development in your state in the last five years? How many people participated in what services? What has been the net impact of participation in the services? What has been the return on this investment?

3. 3 It’s Not Just A Good Idea, It’s the Law [PL 105-220] WIA Chapter 6, Section 136(e)(1) “Using funds made available under [WIA], the State, in coordination with local boards, shall conduct ongoing evaluation … of workforce investment activities …”

4. 4 Why Evaluate? “ … in order to promote, establish, implement, and utilize methods for continuously improving the activities … to achieve high-level performance … Evaluation ensures that states are getting the highest possible return on the investment of public funds If you don’t know what is working and what isn’t, your performance will be driven solely by happenstance and not by design –The best defense IS NOT accidental offense

5. 5 What Should Evaluations of the Workforce Investment System Include? Analysis of Customer Feedback –Not simply the collection of the data for WIA reporting purposes Outcome Measures Process Measures in the Workforce Investment System

6. 6 Fundamental Evaluation Questions What is the program doing; what are the activities? What are the customers’ reactions? –Participants and Employers What are the results? How are the results achieved? What is the return on investment?

7. 7 Isn’t monitoring enough? Monitoring determines the level of compliance with administrative policies and procedures designed to interpret the legislative intent of the program. Monitoring does not measure impact, process or outcomes.

8. 8 Okay, how about reviews? Review determines the extent to which program plans are consistent with legislative intent of the program Compliance Reviews do not measure impact, process or outcomes Reviews are primarily designed to ensure compliance with laws and guidance and are not an evaluation of the program

9. 9 Performance Reporting on 17 Measures Must be Sufficient… Performance Measurement determines the extent to which immediate program outputs and short-term outcomes meet performance expectations established and consistent with the legislative intent. Just because a state reports high performance levels, this does not mean that they are running a program worth emulating. Only an evaluation can make that determination. Success on performance measures under WIA could also reflect negotiation skills and not strong performance design

10. 10 Evaluation is the Only Way Evaluation determines through longitudinal-type studies the extent to which a program is achieving its broader legislative intent in terms of producing the expected effects on the individuals or other entities for whom the program was created. None of the WIA measures provide information on UI savings, the return on investment or business services.

11. 11 Types of Evaluation There are seven (7) types of evaluations that satisfy the legislative intent of the Workforce Investment Act: 1.Program Evaluation 2.Assessment of Need 3.Assessment of Program Theory and Design 4.Process/Implementation Evaluation 5.Outcome Evaluations 6.Impact Evaluations Defined by GAO 7.Cost-Benefit and Cost-Effectiveness Analysis

12. 12 1. Program Evaluation The systematic collection, analysis, and interpretation of information that answers questions about the efficiency and effectiveness of program implementation and outcomes, using the principles and methods of social science research.

13. 13 2. Assessment of Need Considers the assumptions that Employment and Training programs utilize in regards to problems they address for their target population. Problems would be determined through existing data sources, surveys, and forecasts.

14. 14 3. Assessment of Program Theory and Design Compares intervention specified in the program with the needs the program is expected to address, and determines whether the program plan can reasonably expect to improve the problem.

15. 15 4. Process/Implementation Evaluation Assesses the extent to which a program is operating as it was intended. This type of evaluation can provide a linkage between the process and program outcomes (i.e. impact evaluations).

16. 16 5. Outcome Evaluations Focuses on intended and unintended outputs and outcomes to judge program effectiveness; may also employ a process evaluation to establish a complete picture of how the outcomes are produced.

17. 17 6. Impact Evaluations: Defined by GAO One that assesses the net effect of a program by comparing program outcomes with an estimate of what outcomes would have happened in the absence of the program. The “gold” standard is a random assignment study to assign individuals (or …) to one of two groups: a control and an experimental group.

18. 18 7. Cost-Benefit and Cost-Effectiveness Analysis Compares program outputs and outcomes (i.e. outcome evaluation) with the “costs” to produce them. Both measure efficiency of a program using dollars expended; however, Cost Benefit Analysis aims at identifying all relevant costs and benefits of a program. Cost Effectiveness Analysis compares programs with similar goals and compares their costs.

19. 19 Examples Massachusetts Washington

20. 20

21. 21

22. 22

23. 23

24. 24

25. 25

26. 26 Washington

27. 27

28. 28 Take Away Points Start with a question Let question(s) drive the choice of evaluation design/study Remember that often mixed methods will be needed Avail yourself of resources out there Start small to make your point (“proof of concept” studies) Maximize secondary analyses of available administrative data It can be done – It is not an unfunded mandate

29. 29 What are you doing? Audience discussion on evaluation activities

30. 30 Websites/Resources Government Accountability Office (GAO) http://www.gao.gov/special.pubs/erm.html http://www.gao.gov/special.pubs/erm.html Social Policy and Research Associates (SPRA) http://www.spra.com/publications.shtml http://www.spra.com/publications.shtml Mathematica http://www.mathematica-mpr.com http://www.mathematica-mpr.com Environmental Protection Agency (EPA) http://www.epa.gov/evaluate/workshop/index.htm http://www.epa.gov/evaluate/workshop/index.htm Sage Associates http://www.sageways.com/home/et_types.cfm http://www.sageways.com/home/et_types.cfm Commonwealth Corporation – Center for Research and Evaluation http://www.commcorp.org/cre/index.html http://www.commcorp.org/cre/index.html

31. 31 Presenter Information Timothy Theberge ETA - Region 1 617-788-0139 theberge.timothy@dol.gov Johan Uvin Director of Adult Literacy RI Dept. of Elementary and Secondary Education 401-222-8950 johan.uvin@ride.ri.gov