1. Does Government Work? Evaluating Programs and Policies Katherine C. Naff, Ph.D. January 22, 2002

2. Policies Often Respond to Problems Second hand smoke harms people’s health People are living in poverty because they don’t have the education required for a job Cities are plagued by jobless gang members and “antisocial youth”

3. Programs developed to achieve policy objectives Objective: Reduce second-hand smoke Program: Smoking cessation program Objective: Ensure all have access to education Program: Provide support to college students at risk for dropping out Objective: Help prepare gang members and antisocial youth for jobs Program: Finance tattoo removal

4. Government Performance and Results Act (GPRA) Requires federal agencies to: Set strategic goals Measure performance toward achieving those goals Publicly report on their progress

5. GPRA is to “…improve congressional decisionmaking by providing more objective information on achieving statutory objectives, and on the relative effectiveness and efficiency of federal programs and spending.” Answer the defining question: “Do the programs contribute to a real difference in the quality and security of citizens’ lives?” Shift focus from preoccupation with staffing and activity levels to a broader focus on the results or outcomes of federal programs Source: David Walker

6. Traditional performance measures Amount of money spent to support program Number of staff employed Number of tasks completed GPRA measures Outcomes or results

7. Example: OSHA Activity or “Output”: Inspect 2000 plants this year Outcome: Reduce workplace injuries by 20%

8. Example: Welfare Reform Activity or Output: Enroll 50 percent of welfare recipients in training programs Outcome: 50% of welfare recipients are in full-time jobs

9. Performance Measurement vs. Program Evaluation Performance measurement asks, “Are we achieving the goals we set, as shown by measurable outcomes?” Program evaluation asks, “Is the program responsible for the measurable outcomes we observe?”

10. Example: SSS-TRIO program Objective: To help “at risk”* students to stay in college until they earn their Bachelor’s degree Activities: orientation, tutoring, skills workshops, mandatory advising, guaranteed registration in remedial classes, financial aid *low income, first generation college, or individual with disabilities

11. Program theory Engaging students in this set of activities will increase the likelihood that they will graduate with a Bachelor’s degree, and that degree will enable them to “participate more fully in America’s economic and social life.”

12. Outcome Line* Activity Subobjective Outcome of interest Ultimate Outcome (“happiness”) *As described by Lawrence Mohr

13. SSS-TRIO Outcome Line OrientationTutoring Study Skills Workshops Commitment to program Improved grasp of concepts Improved study habits Better Grades Retained Graduate Fuller participation in America

14. Is SSS-TRIO achieving its purpose? Process evaluation: Is the program being implemented as intended? Outcome (impact) evaluation: Is the program achieving specified outcomes?

15. Process Measures # of students who attended orientation # of students who received tutoring # of students who participated in workshops

16. Outcome measures Subobjectives: % of program students who maintained GPA of 2.0 or better each year % of program students with GPAs under 3.0 who increased their GPAs by.2 after 4 semesters % of program students retained each year Outcome of interest % of program students who graduate

17. Expected Outcomes A minimum of 80% of participants will maintain a GPA of 2.0 or better each year. 75% of program participants with GPA’s under 3.0 will increase their GPAs by.2. A minimum of 65% of program participants will be retained each year. A minimum of 45% of participants will graduate each year.

18. Quantifying Program Impact Program Impact = Results – Counterfactual Counterfactual = Estimate of what results would have been if program had not been implemented

19. Example: SSS-TRI Program Impact = 45% graduation rate - 25% graduation rate Program Impact = increase in graduation rate by 20 percentage points

20. Estimating the counterfactual Previous experience External events Attrition Hawthorne effect Control group Selection bias Effects resulting from knowledge about other group

21. Other Questions Are we addressing the “right” problem? Are there other intervening variables? Are there alternative explanations? Is the program theory, as specified in the outcome line, correct?

22. Examining Linkages (formative evaluation) Skills workshop Improved study habits Graduation Skills workshop Graduation Improved study habits

23. How does evaluation compare to other forms of research? Both require Rigorous design Similar techniques for data collection and analysis Concern for issues of validity and reliability Competence and objectivity

24. How does evaluation compare to other forms of research? Evaluation Has “real world” consequences Takes place in a political environment Must deal with the march of time Often faces data and other resource constraints Is supposed to improve programs as well as assess their outcomes.

25. Summing up Programs are intended to achieve policy objectives, often to solve a problem Performance measurement tells us if we are getting results Program evaluation tells us if the program is giving us those results Evaluators must consider issues that might undermine our conclusions Program evaluation is rigorous research in a real world setting.

26. Excellent Sources Program Evaluation Lawrence Mohr, Impact Analysis for Program Evaluation (2 nd edition, 1995) Carol Weiss, Evaluation (2 nd edition, 1998) Steven Isaac and William B. Michael, Handbook in Research and Evaluation (3 rd edition, 1997) Performance Measurement Harry Hatry, et al. Urban Institute Press General Accounting Office (www.gao.gov)