“Measuring” Policies & Programs Program Inputs Program Outputs Program Impact
Policy Inputs Government resources allocated to address a public “problem –Agency personnel assigned (FTEs) –Funds allocated by legislature
Policy Outputs Indicators of government activities in carrying out policy –School Vouchers # of applications for vouchers # of vouchers distributed # of students switching schools # of “failing schools” reinvesting in facilities, teachers, etc. –DARE # of children in the program % of school districts participating –Acid Rain # of SO 2 scrubbers installed # of inspections carried out
Policy Impacts Degree to which the policy is addressing the problem Amount of the problem “solved” –School Vouchers Improved educational attainment among target population –Using vouchers –Not using vouchers Reduction in the % of “failing schools” –D.A.R.E Reduced drug experimentation Reduced substance use –Acid Rain Regeneration of aquatic ecosystems Reduced damage to acid-sensitive structures
Policy Impacts Unintended/Unexpected Impacts –Vouchers Impoverishment of public school districts Reduction in the educational attainment of “superior” schools –D.A.R.E Improved community-police relations
Process Evaluation Is the program carrying out the actions it is suppose to carry out? Measuring program outputs –Did the program make the public aware of the availability of vouchers? –Does the application process for vouchers encourage participation? –Is government distributing vouchers according to the rules?
Impact Evaluation Is the Program Working? –Is it reducing the problem as intended? Measuring Program Impact –Are test scores improving? Are fewer students dropping out? Voucher users Voucher non-users –To what degree are previously “failing schools” attempting to compete – reinvesting?
Uses for Evaluations Go/No Go Decisions Improve Practices & Procedures Expand Program Scope and Reach Allocate Resources among competing programs Test Policy Theory
Stages of Evaluation Program Goal Identification Construction of Impact Model (Policy Theory) Research Design Data Collection Analysis Report/Presentation
Program Goal Identification Who wants the evaluation? –Legislature –Agency –Policy/Program directors –Program staff –Clients –Interest Groups –Scholars
Research Design Isolating the causal effects of the program from other influences Measures of program impact –How would we know “change” if we saw it? Form of analysis –Case study –Focus groups Clients Program staff –Statistical analysis Data collection (sampling) requirements