Evaluation II Outcomes Cost benefit & effectiveness

Outcomes  Dependent on the goals  Multiple goals, do they compete  Multiple stakeholders  Development of measurable outcomes that reflect program goals  If the measurements do not reflect the goals, then they are not useful

Outcomes  A goal may have one than one measure (indicator)  In evaluation research there are often multiple measures of several outcomes  Need to identify the IV and then multiple measures of the DV

Example: Manhattan Bail Project  Use of a prediction instrument to release arrestees on their own recognizance  Goals: relieve jail overcrowding, presence at trial, protect public safety, decrease costs  Measures: % ROR  % that appear at hearing

Example  % released ROR that abscond or commit new crimes (public safety)  Costs of the program minus the amount saved by successful RORs  Might be successful in one sense but not in another

Evaluation designs  Experiment: randomly assign subjects to different treatment groups, i.e., intervention/no intervention  Example: random assignment to EM with home detention and home detention without EM (control)  Logic of experiments

Experiments  Must ensure that the treatment is delivered, i.e., is home detention enforced? Is the EM equipment working properly? This is a matter of monitoring, and must be established before evaluating treatment effects  What is the DV? How will outcomes be measured? Rearrest rates? Other? Depends on goals

Experiments  Treatment integrity and program delivery are important—if they are problematic, then outcomes cannot be evaluated  Monitoring, making sure that program is being followed

Quasi-experiments  Random assignment is often not possible  Sometimes feasible for programs/services, but may be restricted for legal or political reasons  Random assignment typically not possible for evaluation of laws or some policies

Quasi-experiments  Due process, equal protection issues  Example: drug court participants could probably not be randomly assigned, must look at the group as a whole  Violates the logic of experiments—is any change in the DV due to the IV, or to something else (extraneous)?

Quasi-experiments  Ex post evaluations: an evaluation sometime after a program has gone into effect  Full coverage programs—everyone is subjected to some treatment/IV, there is no control  Example: graduated license law in Missouri (how to evaluate)

Quasi-experiments  Treatment units such as neighborhoods  IV in some areas, not in areas  Uncontrolled variables, such as the actions of individuals within the neighborhoods  Target and comparison areas  Usually fear of crime and crime rates

Quasi-experiments  Non-equivalent control group designs  No random assignment  Attempt to select an experimental and comparison group that are as similar as possible  Can assess this by measuring both groups on important variables and determining if there are differences

Quasi-experiments  Time series designs  Interrupted time series designs  Measure before and after an intervention is introduced  Repeat in other settings, other times, other subjects  Single series design—introduce and then remove the intervention

Quasi-experiments  Time series design with switching replications  Combination of the two—can do a single series design in one location and then introduce it in another  Example: helmet laws lead to reduction in motorcycle theft  Checked for displacement to cars, bikes

Cost benefit  Costs of the intervention compared to the money potentially saved  Manhattan bail project, costs of the program compared to money saved when the arrestees are not jailed (costs of jail for estimated specified periods of time)  Other potential savings (wages, taxes)

Cost benefit  Sometimes the cost benefits may be calculated on a long term basis  Unfortunately the government is not necessarily responsive to this argument, as they operate on a more short term basis  Example: costs of corrections (prisons) vs community corrections

Cost benefit  Boot camp evaluations: aside from other outcomes, (i.e., recidivism), do boot camps save money?)  Are inmates deterred/rehabilitated/punished in a shorter period of time?  More inmates could be processed at a faster, less expensive rate

Cost benefit  In this example, note the potential dangers of net widening  Is money saved by performing the intervention in the long run?  Major argument that advocates of prevention programs must use in order to sell programs

Cost effectiveness  Costs of the program compared to the non-monetary outcomes  A program might be cheaper but outcomes might not be better (or even worse, or with unanticipated unwanted side effects)  Community corrections: cheaper, recidivism rates not different

Cost effectiveness  If recidivism is not higher, is the program effective? What are the other issues?  Punitiveness or deterrence (individual, general)? Effect on inmates who avoid prisonization? These are potential outcomes that are nonmonetary in nature

Cost effectiveness  There could be justice issues (i.e., if offenders who have committed murder are released because they are low risk)  Example of release of offenders who have “aged out” of crime  Weighing cost benefit and cost effectivess

Costs and benefits  How effective and what effects, must a program have to be “worth” it?  Value judgments  Political issues  Effects of restricting handguns  Mandatory jail sentences for abortion protestors

Costs and benefits  Policies concerning drugs, drug law sentences  Racial profiling  Local example of pornography  Does a program work—obviously there are stakeholders who want it to work  Many issues in evaluation