1. Evaluation Research and Problem Analysis
Chapter 10
Evaluation Research and Problem Analysis

2. Introduction
Evaluation Research: Refers to a research purpose rather than a specific method; seeks to evaluate the impact of interventions; if some result was produced
Problem Analysis: Designed to help public officials choose from alternative future actions
Policy Intervention: An action taken for the purpose of producing some intended result
Evidence-Based Policy: The actions of justice agencies are linked to evidence used for planning and evaluation

3. The Policy Process
Begins with a demand supporting a new course of action or opposition to existing policy
Policymakers consider ultimate goals and actions to achieve those goals
Outputs - The means to achieve desired goals
Impacts – Refer to basic question about what a policy seeks to achieve
If some policy is taken, then we expect some result

4. Linking the Process to Evaluation
Are policies being implemented as planned?
Are policies achieving their intended goals?
Evaluation seeks to link intended actions and goals of policy to empirical evidence that:
Policies are being carried out as planned (process evaluation)
Policies are having the desired effects (impact assessment)
Often conducted together

5. Getting Started
Learning policy goals is a key first step in doing evaluation research
Evaluability Assessment: “Pre-evaluation” – researcher determines whether requisite conditions are present
Support from relevant organizations
What goals and objectives are; how they are translated into program components
What kinds of records or data are available
Who has a direct or indirect stake in the program

6. Problem Formulation and Measurement
Different stakeholders often have different goals and views as to how a program should actually operate
Must clearly specify program goals – Desired outcomes
Create objectives – Operationalized statements
Definition and measurement –Starget/beneficiary population, decide between using current measures or creating new ones
Measure program contexts, outcomes, program delivery

7. Designs for Program Evaluation
Randomized Evaluation Designs: Avoids selection bias, allows assumption that groups created by random assignment are statistically equivalent; may not be suitable when agency or staff makes exceptions
Case Flow: Represents process through which subjects are accumulated into experimental and control groups
Treatment Integrity: Whether an experimental intervention is delivered as intended; ≈ reliability
Threatened by midstream changes in program

8. Conditions Requisite for Randomized Experiments
Staff must accept random assignment and agree to minimize exceptions to randomization
Case flow must be adequate to produce enough subjects in each group so that statistical tests will be able to detect significant differences in outcome measures
Experimental interventions must be consistently applied to treatment groups and withheld from control groups
Need equivalence prior to intervention, and ability to detect differences in outcome measures after intervention

9. Home Detention: Two Randomized Studies
Combining home detention with ELMO
Juvenile program paid less attention to delivering program elements and using ELMO info than adult
Difficult to maintain desired level of control over experimental conditions
Also difficult when more than one organization is involved
Randomization does not control for variation in treatment integrity and program delivery; utilize other methods

10. Quasi-Experimental Designs
No random assignment to Experimental and Control group
Often “nested” in experimental designs as backups
Lack built-in controls for selection & other Internal Validity threats
You must construct Experimental and Control groups as similar as possible

11. Quasi-Experimental Designs
Ex post evaluation: Conducted after experimental program has gone into effect
Full Coverage Programs: Sentencing Guidelines
Larger Treatment Units: Neighborhood crime prevention program
Interrupted time-series designs: Require attention to different issues because researchers cannot normally control how reliably the experimental treatment is actually implemented
Instrumentation, History, Construct Validity

12. Problem Analysis and Scientific Realism
Problem analysis, coupled with scientific realism, helps public officials use research to select and assess alternative courses of action
Realists suggest that similar interventions will have different outcomes in different contexts
Evaluators should search for mechanisms (IVs) acting in context (assorted intervening variables) to explain outcomes (DVs)
Appropriate in small-scale evaluations directed toward solving a particular problem in a specific context

13. Problem Analysis Problem Oriented Policing
Problem solving: A fundamental tool in problem-oriented policing
How-To-Do-It Guides: A general guide to crime analysis to support problem-oriented policing
Problem & Response Guides: Describe how to analyze very specific types of problems and what are known to be effective or ineffective responses

14. Auto theft in chula vista
Nanci Plouffe and Rana Sampson (2004) began their analysis of vehicle theft by comparing Chula Vista to other southern California cities
Theft rates tended to be higher for cities closer to the border
10 parking lots accounted for 25% of thefts & 20% of break-ins in the city
6 of the 10 lots were among the top 10 calls-for-service locations in Chula Vista
Auto theft hot spots also tended to be hot spots for other kinds of incidents

15. Other Applications of Policy Analysis
Space-and Time-Based Analysis: increased prevalence due to technological advances
Crime maps usually represent at least four different things:
one or more crime types;
space or area;
some time period; and
some dimension of land use, usually streets
Problem solving tools and processes
Strategic Approaches to Community Safety Initiatives (SACSI)

16. Block Groups with > 13% Abandoned Buildings
1998
1997
1996
1995
Block Groups with > 13% Abandoned Buildings

17. Political Context of Applied Research
Different stakeholder interests can produce conflicting perspectives on evaluations
Researcher must identify stakeholders & perspectives
Educate stakeholders on why evaluation should be conducted
Explain that applied research is used to determine what works and what does not
Political concerns & ideology may color evaluation; be careful

18. Why Results Are Ignored
Implications may not be presented in a way that non-researchers can understand
Results sometimes contradict deeply held beliefs
Vested interest in a program