Presentation is loading. Please wait.

Presentation is loading. Please wait.

Evaluation Research and Problem Analysis

Similar presentations

Presentation on theme: "Evaluation Research and Problem Analysis"— Presentation transcript:

1 Evaluation Research and Problem Analysis
Chapter 10 Evaluation Research and Problem Analysis

2 Introduction Evaluation Research: Refers to a research purpose rather than a specific method; seeks to evaluate the impact of interventions; if some result was produced Problem Analysis: Designed to help public officials choose from alternative future actions Policy Intervention: An action taken for the purpose of producing some intended result Evidence-Based Policy: The actions of justice agencies are linked to evidence used for planning and evaluation

3 The Policy Process Begins with a demand supporting a new course of action or opposition to existing policy Policymakers consider ultimate goals and actions to achieve those goals Outputs - The means to achieve desired goals Impacts – Refer to basic question about what a policy seeks to achieve If some policy is taken, then we expect some result

4 Linking the Process to Evaluation
Are policies being implemented as planned? Are policies achieving their intended goals? Evaluation seeks to link intended actions and goals of policy to empirical evidence that: Policies are being carried out as planned (process evaluation) Policies are having the desired effects (impact assessment) Often conducted together

5 Getting Started Learning policy goals is a key first step in doing evaluation research Evaluability Assessment: “Pre-evaluation” – researcher determines whether requisite conditions are present Support from relevant organizations What goals and objectives are; how they are translated into program components What kinds of records or data are available Who has a direct or indirect stake in the program

6 Problem Formulation and Measurement
Different stakeholders often have different goals and views as to how a program should actually operate Must clearly specify program goals – Desired outcomes Create objectives – Operationalized statements Definition and measurement –Starget/beneficiary population, decide between using current measures or creating new ones Measure program contexts, outcomes, program delivery

7 Designs for Program Evaluation
Randomized Evaluation Designs: Avoids selection bias, allows assumption that groups created by random assignment are statistically equivalent; may not be suitable when agency or staff makes exceptions Case Flow: Represents process through which subjects are accumulated into experimental and control groups Treatment Integrity: Whether an experimental intervention is delivered as intended; ≈ reliability Threatened by midstream changes in program

8 Conditions Requisite for Randomized Experiments
Staff must accept random assignment and agree to minimize exceptions to randomization Case flow must be adequate to produce enough subjects in each group so that statistical tests will be able to detect significant differences in outcome measures Experimental interventions must be consistently applied to treatment groups and withheld from control groups Need equivalence prior to intervention, and ability to detect differences in outcome measures after intervention

9 Home Detention: Two Randomized Studies
Combining home detention with ELMO Juvenile program paid less attention to delivering program elements and using ELMO info than adult Difficult to maintain desired level of control over experimental conditions Also difficult when more than one organization is involved Randomization does not control for variation in treatment integrity and program delivery; utilize other methods

10 Quasi-Experimental Designs
No random assignment to Experimental and Control group Often “nested” in experimental designs as backups Lack built-in controls for selection & other Internal Validity threats You must construct Experimental and Control groups as similar as possible

11 Quasi-Experimental Designs
Ex post evaluation: Conducted after experimental program has gone into effect Full Coverage Programs: Sentencing Guidelines Larger Treatment Units: Neighborhood crime prevention program Interrupted time-series designs: Require attention to different issues because researchers cannot normally control how reliably the experimental treatment is actually implemented Instrumentation, History, Construct Validity

12 Problem Analysis and Scientific Realism
Problem analysis, coupled with scientific realism, helps public officials use research to select and assess alternative courses of action Realists suggest that similar interventions will have different outcomes in different contexts Evaluators should search for mechanisms (IVs) acting in context (assorted intervening variables) to explain outcomes (DVs) Appropriate in small-scale evaluations directed toward solving a particular problem in a specific context

13 Problem Analysis Problem Oriented Policing
Problem solving: A fundamental tool in problem-oriented policing How-To-Do-It Guides: A general guide to crime analysis to support problem-oriented policing Problem & Response Guides: Describe how to analyze very specific types of problems and what are known to be effective or ineffective responses

14 Auto theft in chula vista
Nanci Plouffe and Rana Sampson (2004) began their analysis of vehicle theft by comparing Chula Vista to other southern California cities Theft rates tended to be higher for cities closer to the border 10 parking lots accounted for 25% of thefts & 20% of break-ins in the city 6 of the 10 lots were among the top 10 calls-for-service locations in Chula Vista Auto theft hot spots also tended to be hot spots for other kinds of incidents

15 Other Applications of Policy Analysis
Space-and Time-Based Analysis: increased prevalence due to technological advances Crime maps usually represent at least four different things: one or more crime types; space or area; some time period; and some dimension of land use, usually streets Problem solving tools and processes Strategic Approaches to Community Safety Initiatives (SACSI)

16 Block Groups with > 13% Abandoned Buildings
1998 1997 1996 1995 Block Groups with > 13% Abandoned Buildings

17 Political Context of Applied Research
Different stakeholder interests can produce conflicting perspectives on evaluations Researcher must identify stakeholders & perspectives Educate stakeholders on why evaluation should be conducted Explain that applied research is used to determine what works and what does not Political concerns & ideology may color evaluation; be careful

18 Why Results Are Ignored
Implications may not be presented in a way that non-researchers can understand Results sometimes contradict deeply held beliefs Vested interest in a program

Download ppt "Evaluation Research and Problem Analysis"

Similar presentations

Ads by Google