Presentation is loading. Please wait.

Presentation is loading. Please wait.

Evaluation Research and Problem Analysis. o Evaluation Research: Refers to a research purpose rather than a specific method; seeks to evaluate the impact.

Similar presentations

Presentation on theme: "Evaluation Research and Problem Analysis. o Evaluation Research: Refers to a research purpose rather than a specific method; seeks to evaluate the impact."— Presentation transcript:

1 Evaluation Research and Problem Analysis

2 o Evaluation Research: Refers to a research purpose rather than a specific method; seeks to evaluate the impact of interventions; if some result was produced o Problem Analysis: Designed to help public officials choose from alternative future actions o Policy Intervention: An action taken for the purpose of producing some intended result o Evidence-Based Policy: The actions of justice agencies are linked to evidence used for planning and evaluation

3 o Begins with a demand supporting a new course of action or opposition to existing policy o Policymakers consider ultimate goals and actions to achieve those goals o Outputs - The means to achieve desired goals o Impacts – Refer to basic question about what a policy seeks to achieve o If some policy is taken, then we expect some result

4 o Are policies being implemented as planned? o Are policies achieving their intended goals? o Evaluation seeks to link intended actions and goals of policy to empirical evidence that:  Policies are being carried out as planned (process evaluation)  Policies are having the desired effects (impact assessment) o Often conducted together

5 o Learning policy goals is a key first step in doing evaluation research o Evaluability Assessment: “Pre-evaluation” – researcher determines whether requisite conditions are present  Support from relevant organizations  What goals and objectives are; how they are translated into program components  What kinds of records or data are available  Who has a direct or indirect stake in the program

6 o Different stakeholders often have different goals and views as to how a program should actually operate o Must clearly specify program goals – Desired outcomes o Create objectives – Operationalized statements o Definition and measurement –Starget/beneficiary population, decide between using current measures or creating new ones o Measure program contexts, outcomes, program delivery

7 o Randomized Evaluation Designs: Avoids selection bias, allows assumption that groups created by random assignment are statistically equivalent; may not be suitable when agency or staff makes exceptions o Case Flow: Represents process through which subjects are accumulated into experimental and control groups o Treatment Integrity: Whether an experimental intervention is delivered as intended; ≈ reliability  Threatened by midstream changes in program

8 o Staff must accept random assignment and agree to minimize exceptions to randomization o Case flow must be adequate to produce enough subjects in each group so that statistical tests will be able to detect significant differences in outcome measures o Experimental interventions must be consistently applied to treatment groups and withheld from control groups o Need equivalence prior to intervention, and ability to detect differences in outcome measures after intervention

9 o Combining home detention with ELMO o Juvenile program paid less attention to delivering program elements and using ELMO info than adult  Difficult to maintain desired level of control over experimental conditions  Also difficult when more than one organization is involved o Randomization does not control for variation in treatment integrity and program delivery; utilize other methods

10 o No random assignment to Experimental and Control group o Often “nested” in experimental designs as backups o Lack built-in controls for selection & other Internal Validity threats o You must construct Experimental and Control groups as similar as possible

11 o Ex post evaluation: Conducted after experimental program has gone into effect o Full Coverage Programs: Sentencing Guidelines o Larger Treatment Units: Neighborhood crime prevention program o Interrupted time-series designs: Require attention to different issues because researchers cannot normally control how reliably the experimental treatment is actually implemented  Instrumentation, History, Construct Validity

12 o Problem analysis, coupled with scientific realism, helps public officials use research to select and assess alternative courses of action o Realists suggest that similar interventions will have different outcomes in different contexts o Evaluators should search for mechanisms (IVs) acting in context (assorted intervening variables) to explain outcomes (DVs) o Appropriate in small-scale evaluations directed toward solving a particular problem in a specific context

13 o Problem Oriented Policing  Problem solving: A fundamental tool in problem- oriented policing o How-To-Do-It Guides: A general guide to crime analysis to support problem-oriented policing o Problem & Response Guides: Describe how to analyze very specific types of problems and what are known to be effective or ineffective responses

14 o Nanci Plouffe and Rana Sampson (2004) began their analysis of vehicle theft by comparing Chula Vista to other southern California cities  Theft rates tended to be higher for cities closer to the border  10 parking lots accounted for 25% of thefts & 20% of break-ins in the city  6 of the 10 lots were among the top 10 calls-for-service locations in Chula Vista  Auto theft hot spots also tended to be hot spots for other kinds of incidents

15 o Space-and Time-Based Analysis: increased prevalence due to technological advances o Crime maps usually represent at least four different things: or more crime types; or area; 3.some time period; and 4.some dimension of land use, usually streets o Problem solving tools and processes  Strategic Approaches to Community Safety Initiatives (SACSI)

16 Block Groups with > 13% Abandoned Buildings

17 o Different stakeholder interests can produce conflicting perspectives on evaluations o Researcher must identify stakeholders & perspectives o Educate stakeholders on why evaluation should be conducted o Explain that applied research is used to determine what works and what does not o Political concerns & ideology may color evaluation; be careful

18 o Implications may not be presented in a way that non-researchers can understand o Results sometimes contradict deeply held beliefs o Vested interest in a program

Download ppt "Evaluation Research and Problem Analysis. o Evaluation Research: Refers to a research purpose rather than a specific method; seeks to evaluate the impact."

Similar presentations

Ads by Google