1 General Elements in Evaluation Research

2 Types of Evaluations

3 Outcome (or impact) Evaluation One looks at what difference certain things (e.g., services and supports) make in outcome variables of interest. Outcomes typically include beneficial changes for those receiving services and fiscal and other effects for public entities.

4 Process Evaluation Typically, one looks at policies, procedures, and practices in terms of milestones, planned accomplishments, efficiency, and effectiveness. Process evaluations often relate to teamwork, project management, continuous improvement, and/or policy development.

5 Note Those who fund evaluations typically have a set of questions to which they want timely, relevant, and credible answers through systematic inquiry, using: –quantitative (counts; proportions; statistics) and/or –qualitative (observations, interviews) techniques.

6 Research Designs

7 Background Valued outcomes depend on a whole host of factors. Factors in “successful transitions” from school include: (1) person (2) experience in school; (3) physical and social environment; (4) paid people whose jobs include helping with transitions; (5) opportunities for postsecondary education and training; and (6) the local labor market.

8 Random Assignment Difficult to establish causality between an independent and a dependent variable. Threats to inferences are reduced if some of those interested in the services receive them, while others do not, and assignment to treatment and control groups is truly random. Such experimental designs typically reduce differences known to affect outcomes.

9 Random Assignment If there is “control of the experiment” so that “enhanced services” are only received by those in the experimental group, one may be more confident in assertions of likely causality. Such control, however, can be very difficult to achieve in natural settings. Enhanced services may inadvertently end up being provided to members of both groups.

10 Quasi-experimental Designs One may construct “matched comparison groups,” and follow both. Comparison groups should be similar in terms of factors known to be related to (predictive of) outcomes. The groups should be receiving similar services, except for the “enhanced services” being studied in terms of impact.

11 Quasi-experimental Designs Comparison group members should also be in similar schools receiving much the same services, and face similar postsecondary opportunities and labor market conditions. If these things are true, one may be able to “control statistically” for any remaining differences that make a difference. Reasonable inferences as to the apparent influence of the “enhanced services” can then be made.