Presentation on theme: "Chapter 6: Program-Oriented Approaches"— Presentation transcript:
1 Chapter 6: Program-Oriented Approaches Presentation by Jay Kerstetter, Amanda Brown,Jody Yoos, & Jane LightnerJ.K.
2 Orienting Questions: #1 What are the key concepts of the objectives-oriented evaluation approach? #2 How has this approach influenced evaluation?What is it?Objectives-oriented evaluation approach helps determine whether some or all of the program objectives are achieved and, if so, how well they are achieved.Evaluators may work with stakeholders to establish if program objectives are met.Information from this approach can assist with deciding to maintain, terminate, or change approaches within the program.J.K.
3 Tylerian Evaluation Approach 1. Ralph Tyler is credited with initiating this approach in the 1930’s2. Tyler began to formulate his views on education and evaluation.3. His approach included the following steps:Establish goals or objectivesClassify the goals or objectivesDefine objectives in behavioral termsFind situations in which achievement of objectives can be shownDevelop or select measurement techniquesCollect performance dataCompare performance data with behaviorally stated objectives4. This approach was readily adoptable by evaluators and had great influence on evaluation theorists.Tylerian Evaluation ApproachJ.K.
4 Provus’s Discrepancy Evaluation Model: Developed by Malcolm Provus; viewed evaluation as a continuous information management process.Provus stemmed key characteristics of his proposal from Tyler.Provus viewed evaluation as a process.This process was called DEM; Discrepancy Evaluation Model, which are broken into four developmental stages.DefinitionInstallationProcessProductCost-benefit analysis (optional)The DEM was designed to facilitate the development of programs in large public school systems and later applied to state evaluations by federal bureau.The DEM was one of the earliest approaches to evaluation and elements of it can still be found today in many evaluation.J.K.
5 Orienting Question #3: How is the objective-oriented evaluation approach used today? Standards-based testingAccountability in educationPerformance monitoring systems used in many government programsMany refinements to the system since the 1930’sJ.Y.
6 Ralph W. Tyler: Tylerian Evaluation Approach Influenced the Elementary and Secondary Education Act (ESEA) of 1965, first act to require evaluation of educational programsStarted the National Assessment of Educational Progress (NAEP), only way to examine educational achievement of al 50 states.J.Y.
7 Malcolm Provus: Provus’s Discrepancy Evaluation Model Approach evaluated in Pittsburgh public schoolsViewed evaluation as a continuous information- management process designed to serve as “the watch-dog of program-management” and the “handmaiden of administration in the management of program development through sound decision making”Malcolm Provus: Provus’s Discrepancy Evaluation ModelJ.Y.
8 Orienting Question #4a: How are Logic Models used in evaluation? Developed as an extension of objectives- oriented evaluationDesigned to fill in those steps between the program and its objectivesProgram planners/evaluatorIdentify programInputsActivitiesOutputs (immediate program impacts)Outcomes (long-term objectives/goals)J.Y.
9 Today Logic Models are used in program planning/evaluation Help program staff articulate /discuss how program might achieve goalsWhat elements are important to evaluate at any given timeBuild internal evaluation capacity or think in an evaluative wayExample Companies:United way of AmericaWK Kellogg FoundationAnnie E Casey FoundationJ.Y.
10 Orienting Question #4b: How are Program Theories used in evaluation? Theory-Based evaluationIs used by evaluators to gain a better understanding of the programBetter define the evaluation questions the study should addressTo aid their choices of what concepts to measure and when to measure themTo improve their interpretation of results and their feedback to stakeholders to enhance use.Orienting Question #4b: How are Program Theories used in evaluation?J.Y.
11 Steps in Theory Based Evaluation Engage Relevant stakeholderDevelop a first draft of program theory evaluator or evaluation teamPresent draft to stakeholders for further discussion, reaction, and inputConduct a plausibility checkCommunicate findings to key stakeholdersProbe arrows for model specificityFinalize program impact theoryJ.Y.
12 Theory Driven Evaluation Work with stakeholders to identify key questions to be answered in the evaluation and the appropriate designs and methods for answering those questionsEmphasis is on testing the program modelProvide guidance as to what to measure and when to measure itThe selection of the evaluation questions to be addressed depend on the stage of the program and what stakeholders hope to learnProvides evaluator with critical information that can be used throughout the evaluationJ.Y.
13 Orienting Question #5: How do theory-based evaluation and objectives approaches differ? The objectives are identified by group looking for the evaluation.Purpose of activity specified and then those purposes are being acheived.Individual credited- Ralph W. TylerTheory-BasedThe evaluator discusses goals, basics, and objectives of program w/the stakeholders.The evaluator decides how the program should work and then sees if it works that way.Huey Chen and Leonard Bickman helped to develop approaches to theory based.Science based and quantitative.A.B.
14 Question #6: What are the strengths and limitations of program oriented evaluation approaches? Objective oriented – the simplicity of the concept and program makes it easy to understand, follow and possibly implement.Face validity – the evaluator is being held accountable for what is being asked to be evaluated. They want to see if what is currently happening or working based on their own goals and objectives for the program.A.B.
15 Theory oriented – the chance for the evaluator to partake in dialogue with the stakeholders is a strength of the program because it helps them to expand their knowledge on the topic.This then leads to the evaluator to have a clear understanding of the program so they know how to properly evaluate the program.Strengths (con’t)A.B.
16 Weaknesses:Objective oriented – the evaluator can have a single minded focus on the objectives which will cause them to overlook the complications, elements and factors contributing to that programs success or failure.This approach does not ask for the evaluator to gain an understanding of the context in which the program operates this could then be what is affecting the program’s success or failure.Evaluator could ignore the actual value of the objective. Since the evaluator is told what the objectives are they are not asked to evaluate whether that objective even fits the program.A.B.
17 Theory oriented – this like the objective oriented can cause the evaluator to ignore certain aspects of the program that are important because the evaluator is so focused on the theory of how it runs and not how it is actually running.Evaluators may ignore the needs or values of stakeholders involved with the program.This approach may also oversimplify the complexity of the program making it feel easier to evaluate than it really is because not all of the surrounding factors are accounted for in the process.Weaknesses (con’t)A.B.
18 Orienting Question # 7: What is goal-free evaluation? Rationale: “Goals should not be taken as givens.”Developed by Scriven (1972)---believes the most important function of a goal-free evaluation is to reduce the bias that occurs from knowing program goalsThus…increase objectivity in judging the program as a whole.J.L.
19 Orienting Question #8: What does it teach us about conducting an evaluation? Goals can act as “blinders” causing us to possibly miss the most important outcomes not related to the goals.Goal-free evaluation was proposed to primarily indentify the unanticipated side effects that an objectives-oriented evaluation might missJ.L.
20 Major Characteristics of a Goal-Free Evaluation 1. Evaluator purposely avoids becoming aware of program goals.2. Predetermined goals are not permitted o narrow the focus of the evaluation study.3. Goal-free evaluation focuses on actual outcomes rather than intended program outcomes.4. The goal-free evaluator has minimal contact with the program manager and staff.5. Goal-free evaluation increases the likelihood that unanticipated side effects will be noted.J.L.
21 Goal Directed + Goal-Free= can work together Internal Goal Evaluator vs.External Goal-Free EvaluatorGoal-directedHow well is program meeting goalsProvide information to administratorWhat does the program do?Looking at ALL the programs outcomes, intended or notGoal Directed + Goal-Free= can work togetherJ.L.
22 Information taken from Program Evaluation: Alternate Approaches and Practical Guidelines