Presentation is loading. Please wait.

Presentation is loading. Please wait.

Review: Introduction Define Evaluation Define Evaluation How do formal/informal evaluation differ? How do formal/informal evaluation differ? What are two.

Similar presentations


Presentation on theme: "Review: Introduction Define Evaluation Define Evaluation How do formal/informal evaluation differ? How do formal/informal evaluation differ? What are two."— Presentation transcript:

1 Review: Introduction Define Evaluation Define Evaluation How do formal/informal evaluation differ? How do formal/informal evaluation differ? What are two uses of evaluation in education? What are two uses of evaluation in education? What are the pros/cons of using an external evaluator? What are the pros/cons of using an external evaluator?

2 Alternative Approaches to Evaluation Dr. Suzan Ayers Western Michigan University (courtesy of Dr. Mary Schutten)

3 Alternative Approaches Stakeholders: individuals and groups who have a direct interest in, and may be affected by, evaluation; should be involved early, actively & continuously Stakeholders: individuals and groups who have a direct interest in, and may be affected by, evaluation; should be involved early, actively & continuously Program: activities that are provided on a continuing basis; typically what is evaluated Program: activities that are provided on a continuing basis; typically what is evaluated There are a variety of alternative, often conflicting, views of what evaluation is and how it should be carried out There are a variety of alternative, often conflicting, views of what evaluation is and how it should be carried out

4 Why so many alternatives? The way one views evaluation directly impacts the type of activities/methods used The way one views evaluation directly impacts the type of activities/methods used Origins of alternative models stem from differences in: Origins of alternative models stem from differences in: Philosophical & ideological beliefs Philosophical & ideological beliefs Methodological preferences Methodological preferences Practical choices Practical choices

5 Philosophical & Ideological Beliefs Epistemologies (philosophies of knowing) Epistemologies (philosophies of knowing) Objectivism (social science base of empiricism; replicate) Objectivism (social science base of empiricism; replicate) Subjectivism (experientially-based; tacit knowledge) Subjectivism (experientially-based; tacit knowledge) Pros/Cons of each? Pros/Cons of each? Principles for assigning value (parallel obj/subj) Principles for assigning value (parallel obj/subj) Utilitarian : focus on group gains (avg scores); greatest good for the greatest number Utilitarian : focus on group gains (avg scores); greatest good for the greatest number Intuitionist-pluralist : value is individually-determined Intuitionist-pluralist : value is individually-determined Room for both or are these dichotomous? Room for both or are these dichotomous? Philosophical purists are rare (impractical?) Philosophical purists are rare (impractical?) Choose the methods right for THAT evaluation Choose the methods right for THAT evaluation Understand assumptions/limitations of different approaches Understand assumptions/limitations of different approaches

6 Methodological Preferences Quantitative (numerical) Quantitative (numerical) Qualitative (non-numerical) Qualitative (non-numerical) Evaluation is a transdiscipline; crosses paradigms Evaluation is a transdiscipline; crosses paradigms “Law of the instrument” fallacy “Law of the instrument” fallacy With hammer/nails, all appears to need hammering With hammer/nails, all appears to need hammering Identify what is useful in each evaluation approach, use it wisely & avoid being distracted by approaches designed to deal w/ different needs Identify what is useful in each evaluation approach, use it wisely & avoid being distracted by approaches designed to deal w/ different needs

7 Practical Considerations Evaluators disagree whether/not intent of evaluation is to render a value judgment Evaluators disagree whether/not intent of evaluation is to render a value judgment Decision-makers or evaluator render judgment? Decision-makers or evaluator render judgment? Evaluators differ in views of evaluation’s political role Evaluators differ in views of evaluation’s political role Authority? Responsibility?  These dictate eval style Authority? Responsibility?  These dictate eval style Influence of evaluators’ prior experience Influence of evaluators’ prior experience Who should conduct the evaluation and nature of expertise needed to do so Who should conduct the evaluation and nature of expertise needed to do so Desirability (?) of having a wide variety of evaluation approaches Desirability (?) of having a wide variety of evaluation approaches

8 Classification Schema for Evaluation Approaches Conceptual approaches to evaluation, NOT techniques Objectives-oriented: focus on goals/objectives & degree to which they are achieved Objectives-oriented: focus on goals/objectives & degree to which they are achieved Management-oriented: identifying and meeting informational needs of decision makers Management-oriented: identifying and meeting informational needs of decision makers Consumer-oriented: generate information to guide product/service use by consumers Consumer-oriented: generate information to guide product/service use by consumers Expertise-oriented: use of professional expertise to judge quality of evaluation object Expertise-oriented: use of professional expertise to judge quality of evaluation object Participant-oriented: stakeholders centrally involved in process Participant-oriented: stakeholders centrally involved in process See figure 3.1 (p. 68) See figure 3.1 (p. 68)

9 Objectives-oriented Approach Purposes of some activity are specified and then evaluation focuses on the extent to which these purposes are achieved Purposes of some activity are specified and then evaluation focuses on the extent to which these purposes are achieved Ralph W. Tyler popularized this approach in education (criterion ref test) Ralph W. Tyler popularized this approach in education (criterion ref test) Tylerian models Tylerian models Metfessel & Michael’s paradigm (enlarged vision of alternative instruments to collect evaluation data) Metfessel & Michael’s paradigm (enlarged vision of alternative instruments to collect evaluation data) Provus’s Discrepancy Evaluation Model (agree on stds, det if discrepancy exists btwn perf/std, use discrepancy info to decide to improve, maintain, terminate program) Provus’s Discrepancy Evaluation Model (agree on stds, det if discrepancy exists btwn perf/std, use discrepancy info to decide to improve, maintain, terminate program) Logic models Logic models Determine long-term outcomes & backtrack to today Determine long-term outcomes & backtrack to today

10 Objectives-oriented Steps Establish broad goals or objectives tied to mission statement Establish broad goals or objectives tied to mission statement Classify the goals or objectives Classify the goals or objectives Define objectives in behavioral terms Define objectives in behavioral terms Find situations where achievement of objectives can be shown Find situations where achievement of objectives can be shown Select/develop measurement techniques Select/develop measurement techniques Collect performance data Collect performance data Compare data with behaviorally stated objectives Compare data with behaviorally stated objectives

11 Objectives-oriented Pros/Cons Strengths: simplicity, easy to understand, follow and implement; produces information relevant to the mission Strengths: simplicity, easy to understand, follow and implement; produces information relevant to the mission Weakness: can lead to tunnel vision Weakness: can lead to tunnel vision Ignores outcomes not covered by objectives Ignores outcomes not covered by objectives Neglects the value of the objectives themselves Neglects the value of the objectives themselves Neglects the context in which evaluation takes place Neglects the context in which evaluation takes place

12 Goal Free Evaluation This is the opposite of objectives-oriented evaluation, but the two supplement one another Purposefully avoid awareness of goals; should not be taken as given, goals should be evaluated Purposefully avoid awareness of goals; should not be taken as given, goals should be evaluated Predetermined goals not allowed to narrow focus of evaluation study Predetermined goals not allowed to narrow focus of evaluation study Focus on actual outcomes rather than intended Focus on actual outcomes rather than intended Evaluator has limited contact with program manager and staff Evaluator has limited contact with program manager and staff Increases likelihood of seeing unintended outcomes Increases likelihood of seeing unintended outcomes

13 Management-oriented Approach Geared to serve decision makers Geared to serve decision makers Identifies decisions administrator must make Identifies decisions administrator must make Collects data re: +/- of each decision alternative Collects data re: +/- of each decision alternative Success based on teamwork between evaluators and decision makers Success based on teamwork between evaluators and decision makers Systems approach to education in which decisions are made about inputs, processes, and outputs Systems approach to education in which decisions are made about inputs, processes, and outputs Decision maker is always the audience to whom evaluation is directed Decision maker is always the audience to whom evaluation is directed

14 CIPP Evaluation Model (Stufflebeam) Context Evaluation: planning decisions Context Evaluation: planning decisions Needs to address? Existing programs? Needs to address? Existing programs? Input Evaluation: structuring decisions Input Evaluation: structuring decisions Available resources, alternative strategies? Available resources, alternative strategies? Process Evaluation: implementing decisions Process Evaluation: implementing decisions How well is plan being implemented? Barriers to success? Revisions needed? How well is plan being implemented? Barriers to success? Revisions needed? Product Evaluation: recycling decisions Product Evaluation: recycling decisions Results? Needs reduced? What to do after program has ‘run its course’? Results? Needs reduced? What to do after program has ‘run its course’?

15 CIPP Steps Focusing the Evaluation Focusing the Evaluation Collection of Information Collection of Information Organization of Information Organization of Information Analysis of Information Analysis of Information Reporting of Information Reporting of Information Administration of Evaluation (timeline, staffing, budget etc…) Administration of Evaluation (timeline, staffing, budget etc…)

16 Context Evaluation Table 5.1 Objective: define institutional context, target population and assess their needs Objective: define institutional context, target population and assess their needs Method: system analysis, survey, hearings, interviews, diagnostic tests, Delphi technique (experts) Method: system analysis, survey, hearings, interviews, diagnostic tests, Delphi technique (experts) For deciding upon the setting to be served, the goals associated with meeting needs and objectives for solving problems For deciding upon the setting to be served, the goals associated with meeting needs and objectives for solving problems

17 Input Evaluation Objective: identify and assess system capabilities, procedural designs for implementing the strategies, budgets, schedules Objective: identify and assess system capabilities, procedural designs for implementing the strategies, budgets, schedules Method: inventory human and material resources, feasibility, economics via literature review, visit exemplary programs Method: inventory human and material resources, feasibility, economics via literature review, visit exemplary programs For selecting sources of support, solution strategies in order to structure change activities, provide basis to judge implementation For selecting sources of support, solution strategies in order to structure change activities, provide basis to judge implementation

18 Process Evaluation Objective: identify or predict defects in the process or procedural design, record/judge procedural events Objective: identify or predict defects in the process or procedural design, record/judge procedural events Method: monitoring potential procedural barriers, continual interaction with and observation of the activities of the staff Method: monitoring potential procedural barriers, continual interaction with and observation of the activities of the staff For implementing and refining the program design and procedure (a.k.a., process control) For implementing and refining the program design and procedure (a.k.a., process control)

19 Product Evaluation Objective: collect descriptions and judgments of outcomes and relate them to CIP, interpret worth/merit Objective: collect descriptions and judgments of outcomes and relate them to CIP, interpret worth/merit Methods: measure outcomes, collect stakeholder information, analyses of data Methods: measure outcomes, collect stakeholder information, analyses of data For deciding to continue, terminate, modify, or refocus an activity and to document the effects (whether intended or unintended) For deciding to continue, terminate, modify, or refocus an activity and to document the effects (whether intended or unintended)

20 Uses of Management-oriented Approaches to Evaluation CIPP has been used in school districts, state and federal government agencies CIPP has been used in school districts, state and federal government agencies Useful guide for program improvement Useful guide for program improvement Accountability Accountability Figure 5.1 (p. 94) Figure 5.1 (p. 94) Formative and summative aspects of CIPP Formative and summative aspects of CIPP

21 Management-oriented Pros/Cons Strengths: appealing to many who like rational, orderly approaches, gives focus to the evaluation, allows for formative and summative evaluation Strengths: appealing to many who like rational, orderly approaches, gives focus to the evaluation, allows for formative and summative evaluation Weaknessws: preference given to top management, can be costly and complex, assumes important decisions can be identified in advance of the evaluation Weaknessws: preference given to top management, can be costly and complex, assumes important decisions can be identified in advance of the evaluation

22 REVIEW/Qs Why are there so many alternative approaches to evaluation? Why are there so many alternative approaches to evaluation? What two conceptual approaches to evaluation did we discuss tonight? What are their +/-? What two conceptual approaches to evaluation did we discuss tonight? What are their +/-? Which, if either, of these approaches do you think will work for your evaluation object? Which, if either, of these approaches do you think will work for your evaluation object? Identify your most likely evaluation object Identify your most likely evaluation object


Download ppt "Review: Introduction Define Evaluation Define Evaluation How do formal/informal evaluation differ? How do formal/informal evaluation differ? What are two."

Similar presentations


Ads by Google