Presentation on theme: "Summative Evaluation The Evaluation after implementation."— Presentation transcript:
Summative Evaluation The Evaluation after implementation
Involves _______ data CollectingAnalyzingSummarizing
For the purpose of Giving decision makers information on the effectiveness and efficiency of instruction
Effectiveness of Content Instruction solve the problem? Criterion created prior to evaluation? Was the criterion established in conjunction with the needs assessment?
Specifically Did learners achieve the objectives? Learners feeling about instruction? What were the costs? How much time did it take? Was instruction implemented as designed? What unexpected outcomes?
Alternative Approaches to Summative Evaluation ObjectivismSubjectivism
Objectivism Based on empiricism Answering questions on the bases of observed data Goal based and replicable, uses the scientific method
Subjectivism Employs expert judgment Includes qualitative methods observation and interviews evaluate content “Goal Free” evaluators haven’t a clue about the goals
Objectivism (limitations) Examine only a limited number of factors May miss critical effects
Subjectivism (limitations) Are not replicable Biased by idiosyncratic experiences, perspectives, or the people who do the evaluation May miss critical effects
Designer Role in Summative Evaluation? Somewhat controversial
Timing of Summative Evaluation? Not in the first cycle
Summary Diagram Formative Design Reviews Expert Reviews One-to-one Eval. Small Group Eval. Field Trials Ongoing Eval. Summative Determine Goals of the Evaluation Select Orientation Select Design Design or Select Evaluation Measures Collect Data Analyze Data Report Results
Goals of the Evaluation What decisions must be made? What are the best questions? How practical is it to gather data? Who wants the answer to a question? How much uncertainty?
Orientation of Evaluation Goal-based or goal-free A middle ground? Quantitative or qualitative appropriate? Experimental or naturalistic approach?
Select Design of Evaluation Describes what data to collect When the data will be collected And under what conditions Issues to consider: How much confidence must we have that the instruction caused the learning? (internal validity) How important is the generalizability? (external validity) How much control do we have over the instructional situation?
Design or Select Evaluation Measures Payoff outcomes Is the problem solved? Costs avoided Increased outputs Improved quality Improved efficiency
Design or Select Evaluation Measures (2) Learning Outcomes Use instrument you’ve already developed for the summative evaluation But measure the entire program
Design or Select Evaluation Measures (3) Attitudes Rarely the primary payoff goals Ask about learner attitudes toward learning instructional materials subject matter Indices of appeal attention, likeableness, interest, relevance, familiarity, credibility, acceptability, and excitement
Design or Select Evaluation Measures (4) Level of Implementation degree to which the instruction was implemented Costs Cost-feasibility Cost-effectiveness
Alternative Designs Instruction then posttest Pretest then instruction then posttest
The Report Summary Background Needs assessment, audience, context, program description Description of evaluation study Purpose of evaluation, evaluation of the design, outcomes measured, implementation measures, cost- effectiveness info., analysis of unintentional outcomes
The Report (continued) Results Outcomes, implementation, cost- effectiveness info., unintentional outcomes Discussion causal relationship between program & results Limitation of study Conclusion & Recommendations
Summary Summative evaluation is after implementation Limitations of subjective and objective evaluation What to include in the report