1. Developing Evaluation Instruments

2. Evaluations Formative – How are we doing? Summative – How did we do?
Confirmative – How are we still doing?

3. Formative Evaluation Even excellent plans and concepts can be improved
Gauge success by classroom program presentation
Informs the instructor how well instructional package is serving the objectives as it progresses

4. Formative Evaluation Program of instruction is evaluated through
Test results
Learner reactions/comments
General observation of learners during class
Subject-matter expert reviews
Colleague suggestions
Quality control of the development process

5. Formative Evaluation Questions that may be asked
Level of learning acceptable compared to objectives?
Learners using knowledge to perform skills?
Time element acceptable?
Activities appropriate?
Tests measuring objectives?

6. Summative Evaluation Measures success of major outcomes
Usually follows all instruction, projects and testing in a program

7. Summative Evaluation Measures
Learning efficiency (material mastered/time)
Program cost (Development and delivery)
Continuing expenses
Student comments/reactions/evaluations
Long-term benefit of program

8. Confirmative Evaluation
Although instruction initially effective, some problems appear over time
Tracks learner experiences over time to judge program’s validity
Relies upon numerous data-collection instruments (interviews, performance assessments, etc.)

9. Confirmative Evaluation
Questions to investigate
Do learners continue to perform correctly over time?
Do materials still meet original objectives?
How has technology and/or resources, trends, and attitudes changed since inception?
How to best meet clients’ needs over time?

10. Confirmative Evaluation
Changes needed in materials? What are the costs?
If instruction not working as well as before:
Should instruction continue as is?
Should it be revised?
Should it be terminated?
What might replace it?

11. Validity
The test assesses what it is supposed to measure – For example:
Performance tests assess processes and outcomes relating to skills or competencies
Course attitude surveys need to measure reactions to the course

12. Reliability The test will produce consistent results whenever used
More questions relating to specific objectives = more reliable test
Standardized delivery of test
Scoring methods – the less subjective, the better

13. Pretesting
Judges the learner’s preparation to study the material (course or topic)
Determines competencies already mastered

14. Testing for Prerequisites
Standardized paper-and-pencil test
Performance observations
Questionnaire (Looks at learner background, training, and experiences)
Review of past work experience
Interview past supervisors/managers

15. Objective tests
Multiple choice
True or False?
Matching items

16. Constructed-Response
Make learner plan answers and express them in their own words
Short-answer
Essay questions
Problem solving

17. Skills and Behavior Actions observed Process, product, or both?
Constraints/limitations?
Testing conditions simulated or realistic?

18. The Rubric General assessment of overall product
Certain elements attain points
More objective tool
From Designing Effective Instruction 4th Ed., Morrison, Ross, & Kemp, 2004.