1. Training Kirkpatrick’s Four Levels of Evaluation Kelly Arthur Richard Gage-Little Dale Munson Evaluation Strategies for Instructional Designers

2. Lost Keys

3. Where should we be searching for the keys to good evaluation results?

4. Beginning to see the light… 100 80 60 40 20 0 ReactionLearningBehaviorResults 3% 13% 37% 95% 2000 ASTD State of the Industry Report Kirkpatrick’s four levels

5. Billions Wasted About $5.6 billion to $16.8 billion is wasted annually on ineffective training programs that focus on so-called soft skills, based on estimates from a Rutgers University study. USA Today

6. Higher Standard of Accountability In the “new economy”, training and development initiatives are being held to a higher level of financial accountability. Every dollar spent on training should produce measurable business results – increases in key business metrics.

7. Results of Evaluation Survey How important do you think evaluations are in the instructional design process? “Very important, otherwise you don’t know if your training met your objectives” “Evaluations are integral, but often overlooked” “Crucial, they’re the only way to know if we are reaching our goals” “Without evaluations, we have no clue what what the core of the problem is”

8. Results of Evaluation Survey Have you completed an evaluation and how did you feel about it? “I never found them very useful to me personally” “…felt they were a waste of my time” “…dreading it” “…seems ineffective since the next class is presented the same way” “A little annoyed if I was busy”

9. What is Evaluation? Evaluation helps document whether a program is accomplishing its goals or not. It identifies program weaknesses and strengths and the areas of the program that need revision. Asking questions Collecting answers Making decisions based on those answers

10. What is Assessment? Assessment is the systematic gathering of information about “things” that are to be evaluated. Examples: Behaviors, attitudes, performance Identifies needs or gaps Determines whether or not instruction or training is needed

11. Two Types of Evaluation Formative Evaluation Occurs during development phase Focuses on process Used to help instructional designers improve the quality of instruction Summative Evaluation Occurs near, at, or after the end of a project Focus is on the effectiveness Used to help instructional designers validate their training

12. Five Evaluation Models 1. Hierarchy of Evaluation (Kirkpatrick) 2. Objectives Approach (Tyler) 3. Goal-Free (Scriven) 4. CIPP (Stufflebeam) 5. Naturalistic (Guba)

13. Who is Donald Kirkpatrick? Professor at University of Wisconsin Initial articles published in 1959-1960 American Society of Training Directors, ASTD Most widely recognized and used model in the training industry

14. Phases of Evaluation in Training 1234 Training Work Place 1. Reaction 2. Learning 3. Behavior 4. Results 1. Pre-Testing 2. During training event 3. After training 4. In the workplace Kirkpatrick’s four levels

15. Hierarchy of Evaluation Four Levels of Evaluation Level 1 – Reaction (participant satisfaction) Level 2 – Learning (knowledge, mastery) Level 3 – Behavior (transference of skills) Level 4 – Results (community impact) Fifth level was recently “added” for return on investment (ROI) but this was not in Kirkpatrick’s original model.

16. Level One Evaluation Measures reaction Checks participants response to: Session or course Trainer’s presentation style or content Quality of training materials Example: Questionnaires or surveys

17. Example at Sun How satisfied were you with the instructor’s technical expertise? How satisfied were you with the labs/and or exercises provided? How satisfied were you with the accuracy of the course materials?

18. Level Two Evaluation Measures learning Checks participants response to: Knowledge and factual information Skills Attitudes Interpretation of information Example: Pre and Post-Testing

19. Measures behavior Checks participants ability to: Carry out tasks more effectively after the course Provide better care to clients Be more knowledgeable/skillful in job performance Examples: Pre and post tests, surveys and interviews, self-assessments Establishes a baseline of skills/knowledge Level Three Evaluation

20. Measures results Provides information about: Bottom line final results Impact on the organization Cost vs. Benefit Example: Reports Proof vs. Evidence Level Four Evaluation

21. Example at Sun Service call reduction – external customers Reduction in the amount of time spent per service call – internal support engineers

22. Hierarchy of Evaluation LevelMeasuresExample 1ReactionSurveys 2LearningQuizzes 3BehaviorPre and Post Test 4ResultsReports

23. Pre-Test Answers 1. Reaction: Survey, questionnaires 2. Learning: Case studies, tests and quizzes 3. Behavior: Pre and post tests, interviews, self-assessments 4. Results: Reports 5. Formative evaluation 6. Summative evaluation

24. In-Class Activity

25. Your boss, Mr. Grinch, has decided to downsize his company, Hateful Things. The training program that you’ve been conducting on “How to ruin Christmas” is currently under review. You’ve been asked to justify and qualify why this training and your job are important to his company. How will you do this?

26. Remember The “Keys” To Evaluation Reaction Learning Behavior Results

27. References Kirkpatrick, Donald. “Another Look at Evaluating Training Programs.” ASTD, Alexandria, VA. 1998. Kirkpatrick, Donald. “Evaluating Training Programs.” Berrett-Koehler Publishers, San Francisco, CA. 1998.