1. Evaluating Teacher Training changing classroom practices Richard Lambert, Ph.D.

2. Magic bullets and everyday practice In educational evaluation we try to determine the value of the magic bullet. In educational evaluation we try to determine the value of the magic bullet. In the end what we really end up doing is investigating the interaction between the magic bullet and the teachers who were asked to change their practices. In the end what we really end up doing is investigating the interaction between the magic bullet and the teachers who were asked to change their practices.

3. Magic bullets and everyday practice We discussed this question as a part of Process Evaluation We discussed this question as a part of Process Evaluation –Implementation Fidelity –Follow-up technical assistance –Administrative buy-in –Nothing happens without the teacher What process must take place for magic bullets to make it into everyday classroom practice? What process must take place for magic bullets to make it into everyday classroom practice?

4. Magic bullets and everyday practice Teachers have to see a need for change Teachers have to see a need for change Teachers have to believe the magic bullets will meet the need Teachers have to believe the magic bullets will meet the need Teachers have to be “free” to change Teachers have to be “free” to change Resources have to be available Resources have to be available Time to plan Time to plan Training has to be effective Training has to be effective What is effective training? What is effective training?

5. Effective Training Targeted to real needs Targeted to real needs Engaging / Challenging Engaging / Challenging Prompts new thinking Prompts new thinking Modeling Modeling Training with peers / time to process Training with peers / time to process Time to experiment / try out methods Time to experiment / try out methods Learning takes place / new knowledge Learning takes place / new knowledge Leads to change in practice Leads to change in practice

6. Barriers to Change Administrative support / expectations for implementation Administrative support / expectations for implementation Technical assistance and follow-up Technical assistance and follow-up Interactions between the new and innovative strategies from training and what was already in place Interactions between the new and innovative strategies from training and what was already in place Implementation fidelity Implementation fidelity Resources for sustainability Resources for sustainability

7. The Kirkpatrick Model The first task involved in evaluating a training or professional development program, is to design a process for documenting what actually takes place during the training. The first task involved in evaluating a training or professional development program, is to design a process for documenting what actually takes place during the training. The evaluator needs to conduct a mini-process evaluation about the training itself. The evaluator needs to conduct a mini-process evaluation about the training itself.

8. The Mini-Process Evaluation Design an evaluation process to document the following: Design an evaluation process to document the following: Who attended? Who attended? What content was delivered? What content was delivered? What mix of activities and teaching techniques did the trainer use? What mix of activities and teaching techniques did the trainer use?

9. The Mini-Process Evaluation What were the physical characteristics of the space in which the training took place? What were the physical characteristics of the space in which the training took place? How long was the actual contact time between the attendees and the trainer, considering breaks and time-on-task issues? How long was the actual contact time between the attendees and the trainer, considering breaks and time-on-task issues? How engaged were the attendees? How engaged were the attendees?

10. Process Evaluation Strategies Video or audio taping of the training Video or audio taping of the training The tapes can then be translated into transcripts or other records The tapes can then be translated into transcripts or other records The tapes and / or transcripts can be coded The tapes and / or transcripts can be coded Attendance records Attendance records Observers / Checklists Observers / Checklists Interviews Interviews

11. Next Steps Evidence for the effectiveness of Training can take many forms Evidence for the effectiveness of Training can take many forms –Simple short term satisfaction –Causal inferences about the effects of training –High levels of organizational change and long term impact.

12. Next Steps The next task is to determine the level or levels of evaluation evidence that the stakeholders are interested in, and that the resources are sufficient to address. The next task is to determine the level or levels of evaluation evidence that the stakeholders are interested in, and that the resources are sufficient to address. Kirkpatrick’s four levels of evaluation can serve as a guide in this process. Kirkpatrick’s four levels of evaluation can serve as a guide in this process.

13. The Kirkpatrick Model Reaction Reaction Learning Learning Behavior Behavior Results Results The goal is to make INFORMED judgments about the merit or value of training. The goal is to make INFORMED judgments about the merit or value of training.

14. The Levels The four levels from Kirkpatrick’s model The four levels from Kirkpatrick’s model Level I – Reaction Level I – Reaction –How well did the attendees like the training? Level II – Learning Level II – Learning –What concepts, facts, or skills were learned? Level III – Behavior Level III – Behavior –What changes in behavior or attitude did the attendees make once back on the job? Level IV – Results Level IV – Results –What were the tangible impacts of the training program on the organization?

15. The Levels Each of these four levels of evaluation is associated with specific evaluation goals and strategies. Each of these four levels of evaluation is associated with specific evaluation goals and strategies. With each increase in level comes an increased level of difficulty for the evaluator. With each increase in level comes an increased level of difficulty for the evaluator.

16. The Levels It may be relatively easy to document process and gauge participant reaction to a training program. It may be relatively easy to document process and gauge participant reaction to a training program. It may be very difficult to establish causal links between a training program and organizational changes. It may be very difficult to establish causal links between a training program and organizational changes.

17. The Levels A set of specific steps, in the form of questions and activities, will be outlined in the coming weeks to further illustrate the nature of each level. A set of specific steps, in the form of questions and activities, will be outlined in the coming weeks to further illustrate the nature of each level. What evaluation activities / strategies match the levels? What evaluation activities / strategies match the levels? –Reaction –Learning –Behavior –Results