1. Workshop Evaluation Worth the Effort! NLTAPA Conference Boise, ID NLTAPA Conference Boise, ID

2. Evaluation Overview Purpose: To determine the effectiveness of a training program.

3. Evaluation Overview Four-level model developed by Donald Kirkpatrick (1998). Reaction Learning Behavior Results

4. Evaluation Overview (continued) Each successive evaluation level is built on information provided by the lower level.

5. Level 1 Level 2 Level 3 Level 4

6. Reaction (Level 1) What they thought and felt about the training.

7. Level 1

8.  Courses expanded, or condensed  Instructors adjusted  Courses removed from the program pending further review

9. Courses expanded, or condensed Stormwater Management Participant Comments (Fall 2010) Too long and too deep of explanations. Too much info that doesn’t concern our needs. Maintenance section is too repetitive.

10. Instructors Adjusted Winter Maintenance Participant Comments (Fall 2012) Instructor should be someone that worked in this field at public works level. Instructor seems to lack practical experience in this field. Less talk about cost. More on winter maintenance. (That's what our bosses are for– keeping cost down.)

11. Courses Removed Traffic Signals Participant Comments (Fall 2012) Make half days. Break into different sections. Break into different levels – decision planning vs. on-going maintenance.

12. Learning – Level 2 Pre and Post Quiz Developed to evaluate the instruction and content of our training programs (NOT the student) Consists of 6 questions Note your answers on the back of the evaluation sheet

13. Quiz Example Question 1.Denying physical access to someone because of their impairment is: a.Sometimes necessary b.Discrimination c.Permitted by law in certain circumstances d.Unimportant compared to making roads safe for cars

14. Learning  Course content modified  Quiz questions refined

15. “Un-Learning” in Winter Maintenance

16. Behavior - Level 3  Enhanced skills and knowledge identified on the job o Newer process o 6 month evaluation

17. Level 3

18. Results – Level 4 Cost savings of approximately $5,000 related to remediation of current signage $1,500 of savings in equipment expenditures $30,000 of expenses avoided that might have been incurred if gas line had to be relocated after completion Lawsuits avoided

19. Questions... LTAP – Local Technical Assistance Program Pennsylvania Department of Transportation Bureau of Planning and Research 400 North Street, 6 th Floor Harrisburg, PA 17120 E-mail:LTAP@state.pa.us Web Site: www.ltap.state.pa.us