1. Developing a Model of Trainer Evaluation Leslie W. Zeitler, LCSW May 2010: 13 th Annual National Human Services Training Evaluation Symposium “Problem/Brainstorm Session”

2. 2 Problem/Brainstorm Session Not a presentation Not a presentation Is an invitation to contribute to thinking on this topic… Is an invitation to contribute to thinking on this topic…

3. 3 Context Child Welfare Training Evaluation in CA: Child Welfare Training Evaluation in CA: Oversight by Macro Evaluation Team (subcommittee of Statewide Training & Education Committee) Oversight by Macro Evaluation Team (subcommittee of Statewide Training & Education Committee) Began conceptualization in 2002 Began conceptualization in 2002 Began actual evaluations in 2005 Began actual evaluations in 2005 1 st Strategic Plan (through 2009): focused on child welfare worker and supervisor knowledge/skill evaluations. 1 st Strategic Plan (through 2009): focused on child welfare worker and supervisor knowledge/skill evaluations.

4. 4 Data Currently Collected in CA: Under 1 st strategic plan (2004-2009) and continuing through 2 nd strategic plan (2009-2012): Under 1 st strategic plan (2004-2009) and continuing through 2 nd strategic plan (2009-2012): Trainee Demographics Trainee Demographics Trainee Satisfaction (at the regional/county level, but not statewide) Trainee Satisfaction (at the regional/county level, but not statewide) Trainee Knowledge (acquisition of knowledge) Trainee Knowledge (acquisition of knowledge) Trainee Skill (application of knowledge in the classroom via embedded evaluations) Trainee Skill (application of knowledge in the classroom via embedded evaluations)

5. 5 Current (2 nd ) Strategic Plan for CW Training Evaluation in CA: In 2 nd strategic plan, starting to look at additional factors that affect trainees and training: In 2 nd strategic plan, starting to look at additional factors that affect trainees and training: Attitudes, values Attitudes, values Stereotype threat pilot Stereotype threat pilot Item analysis by trainer Item analysis by trainer Development of model of trainer evaluation Development of model of trainer evaluation Etc. Etc.

6. 6 Models of Trainer Evaluation? Do you have a systematic method or model for evaluating trainers? Do you have a systematic method or model for evaluating trainers? If so, is it a model based in theory? If so, is it a model based in theory? If so, which model? If so, which model?

7. 7 Factors for Consideration: Need to identify: Need to identify: Purpose(s) of trainer evaluation Purpose(s) of trainer evaluation Elements/dimensions for evaluation Elements/dimensions for evaluation Methods of evaluation Methods of evaluation Measurement format Measurement format Uses of data Uses of data

8. 8 Purpose Purpose(s) of evaluating trainers Purpose(s) of evaluating trainers Improve quality of training Improve quality of training Standardize training delivery Standardize training delivery Establish basis for trainer fees (On a statewide level) Establish basis for trainer fees (On a statewide level) As part of ongoing professional development for trainers As part of ongoing professional development for trainers For personnel purposes (promotion, probation, firing) For personnel purposes (promotion, probation, firing)

9. 9 I.D. Elements/dimensions for evaluation: Dimensions of trainer performance Dimensions of trainer performance From 2001 APHSA/NSDTA Trainer Competency Model? From 2001 APHSA/NSDTA Trainer Competency Model? From Institute for Human Services competencies? From Institute for Human Services competencies? From other (competency) models? From other (competency) models?

10. 10 Identify Methods of Evaluation: Observation? Observation? By whom? By whom? Peers? Peers? Supervisors? Supervisors? Self-reflection? Self-reflection? Trainee feedback? Trainee feedback? Aggregate data from trainees’ tests? Aggregate data from trainees’ tests? What would be the advantages and disadvantages of each method of evaluation? What would be the advantages and disadvantages of each method of evaluation?

11. 11 Identify Measurement Format: Likert scale? Likert scale? Anchors? (How would we go about anchoring any scale?) Anchors? (How would we go about anchoring any scale?) What tools have other training evaluators used? What tools have other training evaluators used? Sample Tools: Sample Tools: Kentucky (Debbie Dever) Kentucky (Debbie Dever) Indiana (Evoke Communications/Quay Kester) Indiana (Evoke Communications/Quay Kester)

12. 12 Identify Uses of Collected Data: How often are trainer evaluations done? How often are trainer evaluations done? Who gets to see the trainer evaluation data? Who gets to see the trainer evaluation data? What decisions could result from ongoing collection and analysis of trainer evaluation data? What decisions could result from ongoing collection and analysis of trainer evaluation data? High stakes? High stakes? Medium stakes? Medium stakes? Low stakes? Low stakes? What analyses could be done? What analyses could be done? In aggregate, to see improvement over time? In aggregate, to see improvement over time? Etc. Etc.

13. 13 Anything else? What else, if anything, needs to be considered when designing a trainer evaluation model? What else, if anything, needs to be considered when designing a trainer evaluation model? Involvement of trainers in the design process? Involvement of trainers in the design process? Etc. Etc.

14. 14 Additional Thoughts? Contact Leslie W. Zeitler at: Contact Leslie W. Zeitler at: lzeitler@berkeley.edu California Social Work Education Center