1. Professional Development: Evaluation Strategies to Maximize Impact
Jeffri Brookfield
National Center for Systemic Improvement & IDEA Data Center
Chelsea Guillen
Early Intervention Training Program at the University of Illinois

2. Purposes
To examine a framework for developing an evaluation of improvement strategies that rely on professional development.
To examine the usefulness of information yielded by various measurement tools/approaches.
To examine sample evaluation plans with attention to pivotal decision points in the implementation process.

3. Importance of Evaluating Professional Development
Professional development (PD) is costly, too often ineffective, and still necessary to establish changes in professional practice.
A good evaluation plan for PD initiatives will:
Answer the most important questions,
In the most useful manner,
Allowing for timely corrections/modifications,
While minimizing the cost and labor-burden of evaluation.

4. Implementing a PD improvement strategy
Develop training & select trainees
Conduct Training
Implement Practice
Implementation Support
Evaluation Topics
How did participants perceive the training?
Did participants acquire new knowledge/skills?
Did the intervention get implemented as planned?
What implementation support is needed?
Did implementation improve after support was offered?
What content?
Who will participate?

5. Measures: Satisfaction/Perception
How well did the training meet your needs?
Were the trainers knowledgeable?
Did you gain sufficient training in order to…?
What was most valuable (least valuable) about the training?
And so forth……

6. Example Satisfaction Survey
Satisfaction with a consultant
Satisfaction with a training

7. Measures: Knowledge/Skill
Written assessment
multiple choice,
short answer
Written product
IFSP
Behavioral intervention plan
Assessment report
Other…

8. Example Knowledge/Skills Measures

9. Measures: Observational (direct or video)
Rubric
Checklist
Scale
PLACHEK, event, duration, number, frequency, rate….

10. Sample Observational Measures

11. Participants & Initial Training
Characteristics of participants
Any prerequisites?
Characteristics of the training:
Explicit explanation & illustration of practice
Opportunities for practice
Reflection
Sufficient duration and intensity
…..

12. Immediately after training
Knowledge acquired?
Satisfaction with training experience?
Confidence in skills?
Action plan for implementation of new skills?

13. Transfer of training to practice
What practices are being implemented?
Fidelity of implementation?
Frequency/intensity of practice?
Adherence to a standard?

14. Follow-up support Frequency and quality of feedback?
Support/action plan?
Satisfaction of support experienced?

15. Sample measurement tools used by states
EITP Quality Professional Development Rubric
RI Services Rendered Form (monitoring tool)
GVSU Coaching Self-Assessment
UIC ERF Observation Coaching Session Summary
TACSEI Family Coaching Checklist
Implementing the Pyramid Model in Home Visiting Programs: Benchmarks of Quality
Evaluation of Coaching/Support Received
Recommended Practices Performance Checklists
……...

16. Implementing a PD improvement strategy : Pivotal evaluation points
Develop training & select trainees
Conduct Training
Implement Practice
Implementation Support
Evaluation Topics
How did participants perceive the training?
Did participants acquire new knowledge/skills?
Did the intervention get implemented as planned?
What implementation support is needed?
Did implementation improve after support was offered?
What content?
Who will participate?

17. Planning Guide Implementation Step: ___________________
Evaluation question(s) [i.e. what do you need to know]?
What data will be collected and what tools will be used?
When and how will data be collected?
When, how, and by whom will data be analyzed?
Who will use the results to make decisions, and how and when will these decisions be made?
What are potential modifications you may make to the implementation of the improvement strategy based on these data?

18. Discussion Questions Is your state using any of the following?
Post Training Satisfaction Survey
Pre/Post Training Knowledge-Based Assessment
Implementation Fidelity Checklist
Coaching Satisfaction Survey
What additional measurement tools would be useful to your state for determining how well interventions are being implemented?
What is your plan for evaluating the data being collected?
How will the data be used for continuous improvement?

19. Contact information
Jeffri Brookfield
Chelsea Guillen

20. Disclaimer
The contents of this presentation were developed, in part, under grants from the U.S. Department of Education, #H326R and #H373Y However, the contents do not necessarily represent the policy of the Department of Education, and you should not assume endorsement by the Federal Government.
Project Officers: Perry Williams and Shedeh Hajghassemali
Richelle Davis and Meredith Miceli