Professional Development: Evaluation Strategies to Maximize Impact Jeffri Brookfield National Center for Systemic Improvement & IDEA Data Center Chelsea Guillen Early Intervention Training Program at the University of Illinois

Purposes To examine a framework for developing an evaluation of improvement strategies that rely on professional development. To examine the usefulness of information yielded by various measurement tools/approaches. To examine sample evaluation plans with attention to pivotal decision points in the implementation process.

Importance of Evaluating Professional Development Professional development (PD) is costly, too often ineffective, and still necessary to establish changes in professional practice. A good evaluation plan for PD initiatives will: Answer the most important questions, In the most useful manner, Allowing for timely corrections/modifications, While minimizing the cost and labor-burden of evaluation.

Implementing a PD improvement strategy Develop training & select trainees Conduct Training Implement Practice Implementation Support Evaluation Topics How did participants perceive the training? Did participants acquire new knowledge/skills? Did the intervention get implemented as planned? What implementation support is needed? Did implementation improve after support was offered? What content? Who will participate?

Measures: Satisfaction/Perception How well did the training meet your needs? Were the trainers knowledgeable? Did you gain sufficient training in order to…? What was most valuable (least valuable) about the training? And so forth……

Example Satisfaction Survey Satisfaction with a consultant Satisfaction with a training

Measures: Knowledge/Skill Written assessment multiple choice, short answer Written product IFSP Behavioral intervention plan Assessment report Other…

Example Knowledge/Skills Measures

Measures: Observational (direct or video) Rubric Checklist Scale PLACHEK, event, duration, number, frequency, rate….

Sample Observational Measures

Participants & Initial Training Characteristics of participants Any prerequisites? Characteristics of the training: Explicit explanation & illustration of practice Opportunities for practice Reflection Sufficient duration and intensity …..

Immediately after training Knowledge acquired? Satisfaction with training experience? Confidence in skills? Action plan for implementation of new skills?

Transfer of training to practice What practices are being implemented? Fidelity of implementation? Frequency/intensity of practice? Adherence to a standard?

Follow-up support Frequency and quality of feedback? Support/action plan? Satisfaction of support experienced?

Sample measurement tools used by states EITP Quality Professional Development Rubric RI Services Rendered Form (monitoring tool) GVSU Coaching Self-Assessment UIC ERF Observation Coaching Session Summary TACSEI Family Coaching Checklist Implementing the Pyramid Model in Home Visiting Programs: Benchmarks of Quality Evaluation of Coaching/Support Received Recommended Practices Performance Checklists ……...

Implementing a PD improvement strategy : Pivotal evaluation points Develop training & select trainees Conduct Training Implement Practice Implementation Support Evaluation Topics How did participants perceive the training? Did participants acquire new knowledge/skills? Did the intervention get implemented as planned? What implementation support is needed? Did implementation improve after support was offered? What content? Who will participate?

Planning Guide Implementation Step: ___________________ Evaluation question(s) [i.e. what do you need to know]? What data will be collected and what tools will be used? When and how will data be collected? When, how, and by whom will data be analyzed? Who will use the results to make decisions, and how and when will these decisions be made? What are potential modifications you may make to the implementation of the improvement strategy based on these data?

Discussion Questions Is your state using any of the following? Post Training Satisfaction Survey Pre/Post Training Knowledge-Based Assessment Implementation Fidelity Checklist Coaching Satisfaction Survey What additional measurement tools would be useful to your state for determining how well interventions are being implemented? What is your plan for evaluating the data being collected? How will the data be used for continuous improvement?

Contact information Jeffri Brookfield jbrookf@wested.org https://ncsi.wested.org/ https://ideadata.org/ Chelsea Guillen cguillen@Illinois.edu http://eitp.education.illinois.edu/

Disclaimer The contents of this presentation were developed, in part, under grants from the U.S. Department of Education, #H326R140006 and #H373Y130002. However, the contents do not necessarily represent the policy of the Department of Education, and you should not assume endorsement by the Federal Government. Project Officers: Perry Williams and Shedeh Hajghassemali Richelle Davis and Meredith Miceli