1. Sophia Gatowski, Ph.D., Consultant National Council of Juvenile & Family Court Judges Sophia Gatowski, Ph.D., Consultant National Council of Juvenile & Family Court Judges CIP Meeting, Aug 2009 Is Your Training Program Making a Difference? A National Implementation Guide to Conducting an Effective Training Evaluation

2. Why a Dependency Court Training Evaluation Guide? The Need The Vision –NCJFCJ, ABA, NCSC Funded by Children’s Bureau © 2009, NCJFCJ

3. What we learned … Interviews with training administrators, and review of past CIP, Model Court and other Dependency Court training evaluation materials, found: –Constraints on evaluation activities Funding Staff Time –Most evaluations limited to exit surveys Numbers trained & roles trained Reactions to training (faculty, content, venue) Satisfaction with overall experience (expectations)

4. What we learned … –Behavior/ attitude change rarely measured If measured mostly self-report –Training impacts/ outcomes rarely measured Few quasi-experimental or experimental designs Lack of generalizability –Measurement issues What is being evaluated? Multi-method approaches infrequent

5. Training Evaluation Guide Goals: Define basic criteria for judicial training evaluation tools; Provide evaluation guidance and tools that allow for national data collection on training outcomes; Help training administrators to design evaluation tools that go beyond measuring “reactions” to training (more about this later); Provide guidance to CIP, Model Courts and others in designing and implementing effective training evaluations.

6. Training Evaluation Guide TRAINING DESIGN  TRAINING IMPLEMENTATION  IMMEDIATE EVALUATION  FOLLOW-UP EVALUATION  OUTCOMES

7. Guide Development Process Conducted interviews with leading educators and training evaluators Extensive review of literature and resources on effective training models Obtained and reviewed past CIP, Model Court and other dependency court training evaluation materials Convened an Advisory Committee –CIP Directors, judicial educators, evaluation experts

8. Training Evaluation Basics Chapter 1: Essential Components of Effective Evaluation –Designing and planning trainings with evaluation in mind Overview of effective training designs and strategies Consideration of adult learning principles Learning objectives, goal and outcome specification Plan for using the evaluation data

9. Training Evaluation Basics Chapter 2: The Basics of Training Evaluation –Purpose and goals for training evaluation Measurement issues (Constructs, operational definitions, validity, reliability) Logic Models –4 levels for training evaluation Satisfaction Learning/ knowledge acquisition Behavior change Impact of training on outcomes Training Logic Model Example

10. The Levels of Training Evaluation Chapter 3: Level 1- Satisfaction Evaluation –Strategies (when to use them, how to use them) –Recommendations for implementation Question construction tips Data collection strategies –Template satisfaction evaluation Reaction tools –Analysis Sample report formats Taking satisfaction analyses to the next level Reaction Tool Example

11. The Levels of Training Evaluation Chapter 4: Level 2- Learning/ Knowledge Acquisition Evaluation –Strategies (when to use, how to use) –Recommendations for implementation Question construction tips Data collection strategies –Using quasi-experimental designs Advantages, challenges, and strategies –Template learning/ knowledge evaluation –Analysis Sample report format Skill Test Example

12. The Levels of Training Evaluation Chapter 5: Level 3- Behavior Change –Strategies (when to use, how to use) –Recommendations for implementation Question construction tips Data collection strategies –Using quasi-experimental designs Advantages, challenges, and strategies –Template learning/ knowledge evaluation –Analysis Sample report format Self-report Example Court Observation Example

13. The Levels of Training Evaluation Chapter 6: Level 4- Impact/ Outcomes –Strategies (when to use, how to use) –Recommendations for implementation Resource needs (time, staff) Measurement issues –Using quasi-experimental and experimental designs Advantages, challenges, and strategies –Sample outcome evaluation designs –Analysis Analytic approaches Sample report formats

14. Putting it All Together Chapter 7: Combining Levels of Evaluation for a Multi-Tiered Approach –Integrating different evaluation levels into a complete tool – Steps for creating an evaluation sequence Evaluation Sequence Example Appendix –Evaluation templates –Additional readings and resources

15. We want to hear from you … What advice do you have as we finalize the development of the Training Evaluation Guide? Are there particular areas we should focus more on? Do you have any “cool” and effective evaluation strategies or tips you would like to share? Other thoughts or recommendations?

16. Next Steps …. Complete chapters and templates –Incorporate additional ideas, strategies [Please complete our Follow-up Sheet] –Review of Guide by advisory committee –Finalize and disseminate to all CIP, Model Courts

17. Contact Info Nancy Miller, Director Permanency Planning for Children Dept., National Council of Juvenile & Family Court Judges nmiller@ncjfcj.org Sophia Gatowski, Ph.D., Consultant National Council of Juvenile & Family Court Judges sgatowski@ymail.com