Child Welfare Training Evaluation in California Update on the Strategic Planning Process RTA All Staff SPS | March

BACKGROUND  2002 – CalSWEC, RTAs/IUC began development of CC training  Part of an overall strategic plan for child welfare training evaluation  Purpose: to develop rigorous methods to assess and report effectiveness of training so that the findings can be used to improve training & training-related services. 2

What We Know So Far (2005-June 30, 2011) 5162 line workers completed demographic surveys while participating in common core. 86 non-child welfare personnel completed demographic surveys while participating in common core. 650 child welfare supervisors completed demographic surveys while participating in supervisor core. 3

What we know so far (cont’d) Statewide, line workers have made statistically significant improvements from pre to post for the Child & Youth Development in a Child Welfare Context, Family Engagement in Case Planning and Case Management, and Permanency & Placement trainings. Generally, between 86-95% of line workers make three out of four correct decisions as to whether or not physical or sexual abuse occurred in test scenarios. IV-E graduates benefit from Core training. 4

What have we learned…? …FROM TRAINING EVALUATION THAT WE WOULDN’T KNOW IF WE DIDN’T EVALUATE? Who is participating in training, and who is being hired by counties (demographics). That training has been effective. That certain parts of curricula have become obsolete, are unclear, or relevant content is missing (and therefore curricula have been revised). 5

What else have we learned? That data collection has allowed us to build knowledge in the field, in that we can use the data easily for additional studies (CFAS, Stereotype Threat): ◦ Child Forensic Attitude Scale (Survey) administered pre and post CMI2: Training may affect trainee attitudes about children’s disclosures of sexual abuse. ◦ Stereotype Threat Pilot: Test results may be more fair and accurate if we change WHEN we administer the demographic forms (before vs. after testing). 6

How does Training Evaluation help Counties and CDSS? We know basic demographic characteristics of the public child welfare workforce. We know that the average new CWW and new supervisor leaves training with select knowledge and skills related to their jobs. We engage in a systematic process that allows training content and delivery to continually be refined and made as relevant as possible to county CWWs. 7

CURRENT CONTEXT  Flash forward: nearing the end of our second strategic plan for child welfare training evaluation  For next strategic plan (beginning July 2012):  Received feedback from STEC  Received feedback from CWDA  Convened joint Macro Eval Team/CDOG strategic planning meeting (3/1/12) 8

Feedback from STEC (Dec 2011) Information given on the job sometimes contradicts information given in Core training Continuum of transfer of learning? Establish cut-points? Evaluate coaching/training? Evaluate methods of delivery? 9

Feedback from CWDA (Feb 2012) Training evaluation efforts are valuable. Want to assess learning on the job (and tools for use on the job) Want tools that foster critical thinking Interested in comparing modalities of training Perhaps think about moving forward on a transfer of learning tool for supervisors 10

Highlights from Macro/CDOG mtg: Continue to focus the evaluation efforts on what the federal govt requires via the Child and Family Services Reviews (Initial & Ongoing Training, Items 32 & 33): A.Collect data about training effectiveness, including quality assurance results, if available, and other data about: (1) initial staff training (incl. sprvs and mgrs) (2) the content and quality of the training, and (3) how training is reflected in job performance B. Track that staff are meeting State training requirements and to identify those who need training. 11

Highlights from Macro/CDOG mtg: Continue to build our system on the levels that we have, but critically examine how to make the evaluation and data gathering most efficient: ◦ Formative evaluations (feedback on curricula/trainings, quality assurance, etc.) ◦ Demographic data (Common & Supe Core) ◦ Knowledge tests (CYD, CP/CM, PP, SRA) ◦ Embedded evals (CMI1, CMI2, SDM/SRA, Supe Core) ◦ Option to continue stereotype threat, CFAS data 12

Highlights from Macro/CDOG mtg: What will be emphasized in next strategic plan: ◦ A utomation ◦ C ourse evaluation (e.g., focus content on what is essential with help from RTAs, counties and CDSS, looking at curriculum changes such as sequencing) ◦ T ransfer of Learning 13

QUESTIONS? 14

SUGGESTIONS? 15

AND LAST, BUT NOT LEAST: WE VERY MUCH APPRECIATE ALL OF YOU WHO HELP MAKE TRAINING EVALUATION HAPPEN IN CALIFORNIA CHILD WELFARE!!! 16

For more information… Refer to training evaluation section of CalSWEC website: Contact Leslie Zeitler, Training and Evaluation Specialist at CalSWEC: 17