1. An Experimental Study of Child Welfare Worker Turnover Nancy S. Dickinson, University of Maryland ndickinson@ssw.umaryland.edu John S. Painter painter.eval@gmail.com National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

2. Child Welfare Staff Recruitment and Retention: An Evidence Based Training Model Study Objectives Determine the feasibility of using an experimental design to study training outcomes Understand the impact of worker perceptions on their intent to leave child welfare employment Study the effectiveness of the intervention on worker retention National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

3. Retention External Environment Agency’s public image Awareness of jobs Agency Climate Shared mission Affirmation & recognition Shared authority Growth & advancement Org commitment Worker Characteristics Desire to help Self-efficacy Depersonalization Education Supervision Practice support Emotional support Team support The Work Role clarity Role expectations Workload Influences on Recruitment, Selection and Retention Recruitment Selection National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

4. Intervention Components RecruitmentSelectionRetention PosterRealistic Job Preview DVD An Invitation to Choose A Supervisor’s Guide to Retention FlyersCompetency Based Selection Process A Director’s Guide to Retention Custom BrochuresSelection TrainingRetention Training 2 30-second PSA’sRetention Toolkit Slide PresentationTechnical Assistance Recruitment Training National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

5. Research Questions Do workers in the intervention counties show statistically significant differences from those in the control counties on relevant survey scales? Does child welfare worker retention improve in the intervention counties compared with the control counties? National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

6. National Human Services Training Evaluation Symposium Cornell University, June 15, 2011 Procedures Random assignment of county child welfare agencies to 17 intervention and 17 control groups 33 project counties participated in data collection activities (1 agency withdrew after a year)

7. General Design Intervention: RO1XO2 Comparison: RO1O2 R = random assignment O = data collection (or observation) X = intervention or treatment National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

8. Instruments Online worker survey administered 5 times to all project child welfare workers between 6/1/05 and 6/1/08 Human Resources Database gathered employment information on all project workers between 12/1/04 and 9/1/08 National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

9. Worker Survey 17 scales validated using reliability analysis and confirmatory factor analysis Average response rate of 47% (45% - 48%) across 5 waves of delivery to an average sample of 831 workers (731-944) Waves 1 & 2 were pre-intervention; waves 4 & 5 were post-intervention. National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

10. Respondent Demographics Wave 1Wave 2Wave 3Wave 4 Wave 5 Total356354386422446 Gender Female87.1%86.7%88.3%89.2%87.5% Male12.9%13.3%11.7%10.8%12.5% Race African-American23.9%17.8%24.4%26.1%24.4% European- American 68.3%70.8%63.5%63.7%66.1% Other7.9%11.3%12.2%10.2%9.5% Age Average38393839 St. Dev10.51010.5 11 National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

11. Demographics, Continued Wave 1Wave 2Wave 3Wave 4 Wave 5 Total356354386422446 Degree Type Bachelor47.2%45.1%47.4%47.5%46.8% Master7.6%7.4%6.9%7.6%6.4% BSW33.4%33.7%31.7%29.4%30.7% MSW11.7%13.8%14.0%15.5%16.1% Missing(4.2%)(7.6%)(9.3%)(6.6%)(6.4%) Caseload Average Number of Families per Month 1413 12 St. Dev7.67.57.87.67.5 National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

12. Data Analyses Multi-level regression analysis –Scales are compared pre-post intervention Survival analysis –Days employed & status at end of study (exit vs. no exit) National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

13. Statistical Comparisons for Survey Scales Four primary comparisons were made: –Intervention vs. control post-training Individual level County level –Pre vs. post training intervention group only Individual level County level National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

14. Overview of Survey Results ScaleIndiv. Interv. Vs. Control Cnty. Interv. Vs. Control Indiv. Pre. Vs. Post Cnty. Pre. Vs. Post S1 Depersonalization ****** S2 Desire to help ***** S3 Self Efficacy ***** S4 Workload *** S5 Role Clarity ***** S6 Role Expectations S7 Supervisor: Practice Support *** S8 Supervisor: Team Support * S9 Supervisor Emotional Support S10 Organizational Commitment ** S11 Agency’s Negative Image S12 Agency Affirmation *** S14 Shared Mission *** S15 Shared Authority S16 Growth & Advancement Opportunities * S17 Intent To Leave (lower scores indicate lower intent to leave) ***** * P <.05, ** p <.01; *** p <.001 National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

15. Impact of Intervention on Turnover: Data and Sample HR Database used by all project counties –Internet accessible –Interactive database application In 9/08, analysis file of 877 workers hired after January 1, 2004 –485 workers from control counties –392 from intervention counties National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

16. Worker Demographics for Original and Propensity Matched Samples OriginalMatched InterventionControlInterventionControl Total392485318 Degree TypeN%N%N%N% No SW Degree236 60.2%19540.2% 165 51.9%15649.1% BSW or MSW15639.8% 29059.8% 15348.1% 16250.9% Previous Experience None7318.6%7214.8% 6520.4%5617.6% Indirect6616.8%6713.8% 3511.0%5818.2% Direct25364.5%34671.3% 21868.6%20464.2% National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

17. Analysis and Results Cox regression survival analysis assessed the impact of the intervention on undesirable exits Effect of the intervention is statistically significant (p<.05) – 27% of control group sample experienced an undesired exit –17% exit in the intervention group National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

18. National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

19. Type of Exit Post Intervention GroupType of ExitFrequencyPercent Control No Exit 18367.8 Undesired Exit 7427.4 Promoted 20.7 Transferred 103.7 Other 10.4 Total 270100 Intervention No Exit 15870.9 Undesired Exit 3917.5 Promoted 62.7 Transferred 177.6 Other 31.3 Total 223100 National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

20. Summary A rigorous research methodology can be used to test the effectiveness of a training intervention. Undesired exits by child welfare workers can be slowed significantly because of increased skills and behaviors of supervisors and managers. National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

21. Limitations Absence of statewide employee database limits quality of data. –Some concern that project database was used inconsistently –Cannot track workers across counties to determine if worker left the profession or the agency National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

22. What Worked Well Recruiting counties thru site visits Random assignment Providing counties with data on turnover Longitudinal design Control group Lots of personal contact with counties HR dbase data proved key Web surveys were very efficient National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

23. Think Twice… Number of counties in study Number of times surveyed Web reports National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

24. Unexpected Challenges Data management! A beast… Some counties were inconsistent in use of HR dbase Inconsistent response to surveys left gaps in data Collecting baseline data before intervention was finalized National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

25. Acknowledgement This study was supported by the U.S. Children’s Bureau (Grant No. 90CT0114) as part of the project Child Welfare Staff Recruitment and Retention: An Evidence- Based Training Model. THANKS National Human Services Training Evaluation Symposium Cornell University, June 15, 2011