An Experimental Study of Child Welfare Worker Turnover Nancy S. Dickinson, University of Maryland John S. Painter National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Child Welfare Staff Recruitment and Retention: An Evidence Based Training Model Study Objectives Determine the feasibility of using an experimental design to study training outcomes Understand the impact of worker perceptions on their intent to leave child welfare employment Study the effectiveness of the intervention on worker retention National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Retention External Environment Agency’s public image Awareness of jobs Agency Climate Shared mission Affirmation & recognition Shared authority Growth & advancement Org commitment Worker Characteristics Desire to help Self-efficacy Depersonalization Education Supervision Practice support Emotional support Team support The Work Role clarity Role expectations Workload Influences on Recruitment, Selection and Retention Recruitment Selection National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Intervention Components RecruitmentSelectionRetention PosterRealistic Job Preview DVD An Invitation to Choose A Supervisor’s Guide to Retention FlyersCompetency Based Selection Process A Director’s Guide to Retention Custom BrochuresSelection TrainingRetention Training 2 30-second PSA’sRetention Toolkit Slide PresentationTechnical Assistance Recruitment Training National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Research Questions Do workers in the intervention counties show statistically significant differences from those in the control counties on relevant survey scales? Does child welfare worker retention improve in the intervention counties compared with the control counties? National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

National Human Services Training Evaluation Symposium Cornell University, June 15, 2011 Procedures Random assignment of county child welfare agencies to 17 intervention and 17 control groups 33 project counties participated in data collection activities (1 agency withdrew after a year)

General Design Intervention: RO1XO2 Comparison: RO1O2 R = random assignment O = data collection (or observation) X = intervention or treatment National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Instruments Online worker survey administered 5 times to all project child welfare workers between 6/1/05 and 6/1/08 Human Resources Database gathered employment information on all project workers between 12/1/04 and 9/1/08 National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Worker Survey 17 scales validated using reliability analysis and confirmatory factor analysis Average response rate of 47% (45% - 48%) across 5 waves of delivery to an average sample of 831 workers ( ) Waves 1 & 2 were pre-intervention; waves 4 & 5 were post-intervention. National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Respondent Demographics Wave 1Wave 2Wave 3Wave 4 Wave 5 Total Gender Female87.1%86.7%88.3%89.2%87.5% Male12.9%13.3%11.7%10.8%12.5% Race African-American23.9%17.8%24.4%26.1%24.4% European- American 68.3%70.8%63.5%63.7%66.1% Other7.9%11.3%12.2%10.2%9.5% Age Average St. Dev National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Demographics, Continued Wave 1Wave 2Wave 3Wave 4 Wave 5 Total Degree Type Bachelor47.2%45.1%47.4%47.5%46.8% Master7.6%7.4%6.9%7.6%6.4% BSW33.4%33.7%31.7%29.4%30.7% MSW11.7%13.8%14.0%15.5%16.1% Missing(4.2%)(7.6%)(9.3%)(6.6%)(6.4%) Caseload Average Number of Families per Month St. Dev National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Data Analyses Multi-level regression analysis –Scales are compared pre-post intervention Survival analysis –Days employed & status at end of study (exit vs. no exit) National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Statistical Comparisons for Survey Scales Four primary comparisons were made: –Intervention vs. control post-training Individual level County level –Pre vs. post training intervention group only Individual level County level National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Overview of Survey Results ScaleIndiv. Interv. Vs. Control Cnty. Interv. Vs. Control Indiv. Pre. Vs. Post Cnty. Pre. Vs. Post S1 Depersonalization ****** S2 Desire to help ***** S3 Self Efficacy ***** S4 Workload *** S5 Role Clarity ***** S6 Role Expectations S7 Supervisor: Practice Support *** S8 Supervisor: Team Support * S9 Supervisor Emotional Support S10 Organizational Commitment ** S11 Agency’s Negative Image S12 Agency Affirmation *** S14 Shared Mission *** S15 Shared Authority S16 Growth & Advancement Opportunities * S17 Intent To Leave (lower scores indicate lower intent to leave) ***** * P <.05, ** p <.01; *** p <.001 National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Impact of Intervention on Turnover: Data and Sample HR Database used by all project counties –Internet accessible –Interactive database application In 9/08, analysis file of 877 workers hired after January 1, 2004 –485 workers from control counties –392 from intervention counties National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Worker Demographics for Original and Propensity Matched Samples OriginalMatched InterventionControlInterventionControl Total Degree TypeN%N%N%N% No SW Degree % % % % BSW or MSW % % % % Previous Experience None7318.6%7214.8% %5617.6% Indirect6616.8%6713.8% %5818.2% Direct % % % % National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Analysis and Results Cox regression survival analysis assessed the impact of the intervention on undesirable exits Effect of the intervention is statistically significant (p<.05) – 27% of control group sample experienced an undesired exit –17% exit in the intervention group National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Type of Exit Post Intervention GroupType of ExitFrequencyPercent Control No Exit Undesired Exit Promoted 20.7 Transferred Other 10.4 Total Intervention No Exit Undesired Exit Promoted 62.7 Transferred Other 31.3 Total National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Summary A rigorous research methodology can be used to test the effectiveness of a training intervention. Undesired exits by child welfare workers can be slowed significantly because of increased skills and behaviors of supervisors and managers. National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Limitations Absence of statewide employee database limits quality of data. –Some concern that project database was used inconsistently –Cannot track workers across counties to determine if worker left the profession or the agency National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

What Worked Well Recruiting counties thru site visits Random assignment Providing counties with data on turnover Longitudinal design Control group Lots of personal contact with counties HR dbase data proved key Web surveys were very efficient National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Think Twice… Number of counties in study Number of times surveyed Web reports National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Unexpected Challenges Data management! A beast… Some counties were inconsistent in use of HR dbase Inconsistent response to surveys left gaps in data Collecting baseline data before intervention was finalized National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

Acknowledgement This study was supported by the U.S. Children’s Bureau (Grant No. 90CT0114) as part of the project Child Welfare Staff Recruitment and Retention: An Evidence- Based Training Model. THANKS National Human Services Training Evaluation Symposium Cornell University, June 15, 2011