1. Organizational Factors Impacting Evaluation NHSTES May 25, 2016 University of California, Berkeley Janice Saturday and Deidre Carmichael

2. Presentation Overview Context: Realistic Job Previews in Employee SelectionGeorgia’s Employee Selection Protocol (ESP)Initial Evaluation Focus Organizational Factors Influencing Evaluation Implementation Group Discussion: Addressing Organizational Challenges to Evaluation

3. The Context: Realistic Job Previews and Georgia’s ESP The Context: Realistic Job Previews and Georgia’s ESP

4. Realistic Job Previews (RJPs) RJPs provide an accurate but balanced picture of the job Use of RJPs aimed at: Improving prospective employee decision making about the job Reducing turnover caused by failure to understand the job Improving job satisfaction by reducing the likelihood new workers will be surprised by difficult aspects of the job (CPS Human Resources, Champnoise, & Masternak, 2004; Graef & Potter, 2002)

5. Impact of RJPs Available research suggest RJPs can have a positive impact on the selection process, job expectations, and staff retention (Faller, Masternak, Grinnell-Davis, Grabarek, Sieffert, & Bernatovicz (2009).

6. History of Georgia’s ESP GA Retention and Turnover Study (2003) Development of ESP components and tools (2004-2005) Use of ESP with IV-E applicants (2006)

7. Enhance the quality of services for children and families Improve the quality of the job applicant pool for Social Service Case Manager positions Increase retention of new Social Service Case Managers

8. ESP Components Overview of the work Job preview video Self- assessment Application Phone screening On-site assessment

9. ESP Statewide Implementation March 2015 All new case manager job applicants required to complete the ESP process April 2015 Training of hiring managers and other staff involved in employee selection November 2015 “Non- negotiable” memo to Regional-County Directors

10. Initial Evaluation Focus Impact of ESP on reducing early turnover with newly hired employees and on long-term retention of staff. Impact of ESP on the quality of work completed by child welfare staff.

11. Organizational Challenges to Evaluation Inconsistent use of ESP Difficulty obtaining data Changes in staff roles Inter/intra departmental communication and accountability Competing priorities in the Agency

12. Discussion: Addressing Organizational Challenges to Evaluation Discussion: Addressing Organizational Challenges to Evaluation

13. The Good News Is….We Have a Second Opportunity Questions from leadership: Extent of ESP usage in the field Fidelity of usage Need for program updates Short-term impact

14. Group Discussion Questions 1)What organizational challenges to evaluation have you encountered? What did you do to resolve these challenges? How successful were your efforts? 2)What evaluation approaches and/or practices are most effective in fast paced, unpredictable, and changing environments? 3)How do evaluators know: If/when a program is ready for evaluation? If/when the organizational system is ready to engage in the evaluation process?

15. Taking Something Home What is your most important take away from our experiences and today’s discussion? How might you apply what you learned to your future evaluation projects?

16. Selected References Child Protective Services Human Resources, & Champnoise, C. (2007). The RJP tool kit: A how-to guide for developing a realistic job preview. Washington, DC: Cornerstones for Kids. Child Protective Services Human Resources, Champnoise, C., & Masternak, M. (2004, June 17). Realistic job preview: A review of the literature and recommendations. Washington, DC: Cornerstones for Kids. Ellett, A. J., Ellett, C. D., Ellis, J., & Lerner, B. (2009) A research-based child welfare employee selection protocol: Strengthening retention of the workforce. Child Welfare, 88(5), 49-68. Ellett, A. J., Ellett, C. D., Westbrook, T. M., & Lerner, B. (2006). Toward the development of a research-based employee selection protocol: Implications for child welfare supervision, administration, and professional development. Professional Development: The International Journal of Continuing Social Work Education, 9(2), 111-121.

17. Ellett, A. J., Ellis, J. I., Westbrook, T. M., & Dews, D. (2007). A qualitative study of 369 child welfare professionals’ perspectives about factors contributing to employee retention and turnover. Children and Youth Services Review, 29, 264–281. Faller, K. C., Masternak, M., Grinnell-Davis, C., Grabarek, M., Sieffert, J., & Bernatovicz, F. (2009). Realistic job previews in child welfare: State of innovation and practice. Child Welfare, 88(5), 23-47. Graef, M.I., Paul, M.E., & Myers, T.L. (2009). Recruiting and selecting child welfare staff. In F. Alwon, S. Steib, & B. Schmitt (Eds.), On the job in child welfare (pp. 119-161). Arlington, VA: CWLA Press. Graef, M.I. & Potter, M.E. (2002). Alternative solutions to the child protective services staffing crisis: Innovations from industrial/organizational psychology. Protecting Children, 17 (3), 18-31.

18. Contacts Janice Saturday Georgia Division of Family and Children Services Janice.Saturday@dhs.ga.gov Deidre Carmichael Georgia State University dhcarmichael@gsu.edu