1. How Can Evaluation Efforts Illuminate Systems Change in Courts, Tribes, and States? 19 th Annual National Human Services Training Evaluation Symposium University of California, Berkeley May 24, 2016 James DeSantis, Ph.D. Anita P. Barbee, MSSW, Ph.D.

2. Consultative Workshop

3. Background: Evaluation of Children’s Bureau’s Training and Technical Assistance T/TA system funded by CB to support State and Tribal child welfare systems for several decades First national evaluation of the T/TA network (i.e. National Resource Centers), 2004- 2009 (Cunningham, Barbee, & RoBen Associates, LLC) Second national evaluation of expanded T/TA network (i.e., 5 Implementation Centers and 10 National Resource Centers), 2008– 2014 (James Bell Associates & ICF International) Current national evaluation of CB’s new T/TA structure: Capacity Building Collaborative, comprised of 3 national Centers (States, Tribes, Courts), 2014-2019

4. High/Low Capacity High/Low Utilization Conceptualization States with Low Level of Capacity States with High Level of Capacity States with Low level of TA Utilization Low Capacity, Low Utilization High Capacity, Low Utilization States with High level of TA Utilization Low Capacity, High Utilization High Capacity, High Utilization

5. Cross-Site Evaluation of ICs and NRCs: Implementation Capacity Implementation capacity was examined in 19 implementation projects across the nation to understand: IC role in building capacities Effects of implementation projects on approaches to systems change Challenges: Lack of baseline data collected in uniform way to address baseline capacities/strengths of CW system Variability in how ICs reported on capacities Data inconsistencies Limitations in drawing conclusions

6. Cross-Site Evaluation of ICs and NRCs: Systems Change

7. Cross-Center Evaluation of the Capacity- Building Collaborative Captures more data about the initial capacity of each jurisdiction through administration of a Foundational Assessment tool at baseline Carefully measures dosage of various types of capacity building efforts Measures short-term and medium-term outcomes resulting from Centers’ work with jurisdictions Enables the linkage between dosage and outcomes in high versus low capacity jurisdictions to be explored Measures increases in capacity from baseline to follow-up after jurisdictions engage in different types of capacity-building work with Centers

8. Research Questions address the following topics: Service Interventions What are the capacity building service interventions delivered by the centers? To what degree are they evaluable? To what degree are centers following common protocols? To what degree are service interventions being delivered/performed as designed? What is being received (i.e., intensity, frequency)? Service Quality How satisfied are recipients with services received? Service Interventions Outcomes How effective are the service interventions? Which service approaches are most effective and under what conditions? Relationships Across and Within Centers How and to what extent are key partners across and within the Centers collaborating?

9. MethodDescription Leadership InterviewsTelephone interview with State and Tribal CW directors and CIP directors Satisfaction surveysAdministered by Center evaluators and by Cross-Center team to direct recipients of interventions Knowledge gain testsAdministered by Center evaluators– largely constituency services Capacity surveyAdministered to direct recipients of intensive tailored consultation; survey includes Innovation-specific items and Change Management items

10. MethodDescription Foundational Assessment SurveyAdministered at baseline and annually Focus groups and interviewsAnnual focus groups/interviews with leadership and staff from Centers and Federal staff Center Collaboration SurveyAdministered annually to staff from each Center CapTRACK systemCaptures inputs, assessment, work plan, frequency & intensity of service delivery; used by Center staff Document reviewReviews of Centers’ written protocols & procedures, progress reports, evaluation reports Sub-studies of priority topicsCourt Improvement Program; Implementation of P.L. 113-183 (Human Trafficking); Operationalization of Change Management

11. Assessment of Foundational Capacity--example

12. Capacity Survey-- example

13. Framework for Cross-Center Evaluation

14. Implications of Evaluation Results of the Cross-Center evaluation will have implications for: How to prepare a jurisdiction so that it is ready to receive assistance and build capacity The effectiveness of the capacity building method chosen by the Centers The effectiveness of service interventions in contributing to desired change CFSR policy and practice State legislation

15. Potential or Existing Barriers

16. What do you think are the key variables that we should be measuring that impact systems change? What role do you think training and technical assistance play in ensuring that systems change occurs? What role do you think the evaluation of training, technical assistance and other capacity building strategies has on understanding systems change? What are some of the most important predictive models that should be explored in making the connection between training, technical assistance, and other capacity building strategies and systemic change in child welfare?

17. Jim DeSantis James Bell Associates desantis@jbassoc.com desantis@jbassoc.com Anita Barbee Kent School of Social Work University of Louisville anita.barbee@Louisville.edu anita.barbee@Louisville.edu