1. Evaluating Organizational Change in a Complex Agency Environment Cindy Parry, Ph.D. CF Parry Associates, Inc. Helen Cahalane, Ph.D. Child Welfare Education and Research Programs University of Pittsburgh, School of Social Work 19 th Annual National Human Services Training Evaluation Symposium May 26, 2016

2. What is Organizational Effectiveness?  A step by step approach to improving organizational capacity, performance and impacts on those served  Tools and techniques designed with and through collaboration with human services agencies  Group facilitation organized around a three-tier teaming structure and the DAPIM™ critical thinking and continuous improvement cycle  Organization and OE facilitator mutually set a scope of work  Work may include some or all of the DAPIM™ model, as well as specific activities geared toward the work being done

3. DAPIM™ Model Work Products Define Work products – baseline surveys and measures, alignment notes, initial feelings, ground rules, defined areas for improvement Assess Work products – findings: strengths and gaps, root causes and general remedies Plan Work products – quick wins, mid- and long- term improvement plans, communication and capacity plans Implement Work products – team activities, action plans, charters for working teams, communication efforts Monitor Work products – monitoring quick wins, and other changes, evaluations and measures of progress and impact Performance & Capacity

4. Applying the DAPIM™ Model to Child Welfare Organizations: A State Example  Pennsylvania was an “early adopter” of OE interventions in 2004  PA’s Child Welfare Resource Center began a transformation initiative following the first round of the CFSR, moving from a traditional child welfare training program to a system with the capacity to support agencies in improving overall organizational effectiveness  Practice Improvement Specialists coached and mentored in the OE/DAPIM model  OE facilitation first introduced to county child welfare agencies as part of practice improvement efforts identified in PA’s Practice Improvement Plan (PIP)  Key areas identified: family engagement, youth engagement, building systems of care, staff retention, quality visitation, risk/safety assessment

5. Supporting Organizational Change in PA Child Welfare Systems  DAPIM™ model used to identify and support local child welfare agencies in organizational change efforts  In conjunction with Quality Service Review (QSR) and Continuous Quality Improvement (CQI) processes  OE facilitation and technical assistance provided to child welfare agencies to support local level system change  County-level data (quantitative and qualitative) used to identify strengths, challenges in practice, and desired child/family/organizational outcomes  County Improvement Plans developed through a facilitated OE/DAPIM process designed to strengthen the agency’s capacity to address change

7. Practice Challenges and Lessons Learned  OE work becomes a “way of doing business” for agencies that employ it over time  Use of the DAPIM model involves trusting the process  Change takes time  Importance of leadership, teaming and engagement  “Results Dip” is a normal part of the change effort  Facilitation of the DAPIM process requires a commitment to professional development  OE work is a parallel process with effective front-line practice

8. The Evaluation  Part of a larger study of the effectiveness of the APHSA organizational effectiveness model  In-depth case study of two PA counties engaged in facilitated organizational development work  Assessed:  Achievement of organizational goals  Change in organizational capacities and functioning in areas targeted by the county’s continuous improvement plan  The value of enhanced monitoring using Goal Attainment Scaling

9. The Evaluation Context  Small sample of eligible counties with different identified performance needs and improvement plan goals  An intervention that is designed to be collaborative, flexible, and dynamic  Work taking place in a complex environment characterized by  multiple changes in the state’s child welfare laws  implementation of a new CWIS system for child abuse reporting  financial uncertainties as a result of delays in passage of state budget

10. Evaluation Design  Mixed methods  Review of documents (e.g. plans, tools, and after action reviews)  Pre and Post comparison of scores on the Organizational Readiness for Change (ORC)  Tracking progress toward mid and long range goals using Goal Attainment Scaling (GAS)  Participant interviews providing feedback on the GAS process and goal attainment

11. What is Goal Attainment Scaling?  Flexible and strengths-based way to measure and document incremental change for individuals or teams  Developed in Mental Health to provide a common framework for measuring the progress of patients working on individual therapy goals  Progress is tracked on collaboratively developed, anchored measurement scales customized to the goals of the individual or group

12. What are the advantages of using it?  Both an evaluation tool and a monitoring tool  Can be applied to a wide range of goals and activities while still providing a common framework for summarizing very different project outcomes.  Compatible with a CQI approach: As goals are achieved, new goals can be set that build on the previous goal attainment  Parallel process involving many of same skills as case planning

13. How are scales developed? All scales use the same 5 point format Development of the scales typically follows the sequence: 1.Describe the expected level of performance: where the individual or team expects to be in relation to the plan goal at the end of some period of time 2.Define what it would look like to do a little better than expected and a lot better than expected 3.Define what it would look like to do a little worse than expected and a lot worse than expected 4.Check the scale for gaps, overlaps or shifts in focus (e.g. from quality to frequency) 5.Identify strategies for achieving the goal 6.Identify sources of performance feedback (e.g. supervisors, peers, routinely collected records and data)

14. What do scales look like? Goal #1: Fathers will be engaged at the point of assessment/investigation. Scale 1: Fathers will be contacted either by a face to face visit or by phone within the first 30 days of an initial intake Rating Scale 2: Intake screeners will obtain and identify fathers’ contact information on initial contact or by whatever means possible. Rating 5 Much more than expected level of outcome More than 61% of the timeMore than 71% of the time 4 Somewhat more than expected level of outcome 46- 60% of the time51% - 70% of the time 3 Expected level of outcome25- 45% of the time26% - 50% of the time 2 Somewhat less than expected level of outcome 6- 24% of the time21% - 25% of the time 1 Much less than expected level of outcome Less than 6% of the timeXLess than 20% of the timeX

15. How are scales scored?  Multiple options  Traditionally scales are scored as -2, -1, 0, 1, 2, with 0 as the expected level, 1 as “a little better”, 2 as “a lot better”, -1 as “a little worse” and -2 as “a lot worse”  Other transformations have been used and we used 1 to 5 with 3 as the expected level to avoid negative numbers  Composites are created and standardized to account for differences in numbers of scales being worked on by different individuals or teams  Averages  T-scores (mean of 50 and standard deviation of 10)  Scales may be weighted to reflect different levels of importance or unweighted

16. What data does it provide? Focus areaGoalRatings Dec. 2013 July 2014 Jan. 2015 March 2015 Father Engagement 1.All fathers will be contacted either by a face to face visit or by a phone contact within the first 30 days of an initial intake. 1543 2.Intake screeners will obtain and identify fathers’ contact information on initial contact or by whatever means possible. 1555 Sum21098 Mean for goal154.54

18. What did we learn about GAS?  GAS was a more sensitive measure of change for the organizational outcomes of interest than the ORC instrument  Team members saw value in the process despite an initial learning curve and planned to use it again  It helped teams define desired changes concretely and specifically and break down large goals into manageable pieces  Helped teams see the value of data in decision making  It provided a mechanism for tracking progress and for opening conversations about a lack of progress, including possible reasons and remedies  Team members thought it worked best for things that were: concrete, less subjective, easily defined and counted, under the control of a single supervisor or a group of supervisors within the same area, or where their were existing process or data to build on.

19. For Further Reading Kiresuk, T. J., Smith A., Cardillo, J. E. (1994). Goal Attainment Scaling: Applications, Theory, and Measurement. Hillsdale, N.J. : L. Erlbaum Associates American Public Human Services Association (2013). Organizational effectiveness handbook, Version 5.0. Washington, DC: author.

20. Questions for Discussion 1.What uses do you see for GAS? 2.What are its limitations? 3.How can we control for local-level history, team relationships and agency context? 4.What level of dosage is best for maintaining momentum and diffusing the use of GAS in change efforts beyond a single agency unit or department? 5.What degree of ongoing communication is needed to keep the spotlight on the change process? 6.Because organizational effectiveness goals and interventions are highly individualized and flexible, how can evaluation findings be generalized when using GAS as an evaluation method? 7.How can this work be translated into wider organizational change?