1. Improving Practice to Wraparound Through Use of Fidelity Measures
Enrica Bertoldo, Quality Support Manager
Eleanor Castillo, Director of Outcomes & Quality Assurance
Mary Ann Wong, Research Specialist
Veronica Padilla, Manager of Outcomes & Evaluations
26th Annual California Mental Health Advocates for Children & Youth Conference, Asilomar, California
May 4, 2006

2. EMQ Children & Family Services
Our Mission
To work with children and their families to transform their lives, build emotional, social and familial well-being, and to transform the systems that serve them.
Our Philosophy
Family Voice  Team Based
Collaborative/Integrative  Community-Based
Culturally Competent  Individualized
Strength-Based  Natural Supports
Persistence  Outcomes-Based
Background on EMQ
EMQ Children & Family Services (a.k.a. Eastfield Ming Quong) is a leading provider of children's mental health, Wraparound, and social services in No. & So. California.. Headquartered in Campbell, California, EMQ offers an integrated network of services that range from community-based services such as Wraparound to residential treatment, support and educational services, and foster placements. Our philosophy incorporates many of the elements and values seen in wraparound into our other programs. We are recognized for innovation in family-centered, strengths-based programs for children and adolescents. We have wraparound in 3 sites…
Community Based (Wraparound, Matrix, SOC): Wraparound Services in 3 sites: Santa Clara, Sacramento, and San Bernadino
Residential Treatment (Long-term, Short-term)
Support Services (Outpatient, Giaretto)
School-based (Day Treatment Program)
Addiction Prevention Services (School-based and Agency-Based Prevention Programs)
24 Hour Mobile Crisis
Foster Family Agency

3. Purpose
Given the complexity of Wraparound, existing fidelity measures are utilized to improve practice.
This workshop will share successes and challenges in the implementation of fidelity measures (WFI-3.0 & WOF-2.0) that affect practice and supervision to Wraparound.
A pilot in utilizing a Wraparound supervisory adherence measure (W-SAM) will also be discussed.

4. What is Wraparound?
Team-based, individualized service planning process
Principles of community-based, strength-based, family-centered, and culturally competent
Wraparound is described as a promising practice (Burns, Hoagwood, & Maultsby, 1998)

5. 11 Core Elements of Wraparound
Voice and Choice
Youth/Family Team
Community-Based Services/Supports
Cultural Competence
Individualized Services/Supports
Strengths-Based Services/Supports
Natural Supports
8. Continuation of Services/Supports
Collaboration
Flexible Resources/ Funding
Outcome-Based Services/Supports

6. Essential Elements of Wraparound
Element 1: Voice and Choice
Families must be full and active partners at every level of the wraparound process. If the team cannot reach consensus, the final decision should be up to the caregiver.
Element 2: Youth and Family Team
Wraparound is a team-driven process involving caregivers, youth, natural supports, and community services working together to develop, implement, and evaluate the individualized plan.
Element 3: Community-based Services and Supports
Services and supports that the youth and family receive should be based in their community. The family should not have to leave their community if more restrictive services are necessary.
Element 4: Cultural Competence
The team should not only be respectful of the family’s beliefs and traditions, but also actively seek to under-stand the family’s unique perspectives and convey them to others.

7. Element 5: Individualized Services
This means that services and supports are tailored to the unique situation, strengths, and needs of teach individual, and may involve existing categorical services and informal sup-ports; modifying existing services and supports; and or creating new services and supports. Further, the team should create a specific plan to meet the family’s goals and crisis/ safety plan to manage potential emergencies.
Element 6: Strengths-Based Services
The focus of the team should be on what is working and going well for the family. While goals may be drawn up based on the family’s needs, the plan should capitalize on the family’s positive abilities and characteristics.
Element 7: Natural Supports
Services and supports should reflect a balance of formal and informal community and family supports rather than a reliance on formal professional services.
Element 8: Continuation of Care
Services and supports must be provided unconditionally. In a crisis, services and supports should be added rather than placing the youth with a new provider.

8. Element 9: Collaboration
The team should coordinate services and supports so they seem seamless to the family rather than disjointed.
Element 10: Flexible Funding and Resources
Successful wraparound teams are creative in their approach to service delivery and have access to flexible funds and resources to implement their ideas.
Element 11: Outcome-Based Services
Specific, measurable out-comes should be monitored to assess the youth and family’s progress toward goals.
The description of each element is taken directly from Wraparound Fidelity Index 3.0 (2002), produced by the Wraparound Evaluation and Research Team at the University of Vermont.

9. What is Fidelity to Wraparound?
Treatment Fidelity = The degree to which a program is implemented as intended (Rast & Bruns, 2003; Moncher & Prinz, 1991)
Adherence to the 11 Core Wraparound Elements (e.g., WFI-3.0; Suter et al., 2002)
Adherence to the Child and Family Team Process (e.g., WOF-2.0; Epstein et al., 2002)

10. Why Measure Fidelity?
Measuring fidelity is essential to families, providers, policy makers and researchers
Without measuring fidelity, how do you ensure the Wraparound process is occurring?
Without outcomes, the Wraparound process is just one more fad
High fidelity has been associated with positive outcomes in some studies
It improves quality assurance
It helps agencies secure more funding by proving outcomes
It can even help create legislation on how families and kids receive services

11. More Why’s…..
All nationally recognized Wraparound programs have extensive fidelity measurements in place.
Why?
Without fidelity measurement standards firmly in place, Wraparound programs and team members have difficulty knowing what they are doing well and what they need to improve in order to achieve the ideal as presented in the Wraparound vision.

12. Fidelity Implementation Measures
Wraparound Fidelity Index-3.0 (WFI-3.0)
Measures fidelity to Wraparound principles through structured interviews
Administered to Youth, Caregiver, & Resource Facilitator
Higher Score = Greater Fidelity
Wraparound Observation Form-2.0 (WOF-2.0)
Measures fidelity to Wraparound through observation of the Child and Family Team meetings

13. Fidelity to Wraparound
EMQ Sacramento has collected the Wraparound Fidelity Index (WFI) since 2001
Data from WFI-3.0: January 2003 – September 2005
EMQ Sacramento has collected a fidelity measure since 2001, beginning with the WFI 2.0 and then the WFI 3.0. Our measure of fidelity in this study was the WFI Version 3. We conduct WFI Interviews with the Wraparound resource facilitator, caregiver, and youth six months after admission and every six months thereafter for each youth until discharge.
(The responses from each interview results in a total fidelity score, ranging from 0 to 8, with 0 indicating low adherence to the Wraparound philosophy and 8 indicating high fidelity to the Wraparound philosophy. An overall fidelity score is also calculated by combining the reports of the three respondents)
Participants
Participants in this study included youth (N=146), caregivers (N=124), and resource facilitators (N=183) for families receiving Wraparound services by a large family service agency in the Sacramento region.

14. A Comparison of EMQ Data & National Data

15. WFI Total Scores by Respondent: EMQ & National Data
*National WFI-3 Dataset came from the Wraparound Evaluation and Research Team presentation for the 18th Annual Research Conference on Systems of Care and Children's Mental Health in Tampa on 3/7/05 by Eric J. Bruns, Ph.D. for "Is it Wraparound Yet?" - Bootstrapping wraparound fidelity standards using the WFI .

16. WFI Elements: EMQ & National Data
*National WFI-3 Dataset came from the Wraparound Evaluation and Research Team presentation for the Technical Assistance Partnership Webinar on 4/5/04 by Eric J. Bruns, Ph.D. for "Ensuring High-Quality Wraparound"

17. WFI Elements: EMQ & National Data
*National WFI-3 Dataset came from the Wraparound Evaluation and Research Team presentation for the Technical Assistance Partnership Webinar on 4/5/04 by Eric J. Bruns, Ph.D. for "Ensuring High-Quality Wraparound"

18. The Relationship Between Fidelity & Outcomes

19. The Relationship Between Fidelity & Outcomes
Previous research has found an association between greater Wraparound fidelity and better child and family outcomes (Bruns, 2004), but the relationship has not been clearly understood
EMQ (2005) completed a study to further understand the relationship between fidelity to the 11 core elements of Wraparound and treatment outcomes
Over the past couple of years, EMQ has been increasingly interested in examining the relationship between fidelity and outcomes. Previous studies have used the WFI to demonstrate a relationship between fidelity and outcomes (Bruns, 2004, presentation given at 11th Annual Building on Family Strengths Conference) but the exact nature of the relationship has not been clear.(Past studies have shown WFI Scores to be associated with BERS, CAFAS, and CBCL concurrent and future outcomes) The purpose of this study was to further understand the relationship between fidelity to the 11 core elements of wraparound and treatment outcomes. Specifically, we were interested in knowing what elements of the wraparound process, if any, are associated with outcomes.

20. Demographics Average Age at Admission: 14 years 63% Male; 37% Female
62% Caucasian; 26% African-American; 8% Latino; 2% Asian/Pacific Islander; 1% Native-American; 1% Other
Average Length of Stay: 15 months
The demographics for youth in this sample are as follows:
The average age of these youth at the time of admission was 14 years (SD=2.55). Gender composition was 63% male and 37% female. Ethnicity was 62% Caucasian, 26% African-American, 8% Latino, 2% Asian/Pacific Islander, 1% Native-American, and 1% other. The average length of participation in Wraparound services was 15 months (SD=11, range months).

21. Fidelity to Wraparound
Implementation Measure:
Wraparound Fidelity Index – 3.0 (WFI-3.0)
Collection since 2003
Sample in study included the following:
146 WFI’s collected from Youth
124 WFI’s collected from Caregivers
183 WFI’s collected from Resource Facilitators
EMQ Sacramento has collected a fidelity measure since 2001, beginning with the WFI 2.0 and then the WFI 3.0. Our measure of fidelity in this study was the WFI Version 3. We conduct WFI Interviews with the Wraparound resource facilitator, caregiver, and youth six months after admission and every six months thereafter for each youth until discharge.
(The responses from each interview results in a total fidelity score, ranging from 0 to 8, with 0 indicating low adherence to the Wraparound philosophy and 8 indicating high fidelity to the Wraparound philosophy. An overall fidelity score is also calculated by combining the reports of the three respondents)
Participants
Participants in this study included youth (N=146), caregivers (N=124), and resource facilitators (N=183) for families receiving Wraparound services by a large family service agency in the Sacramento region.

22. Outcome Measures at Discharge
Child and Adolescent Functional Assessment Scale
(CAFAS; Hodges, 2000)
Measures youth’s level of functioning; completed by Resource Facilitator
Lower Score = Greater Functioning
Exit Total CAFAS Score
Child Behavior Checklist (CBCL; Achenbach, 2001)
Measures youth’s level of behavior problems; administered to Caregiver
Lower Score = Lower Behavior Problems
Exit Total, Externalizing, & Internalizing CBCL Scores
Living Arrangement at Discharge: Community or Facility
We routinely collect the CAFAS & CBCL at admission, every 6 months, and discharge. For this study we looked at CAFAS Total score at discharge, the CBCL Total, Externalizing, and Internalizing scores at discharge. In addition, we examined the youth’s living arrangement at discharge, living in the community or facility. We define community as living with bio/adoptive parents, kinship care, foster family, or independent living program.
(The Child and Adolescent Functioning Assessment Scales (CAFAS; Hodges, 2000) measure a youth’s emotional and behavioral functioning across eight domains (School/Work Role Performance, Home Role Performance, Community Role Performance, Behavior Toward Others, Moods/Emotions, Self-Harmful Behavior, Substance Use, and Thinking). Subscale scores on these dimensions are summed to produce a total CAFAS score, with a higher value indicating more severe impairment.
(The Child Behavior Checklist (CBCL; Achenbach & Rescorla, 2001) measures a youth’s competence and problem behaviors and is administered to caregivers. Scoring produces a Total Problems score as well as raw scores within two groupings of syndromes (Internalizing and Externalizing Problems) A higher value indicates more severe impairment.

23. Correlations Among WFI Scores and Outcomes at Discharge
CBCL
Total
CBCL Externalizing
CBCL Internalizing
CAFAS
Living in Community
1. Voice and Choice
.12
.07
.15*
.47
2. Youth and Family Team
-.04
.01
.10
3. Community-Based Services
-.19*
-.25**
-.21*
-.40**
.21**
4. Cultural Competence
.06
.02
.15
5. Individualized Services
.09
.11
.05
6. Strengths-Based
-.10
-.09
-.14
.08
7. Natural Supports
-.12
-.13
-.07
.13
8. Continuation of Services
-.06
-.02
9. Collaboration
-.03
-.17*
-.11
10. Flexible Resources/Funds
.04
.20**
11. Outcome-Based Services
-.16
-.18*
.14*
12. RF Total Fidelity
-.20
-.08
-.15
.16
13. Caregiver Total Fidelity
14. Youth Total Fidelity
-.01
-.05
15. WFI Total Fidelity
Non-parametric correlations (Spearman’s rho) were conducted to demonstrate relationships between WFI total fidelity and element scores and outcome measures at discharge. The average score of each element across the respondents was calculated for the analysis (youth does not have elements 9, 10, 11). The analysis also factored in the pre-test scores on the outcome measures to account for the differences in the pre and post scores on youth functioning.
Significant Correlations:
Between fidelity to Community-Based Services and Supports and various outcome measures:
CBCL Total Problem Score at Discharge ( r = -.189, p<.05) N=143)
CBCL Externalizing Problem Score at Discharge, (r = -.254, p<.01 N=143)
CBCL Internalizing Problem Score at Discharge (r = -.210, p<.05 N=143),
CAFAS Total Score at Discharge (r = -.40, p<.01 N=171)
Living in Community (r = .21, p<.01 N=?)
Other significant relationships occurred between Natural Supports & CBCL Total (r=-.19, p<.05 N=143); Collaboration & CBCL Internalizing (r = -.17, p<.05 N=142); Outcome-Based Services & CBCL Internalizing (r = -.18, p<.05 N=141) & CAFAS (r=.14, p< .05, N=?) & Living in Community (r=.21, p<.01 N=?) ; Flex Funds & Living in Community (r=.20, p<.01 N=?); Voice-Choice & CAFAS (r=.15, p<.05, N=?)
The difference in the pre and post scores on youth functioning did not impact the relationship with fidelity.
Note: *p<.05; **p<.01

24. Community-Based Services
Significant Results
Community-Based Services
CBCL Total
-.19*
CBCL Externalizing
-.25**
CBCL Internalizing
-.21*
CAFAS Total
-.40**
Living in Community
.21**
Note: *p<.05; **p<.01
Greater community-based services is related to lower impairment on the CBCL and CAFAS at discharge
Greater community-based services is related to community living arrangement at discharge
As you can see from the table, community-based services and supports had the most consistent relationship to outcomes. Significant relationships were found with all 5 outcomes measures used in this study….
Means:
CBCL Total= (SD=19.29, N=242)
CBCL Externalizing=87.16 (SD= 19.71, N=242)
CBCL Internalizing= (SD=25.20, N=242)
CAFAS(7)=94.78 (SD=43.78, N=324)
WFI Total=5.81 (SD=.82, N=186)
RF Total WFI =6.30 (SD=.78, N=183)
CG Total WFI=5.39 (SD=1.21, N=124)
Youth Total WFI= (SD=1.22, N=146)
Elements are the average across 3 respondents
Community-Based=5.00 (SD=1.72, N=186) Flexible=5.2 (SD=1.32, N=184)
Voice/Choice=6.66 (SD=1.16, N=184) Outcomes=6.83 (SD=1.37, N=184)
CFT=4.5 (SD=1.26, N=186)
Cultural Com=6.64 (SD=1.04, N=186)
Individualized=6.85 (SD=1.28, N=186)
Strengths=6.43 (SD=1.14, N=186)
Natural Supports=3.78 (SD=1.52, N=186)
Continuation=6.04 (SD=1.30, N=186)
Collaboration=6.10 (SD=1.34, N=185)

25. Results
Greater fidelity to the provision of Community Services/Supports in Wraparound is related to positive outcomes
No other elements had as significant correlations to outcomes as Community Services/Supports
No significant relationship found between Total Fidelity Scores and outcomes at discharge
Correlations were consistently low even when significant
In summary, greater fidelity to community-based services to wraparound is related to positive outcomes. No other elemnts had as significant or consistent relationships to outcomes. Unlike previous studies, no relationship was found between total fidelity scores and outcomes at discharge. Other significant correlations were found but not as consistent as Community-based services and supports. Significant findings with use of flex funds and focus on outcome-based services with the community living arrangement at discharge.

26. Implications
Implications for teams to support a youth’s school attendance and involvement in work, training, and other community activities (e.g., church, sports, art, etc.) as well as support community-based living situations for youth (as an alternative to residential or institutional care) in order to improve outcomes.
Implications for teams to consider the questions that comprise Element 3 of the WFI- Community-Based Services and Supports
How many hours a week does the youth spend at a regular community school, working at a paying job, in a job training program?
Are services and supports for the family accessible?
Does the team help the youth get involved with activities in the family’s community? (Give 2 examples)
Different placements for youth over past 30 days?
This study’s finding that effective Wraparound services focus on the enhancement of community adjustment is consistent with the results of a study of system reform performed by Hyde, Burchard, and Woodworth (1996). These investigators assessed community adjustment and involvement of four groups of high-risk youth, two of which received Wraparound services. Positive findings revealed that youth in the Wraparound groups displayed higher levels of adjustment to their communities (as evidenced by living in the community, attending school, working, engaging in relatively low rates of self-harmful behavior). Furthermore, a higher percentage of these youth participated in community activities, such as church, sporting events, and volunteer work.

27. Challenges of the WFI Direct feedback to teams was not provided
The utility of the WFI in its relationship to outcomes was limited based on EMQ’s analysis of the data
Time intensive and limited resources available to conduct 3 structured interviews per youth

28. EMQ Practice Standards Initiative

29. Initiative to Improve Practice
Quality Improvement Project launched in January 2004 to implement practice standards and feedback tools across positions
Family Facilitator
Family Specialist
Family Partner
Goal was to standardize job positions, provide role clarity, and improve consistency in job performance and supervision
Use of the WOF-2.0 to observe the CFT process
Development of practice standards for facilitators and other team members, including supervisors, will look at adherence to fidelity to wraparound and outcomes.

30. Example of EMQ Feedback Tool: Family Facilitator
Scoring:
Opportunity for Improvement (1)
Acceptable (2)
Exemplary (3)
Not Observed (*)
3 Dimensions:
A. Families receive services that are consistent with wraparound values
B. Families receive services that are collaborative, integrated and adhere to best practices
C. Children and Families served are safe and stable at home and in the community

31. Facilitator Field Feedback Tool
A. Families receive services that are consistent with wraparound values
1. Arrived on time and was prepared for the CFT
2. Started the CFT with “what’s working” to maintain strength based focus and future orientation
3. Facilitated the CFT to validate and incorporate ideas, requests, and concerns of all members
4. Evidence that needs and strategies were reviewed within the CFT to assure the family voice and preference guide all activities
5. Evidence that plans were reviewed for progress on current measurable, observable goals and updated as needed to address new and changing needs
6. Evidence that the next CFT and staff visits were scheduled according to family preference

32. Facilitator Field Feedback Tool
B. Families receive services that are collaborative, integrated and adhere to best practices
1. For children open six months or longer, evidence that the CFT was comprised of at least 50% non-paid informal members
2. Facilitated the CFT to allow pertinent agenda items to be covered according to family preference and time availability
3. Demonstrated engagement and rapport with all team members including system partners
4. Therapy, medications and adjunctive services were evaluated relative to plan goals as needed
Specific tasks were assigned
C. Children and Families served are safe and stable at home and in the community
1. Evidence that team has a plan B to back up plan A
2. If the CFT was for a child in out of home placement, this was addressed as a priority with specific plan, tasks, and target date
3. Evidence that strategies and interventions developed emphasize the use of typical resources in the child’s community

33. EMQ Feedback Tools
Data analysis showed that the tools were not measuring what was intended; therefore, tools were discarded
Led to an examination of the national standards for Wraparound
EMQ Sacramento decided to collect data on the Child and Family Team process using the Wraparound Observation Form – Version 2.0 (WOF-2.0)

34. Wraparound Observation Form – Second Version (WOF-2)
Measure to assess the implementation of the Wraparound approach by direct observation of youth and family team meetings
WOF has demonstrated good inter-rater reliability (kappa=.886; Nordess & Epstein, 2003)
48 items that measures 8 characteristics:
Community-Based  Unconditional Care
Individualized  Measurable Outcomes
Family-Driven  Management of Team Meeting
Interagency Collaboration  Care Coordinator

35. Implementation of the WOF
WOF data collection began July 2005
Feedback provided in individual supervision if completed by direct supervisor
Verbal feedback provided to facilitators after the child and family team meeting if completed by neutral observer. Copy of tools also provided to supervisors

36. Wraparound Observation Form – Second Version (WOF-2)
Community-based resources (5 items)
1. Information about resources / interventions in the area is offered to the team.
2. Plan of care includes at least one public and/or private community service/resource.
3. Plan of care includes at least one informal resource.
4. When residential placement is discussed, team chooses community placements for child(ren) rather than out-of-community placements, whenever possible.
5. Individuals (non-professionals) important to the family are present at the meeting.

37. Individualized services for the family (9 items)
If an initial plan of care meeting, the parent is asked what treatments or interventions he/she felt worked/didn’t work prior to WRAP.
Facilitator advocates for services and resources for the family (e.g., identifies and argues for necessary services).
All services needed by family are included in plan (e.g., no needed services were not offered).
Barriers to services or resources / interventions are identified and solutions discussed.
The steps needed to implement the plan of care are clearly specified by the team.
Strengths of family members are identified and discusses at the meeting.
Plan of care that includes life domain(s) goals, objectives, and resources/interventions is discussed (or written).
Plan of care goals, objectives, or interventions are based on family/child strengths.
Safety plan/Crisis plan developed/reviewed.

38. Family-driven services (10 items)
15. Convenient arrangements for family’s presence at meeting are made (e.g., location, time, transportation, day care arrangements).
16. The parent/child is seated or invited to sit where he/she can be included in the discussion.
17. Family members are treated in a courteous fashion at all times.
18. The family’s perspective is presented to professionals from other agencies.
19. The family is asked what goals they would like to work on.
20. The parent is asked about the types of services or resources/interventions he/she would prefer for his/her family.
Family members are involved in designing the plan of care.
In the plan of care, the family and team members are assigned (or asked) tasks and responsibilities that promote the family’s independence (e.g., accessing resources on own, budgeting, maintaining housing).
The team plans to keep the family intact or to reunite the family.
Family Members voice agreement/disagreement with plan of care.

39. Interagency collaboration (7 items)
Staff from other agencies who care about or provide resources/interventions to the family are present at the meeting.
Staff from other facilities or agencies (if present) have an opportunity to provide input
Informal supports (if present) have an opportunity to provide input.
Problems that can develop in an interagency team (e.g., turf problems, challenges to authority) are not evident or are resolved.
Staff from other agencies describe support resources interventions available in the community.
Statement(s) made by a staff member or an informal support indicate that contact/ communication with another team member occurred between meetings.
Availability of alternative funding sources is discussed before flexible funds are committed.

40. Unconditional Care (3 items)
Termination of Wrap services is discussed because of the multiplicity or severity of the child’s/family’s behaviors /problems.
Termination of other services (not Wrap) is discussed because of the multiplicity or severity of the child’s/family’s behaviors /problems.
For severe behavior challenges (e.g., gangs, drugs), discussion focuses on safety plans/crisis plans (e.g., service and staff to be provided) rather than termination.
Measurable Outcomes (3 items)
35. The plan of care goals are discussed in objective, measurable terms.
36. The criteria for ending Wrap involvement are discussed.
Objective or verifiable information on child and parent functioning is used as outcome data.

41. Management of team meeting (5 items)
38. Key participants are invited to the meeting (e.g., family members, CPS worker, teacher, therapist, others identified by family).
39. Current information about the family (e.g., social history, behavioral and emotional status) is gathered prior to the meeting and shared at meeting (or beforehand).
40. All meeting participants introduce themselves (if applicable) or are introduced.
41. The family is informed that they may be observed during the meeting.
42. Plan of care is agreed on by all present at the meeting.

42. Facilitator (6 items)
Facilitator makes the agenda of meeting clear to participants.
Facilitator reviews goals, objectives, interventions and/or progress of plan of care.
Facilitator directs (or reflects) team to discuss family/child strengths.
Facilitator directs (or reflects) team to revise/update plan of care.
Facilitator summarizes content of the meeting at the conclusion of the meeting.
Facilitator sets next meeting date/time.

43. EMQ WOF-2 Results: Total Element Scores (N=30)

44. Results of WOF
EMQ Sacramento shows higher scores on elements of Family-driven, Collaboration, Unconditional Care & Management of CFT Meetings
EMQ Sacramento continues to be challenged by lower scores on Community-based Resources, inclusive of natural supports
In contrast to the results of the WFI for Outcomes (high scores), the WOF showed low scores on this element
Slight increase in Total WFI scores since the implementation of the WOF

45. Back to the WFI Could the use of the WOF impact WFI scores?
A look at WFI scores across time

46. WFI Total Scores Across Time
WOF Implementation Begins 7/05

47. WFI Respondent Scores Across Time
High Scores on Voice and Choice, Cultural Competence, Individualized Services, Strengths,
Outcome-Based Services
Low Scores on Natural Supports, Community-Based Services, and CFT Element

48. Challenges of the WOF
Limited resources in staff to become neutral observers
Managers wanted to use the WOF as a supervision tool
Willingness and comfort level of staff to participate in direct observation
Staff perception of fear of families being judged by observers
Sponsorship by management
Competing priorities in data collection

49. Next Steps
Supervisors will continue to utilize the WOF to provide feedback
Operational realities limit the use of neutral observers
EMQ will no longer collect the WFI due to limitations in resources, the utility of the tool for feedback, and its relationship to outcomes
Continue to develop practice standards

50. References
Achenbach, T. M., & Rescorla, L.A. (2001). Manual for the ASEBA school-age forms & profiles. Burlington, VT: University of Vermont, Research Center for Children, Youth, & Families.
Epstein, M.H., Nordess, P.D., & Hertzog, M. (2002). Wraparound Observation Form-Second Version.
Epstein, M.H., Nordess, P.D., et al. (2003). Assessing the Wraparound Process During Family Planning Meetings.
Hodges, K. (2000). Child and Adolescent Functional Assessment Scale (CAFAS). CAFAS Self-Training Manual and Blank Scoring Forms (2003). Ann Arbor, MI
Nordness, P.D., & Epstein, M.H.(2003). Reliability of the Wraparound Observation Form Second Version.
Suter, J., Force, M., Bruns, E., Leverentz-Brady, K., & Burchard, J. (2002). Manual for training, administration, and scoring of the WFI 3.0. Burlington: University of Vermont.

51. Contact Information
Enrica Bertoldo
Quality Support Manager, Sacramento
Mary Ann Wong
Research Specialist, Sacramento

52. Wraparound Supervisor Adherence Measure (W-SAM): A Pilot
Eleanor Castillo, Ph.D.
Veronica Padilla

53. Rationale Supervision identified as a critical aspect of Wraparound
No tool to date to measure this aspect of Wraparound
Concept adapted from Multisystemic Treatment’s (MST) Supervisor Adherence Measure (SAM)

54. Rationale
WFI-3.0 measures adherence to Wraparound principles from the Youth, Caregiver, and Resource Facilitator perspective
WOF-2.0 measures the Child and Family Team aspect of Wraparound

55. Description Facilitator rates the Clinical Program Manager (CPM)
40 Items
Likert Scale (1 = Never; 5 = Almost Always)
Administered quarterly
Highly reliable (alpha = )

56. Process Implementation
Sponsorship from the Directors
Program quality improvement versus “mandate”
Immediate feedback

57. Quality Management Training Evaluation Performance Appraisal
Overall Program Results
Individual CPM
Performance Appraisal
360
Annual

58. Quality Management 3-Tiered: Agency-wide, Program, and Individual
Training and Coaching
Evaluation
Performance Appraisal
Quarterly 360 feedback
Annual
Correlation between supervisor focus on the Wraparound process and youth outcomes

59. Quality Management Trend analyses Fidelity consistency over time
What factors affect fidelity
Learning curve with new manager or facilitators

60. Next Steps Continue pilot
Conduct further analyses on the psychometric properties of the scale
Factor analyses
Items reduction

61. Director of Outcomes & Quality Assurance
Contact Information:
Eleanor Castillo, Ph.D.
Director of Outcomes & Quality Assurance