1. UC San Diego, Department of Family Medicine and Public Health
BHICCI Health Care Organizations’ Learning: Review of Initial BHICCI Learning Evaluation Report
Todd Gilmer PhD
UC San Diego, Department of Family Medicine and Public Health

2. Evaluation Team Todd Gilmer, PhD – Evaluation Project Director
Ben Henwood, PhD – Senior Associate
Kimberly Center, MA – Evaluation Program Manager
Amy Panczakiewicz, MA – Program Evaluation Specialist
Lucyna Klinicka, BA – Program Evaluation Specialist

3. BHICCI Evaluation A primary goal of the BHICCI evaluation is learning.
The term “evaluation” may conjure many different words in your mind. The evaluation design and methodology that we will use is tailored to meet the needs of IEHP and the BHICCI in consideration of the unique nature of the initiative. The evaluation is not meant to make comparisons between organizations but to document each organizations progress to becoming health homes to provide recommendations to IEHP for further spreading transformation changes to other organizations.
A primary goal of the BHICCI evaluation is learning.

4. Evaluation Methods Formative Components Summative Components
Allow for feedback to be incorporated during program implementation
BHICCI Roadmap Checklist
BHICCI Site Self-Assessment
Qualitative site visits
Summative Components
End of the program cycle
Overall description of program effectiveness
PROMIS Global Health
Registry and measurement-based practices
Utilization and cost
Cost-effectiveness
Evaluation methodology includes both formative and summative components. Formative evaluations allow for feedback to be incorporated during the implementation of a program. Summative evaluations occur at the end of a program cycle and provide an overall description of program effectiveness.
NOTE: We need to be careful to use the term measurement-based practices instead of T2T.
The next cycle of reports will include summative data components.

5. BHICCI Roadmap Progress
Foundation for Improvement
Developing Complex Care Systems
Sustaining Changes
Practice changes are conceptualized as steps within three phases for practice transformation: Foundation for Improvement, Developing Complex Care Systems/Whole Health Homes and Sustaining Changes. While the goals and practice changes described in the Roadmap are not strictly sequential, leadership who supports an organizational culture that is committed to quality and providing whole person health care, which includes attention to employee engagement and wellness and using tools for quality improvement, provides the foundation that is necessary to sustain enhancements to the service system.

6. Roadmap Progress: Phase 1
Engaging Leadership
Engaging Employees
Adopting QI Methodology
Use Data and Measurement to Guide Practice Changes
Team Reports submitted in April 2016, responses to the baseline BHICCI Site Self-Assessment and qualitative interviews with the practice coaches conducted in June 2016 were used so evaluation team members could coded progress for each BHICCI practice change goal listed on the Roadmap as “not started or not included in report,” “implementation phases,” or “practice change made.” Coding within the implementation phase includes any key activities that were described as being planned or tested.
Engaging Leadership: The majority of organizations (83%) have started implementing or have made at least one practice change. There is the most variability in organizations’ progress promoting an organizational culture that creates incentives for workforce engagement in quality, learning and development, and promotes pride and joy in work, and developing collaborative relationships with other systems. Only 25% of organizations are implementing this change at this point.
Engaging Employees: Half of the organizations have started implementing all three employee engagement practice change goals. As described in the team reports, hiring and recruitment has been a challenge for many organizations. Despite these challenges, 83% have taken steps towards the development of a specific department or leader that measures employee experience and engagement as a core focus.
Adopting QI Methodology: All of the organizations participating in the BHICCI started implementing at least one of the five quality improvement methodology practice change goals. Two-thirds of the organizations have started to use health information technology, and have made progress establishing and monitoring client/patient, population, and system-level metrics.
Use of data and measurement: 25% of organizations have started to implement all four data and measurement practice change goals. The majority of organizations (83%) have taken steps to train and coach staff at all levels to use both system and clinical-level data to track and improve results over time.

7. Roadmap Progress: Phase 2
Identify Target Populations
Build Multidisciplinary Complex Care Team
Offer Integrated, Complex Care Management
Engage Individuals and Families in Whole Health
Identify target population: 92% of organizations have started to identify their target population, select individuals for testing complex care management and established practices to ensure that ROI are signed
Build multidisciplinary complex care team: Every organization participating in the BHICCI has initiated implementation of at least two of the six practice change goals necessary to build the multidisciplinary, complex care team. 41% of organizations have initiated daily team communications, and half have increased knowledge of substance use and addictive disorders and intervention capacity within their multidisciplinary team.
Offer integrated, complex care management: Half of the organizations have started the process of measuring and tracking screening, treatment, and wellness outcomes in a registry. 58% of organizations have started coordinating the care of clients/patients with complex conditions.
Engage individuals and families in whole health: Two-thirds of organizations have started the implementation phase of linking patients and/or families to needed social services, housing, and community resources.

8. Roadmap Progress: Phase 3
Sustaining Practice Changes
Most organizations participating in BHICCI have not started to implement any of the practice changes related to sustainability. However, it is early in the implementation process, and not really anticiatied that many organizations would have made progress sustaining changes at this point.

9. Roadmap Progress Summary
Overall, each organization has started implementing at least some practice change goals
At some organizations, some practice change goals were implemented prior to participation
It is anticipated that organizations will move from “not started” and “implementation phase” to “practice change made” on these goals over time
Overall, all of the organizations participating in the BHICCI have started the implementation phase of at least some of the practice change goals.
Many of the organizations had implemented some of the elements of the Roadmap prior to participation in the BHICCI. BHICCI is supporting the pathway that organizations were already on.
As the initiative continues, it is anticipated that organizations will move from the “not started” and “implementation phase” on these goals to “practice change made.”

10. BHICCI Site Self Assessment (SSA)
Adapted from the Maine Health Access Foundation (MeHAF) SSA
BHICCI SSA has 21 characteristics
11 related to services (more directly impacting clients/patients)
10 related to organizations (more directly impacting health care team members)
Health care teams rate each characteristic using a scale.
11 characteristics combined to create person-centered, integrated care domain
10 characteristics combined to create practice/organizational integration domain

11. Baseline Site Self Assessment Results
Person-Centered Integrated Care Domain
Most common average score was 5.09.
Interpretation of average rating
Practice/Organizational Integration Domain
Most common average score was 4.10.
A rating score of “5” suggests that behavioral health and physical health providers have started to work in consultation and are conducting screenings and measurement regularly, but clients/patients and family are only sometimes included in conversations about screening results and treatment planning. Average scores for the person-centered domain ranged from 3.45 to 7.91.
A rating score of “4” suggests that engagement by the executive leadership and providers’ is limited, coordination of care is not always assured and data systems and registries are not consistently integrated. Average scores for the practice/organization domain ranged from 2.70 to 8.30.

12. Baseline Site Self Assessment Results Highlights
There was high variability in responses for several items.
Organization’s ratings on these items were distributed between almost every level from “1” to “10”
Greatest variability for following items:
- Individual/family involvement in shared care planning
- assessment of clients/patients’ strengths, preferences and social determinants of health
- services are provided by multidisciplinary team

13. Baseline Site Self Assessment Results Highlights
Half (50%) of organizations are engaging in some aspects of:
Average domain scores were more spread out between levels.
Results suggest organizations may be implementing different components of BHICCI, dependent on their current capacity for change.
Complex care management
Integrated care
Population health management
Team-based care
Measurement of health care team experience
Integration of leadership

14. Baseline Site Self Assessment Summary
Different Stages of Change
Practice Change Goal Status
Results and Future Reporting
as evident by high ratings at baseline
- Results show that organizations are currently at very different stages of BHICCI planning and implementation.
- Some organizations have integrated several BHICCI components into practice, but most organizations are currently working towards integration and implementation in each BHICCI domain.
- These results are promising, and future reporting will focus on changes in BHICCI domains within the first year of the initiative.

15. Fact Sheets
It was important to the evaluation team to shared the results from the baseline SSA back with organizations to promote a learning culture. Baseline BHICCI Site Self Assessment Results are described in the Baseline Integration report, and summarized for each organization in their Fact Sheet. The 1st Fact Sheet includes a brief description of the organization, a summary of the Physical Health and Behavioral Health diagnoses included in the ACG report (only if IEHP has authorization to access client/patient information) and information from the Site Self Assessment. The Fact Sheets will be distributed to each organization tomorrow morning. Please feel free to contact evaluation team with any questions.

16. Documenting the BHICCI Learning Throughout the Process
Ben Henwood, PhD
University of Southern California (USC)
School of Social Work

17. Methods to Document Learning
Documentation of learning through qualitative interviews during site visits
Key stakeholders
Health care teams
Documentation of learning collaborative and team meetings
Capture challenges
Identify best practice solutions
A major component of the evaluation will be the documentation of learning from the implementation of the BHICCI.
This will be achieved through qualitative interviews with key stakeholders’ and health care teams during site visits.
Learning collaborative and team meetings will also be documented to capture challenges and best practice solutions encountered during the course of implementation at the organizational level. 

18. Learning from Qualitative Interviews
Interviews with key stakeholders will help identify emerging themes and challenges.
Site visits and interviews with health care team will promote learning about transformation process and best practices.

19. Early Lessons Learned
Importance of Engaging Executive Leadership in BHICCI
Implementation of Cross-System Leadership skill building workshops
Intentional practice of sharing and communication

20. Moving Forward July 2016 January 2017 November 2016 August 2016
PROMIS Global Health pilot testing
August 2016
Site visits to 3 or 4 organizations
November 2016
Second administration of BHICCI SSA
January 2017
Additional data in next set of reports
Pilot testing of the PROMIS Global Health will begin in July 2016
First round of site visits at 3-4 organizations will occur in August 2016
Second administration of BHICCI SSA will occur in November November assessment period will be baseline for RUHS sites.
Next set of reports (January 2017) will include data from the PROMIS Global Health and Measurement- based practices, describe SSA progress and learning from site visits.