1. Dr. Gary Walby Complex Systems Innovations
Partners Health Summit: Whole Person Integrated Care The WPIC Evaluation
Dr. Gary Walby
Complex Systems Innovations

2. WPIC Evaluation Targeted Components

3. From Investigation of Context came the Core Integrating Concept of WPIC
Wellness is a unifying concept within and between the HUBs and the 3 Tiers of WPIC.
Wellness is a critical Tier 2 concept that bridges Tier 1 (focus on the individual/family via the Adapted Collaborative Care Model (ACCM)), and Tier 3, where wellness is already proven as significantly and positively associated with SDOH.

4. Context Informed Evaluation
Implement
Planning & Design
C/I = Emerging Factors, organizational changes, unexpected events
C/P&D = Laws, policies, relationships, organizational and community assets
THEORY INFORMED
THEORY INFORMED
ALL = Improved Evaluation Clarity to Support Evaluand Effectiveness
P&D/I = Fidelity, timing, feedback, evidence-based changes

5. What is Being Evaluated? What is the Context?
WPIC – Three Tier Model - Overview
Tier 3
The larger community +
Time Bank to address SDOH
Tier 2
A community forum/learning collaborative that links to and supports Tier 1 partners
Tier 1
Health Providers moving to best practices with customized support
(HUBs, PCPs, etc.)

6. Drilling Down: Evaluation Design Decisions
Evaluation Questions
Evaluation Type
Analytic Plan
Data Collection Methods, Measures and Tools
Specific Outcomes and Outputs
Evaluation Models or Schools

7. Model Theoretical Content in Relation to Evaluation

8. WPIC Evaluation Model Components

9. Developmental & Collective Impact Evaluation
Formative & Implementation Evaluation
Outcome Evaluation
Impact Evaluation & Case Studies
Multiple Evaluation Models and processes were selected to mirror, assess value of, and assist in implementation of the complex WPIC intervention model. This includes outcomes and measurement selection.

10. Points Relevant to Context, Planning/Design and Implementation
Develop an evaluation plan that accomplishes evaluation objectives efficiently
Gain agreement on the evaluation plan
Identify performance needs for both the object of the evaluation (program, organization) and for the evaluation itself
Cannot assume access, flawless data collection, buy in by all parties, or that the context will not change
Finalize initial timelines and deliverables but build in and communicate flexibility

11. Evaluation Plan Generation
Defining Evaluation Component Research Questions
Defining Outcomes by Component
Selecting Activities to Track (Processes)
Defining Activities in Relation to WPIC Tier
Selecting Data Collection Tools
Planning Component Consistent Data Collection Methodologies
Finalizing an Analytic Plan
Reporting and Disseminating

12. Setting the Stage for Evaluation and WPIC Implementation Success
Forethought and Setting the Stage: Assessing and improving the context for evaluation through two methods that will improve the context for evaluation activities and link evaluation implementation to program outcomes.
Evaluation component activities checklists – micro and meso context and process
Identifying performance needs and testable solutions – all Tiers & context levels

13. Examples Evaluation Component Activities Checklists
Outcome Evaluation
Developmental Evaluation
Implementation Evaluation
Develop a shared measurement system across sites
Discuss training, expected outputs, and outcome links to peer navigators
Establish a data collection monitoring protocol to assess timeliness and accuracy of data entry across sites
Collaboratively define patient level outputs and outcomes (clinical, wellness, quality of life, satisfaction)
Assist and assess training of staff in the combined model (Quadruple Aim, AIMS Collaborative Care Model, Collective Impact)
Finalize standard processes for data collection and persons responsible
Create the framework to include partner specific (non-shared) outcomes to support sites as well as the framework to add additional, emergent outcomes
Poll collaborative partners for best ways to embed, implement, and sustain short and frequent communication efforts
Implement data collection and ‘check in’ processes to maintain evaluation focus and detect emerging issues within the Collective Impact Emergence Framework

14. Identifying Performance Needs and Factors Relevant to Program & Evaluation Implementation and Outcomes
Goal is to identify appropriate outputs, outcomes, and performance needs that accurately encompass the evaluation goals, questions, and context of the project
Performance Need = a state of existence or level of performance required for satisfactory functioning. In order for success there is a…
Need to do something
Need to be something
Need to be able to do something

15. Performance Needs…
Another way to think about performance need is as an actual or potential problem
Skill deficit in the program or the evaluation
Lack of knowledge or understanding
Lack of extrinsic (incentives) or intrinsic (interest) motivations
Lack of resources
Organizational culture counter to goals/outcomes
Use of a temporary logic model can be helpful
Identifying MET and UNMET needs (barriers)
Met needs: Maintain and sustain
Unmet needs: Plan activities to increase chance of success
Both: Coordinate vigilance to detect unintended consequences

16. Sample WPIC Evaluation Questions & Outcomes Collective Impact
Is the process of collective impact resulting in shared decision making?
Are the connections between organizations in the Health Forum, the HUBs, and Partners sufficiently dense and adaptable to sustainably address social determinants?
Are implementation strategies/activities of WPIC successful in promoting a collective effort in meeting implementation goals for Tiers 2 and 3?
CI Condition
Outcome
Indicator
Measurement
Backbone Structure
80% of participants endorse that the Backbone Organization (Partners) has sufficiently guided the vision and strategy of WPIC
Backbone (BB) staff effectively manage complex relationships
Online CI specific survey;
Interviews
Shared Measurement
80% of participants agree that shared measurement was collaboratively designed
Partners feel a collective accountability for results

17. Sample of Collective Impact Activities
Core Condition
Sample Activities
Backbone Function
The backbone group will meet regularly to discuss and reflect on progress and obstacles, planning interventions and support.
Training in Collective Impact will be provided by the evaluation team to the core members above and others.
Common Agenda
The evaluation team supports dialogue for a common agenda using participatory evaluation activities.
The evaluation team attends or reviews recordings of at least 80% of meetings, extracting information that supports and obstructs a common agenda and reports to all participants transparently.
Shared Measurement System
The evaluation team will complete quarterly reports related to common measurement indicators and outcomes.
The evaluation team will assist the Health Forums and Time Banks in selecting data elements that support rigorous analysis.
Continuous Communication
The evaluation team will extract information on indicators and report regularly to identified recipients.
Data analysis will be used to determine the degree that partners coordinate efforts, frequency of communication, clarity of communication, and how feedback is integrated.
Mutually Reinforcing Activities
Organizational evaluation techniques and tools (e.g. Collaborative Development Surveys) will be used to assess the development process and provide feedback.

18. Sample WPIC Evaluation Questions Developmental Evaluation
What are the initial and evolving conditions in each HUB, Health Forum, Time Bank, the cross HUB implementation team, and others as they emerge, that affect full and accurate implementation of WPIC?
What are the benefits, issues, unintended results, and opinions of the Care Manager and PEER Mentor/Navigator in relation to implementation and perceived impact of the WPIC model?
In what ways are HUBs altered due to integrating WPIC (HUBs) and how are community partnerships developed that are based on WPIC (Health Forums)?

19. Developmental Evaluation Interpretive Lens and Activity Generator
Emergence
Adaptation
Uncertainty
Interpretive Frameworks
Emergence: Patterns emerge from the self-organization of interacting agents. Like water finding its way downhill, most patterns emerge as a function of the path of least resistance.
Adaptation: Organizational elements in and outside of HUB’s will adapt to each other and continue to adapt as they become privy to information that is then interpreted within each entities organizational knowledge.
Uncertainty: Organizations and partnerships will change and grow in response to the expectations of WPIC. However, due to emergence and adaptation, expected processes and predetermined outcomes must be viewed as not completely predictable or controllable.
Interpretive Frameworks: Processes and even facts are subject to varying interpretations based on prior learning, the desire to fit ‘facts’ to current belief systems, and perceived or real limitations of data.

20. Sample Developmental Evaluation Methods/Activities
Rapid assessment via consistently used short online surveys and extraction of key information from all attended or recorded meetings
Ongoing environmental scanning and outcomes monitoring
Systems change mapping (looking at changes in resource dedication, organizational structure, staffing patterns, and similar)
Interviewing teams in each HUB via focus groups to assess their understanding of WPIC, evaluation, capacity to interpret objectively data and findings, and openness to change based on evidence

21. Outcome Evaluation
Four evaluation questions are linked to multi-level outcomes:
What are the key facilitators, barriers, and solutions that influenced HUBs adoption of WPIC?
What are the WPIC practices, specifically of the WPIC implementation of the AIMS model, that influence/facilitate fidelity to the model, provider satisfaction, and clinical outcomes?
What are the key factors for persons served that influence the adoption of self-management solutions into their lifestyles?
Are resource gaps being addressed and, if so, is there sufficient increase in resources to implement and maintain WPIC over time?

22. Sample WPIC Evaluation Outcomes Outcome Evaluation
25 outcomes segmented into HUB, Consumer, Provider, and Community Level Outcomes
Level
Outcome
Data Source/Method
HUB
Rates of ER visits will decline by 15% compared to a similar time frame from a previous year
Partners service data
Consumer
Person served completion of services will improve by 20% compared to a similar time frame from a previous year
HUB service data
80% of persons served will improve in clinical outcomes related to behavioral health indicators
Duke 8 Survey
Provider
At least 80% of practitioners will indicate satisfaction with the WPIC service delivery model
Evaluation survey; interviews
Community
Time Bank data will endorse improvement in community participation from baseline for 70% of Time Bank WPIC members
Project specific online survey; Time Bank data

23. Sample of Activities and Methods for Outcome Evaluation
A retrospective interview process to gather historical recollections and experiences of staff and practitioners in the HUBs before and after WPIC implementation.
The HUBs current data collected to develop a data base of shared items that will help to describe persons served, staff, clinical outcomes, and also unique fields for assessing HUB specific outcomes.
The use of an online survey and follow-up interviews to describe care coordination practices and how practices are adapting to and engaging in WPIC. This will include measuring satisfaction with WPIC implementation, practice pathways, and the perceived link to outcomes. Surveys and interviews will be repeated.

24. Case Studies
Case studies start by defining the case. For this project, a “case” is an individual family that one or more members (preferably more) are being served via the WPIC system.
A stratified system of deliberate sampling will be completed (e.g. age, gender, presenting problem), with final variables TBD.

25. Sample Case Study Questions
What are the experiences of each case in the WPIC model?
What are the perceived benefits or changes for cases in response to exposure to WPIC?
What were the expected, planned outcomes due to being involved with WPIC? How well were they met?
What were unexpected outcomes or effects from involvement with WPIC?

26. Implementation Evaluation
The design is WHAT you are going to do in the evaluation
Implementation is HOW you are going to do it
Note: Implementation evaluation focuses on how a program is or is not achieving its goals through looking at contextual and fidelity factors
Useful for multi-site projects to determine differential effects

27. Implementing the Evaluation
Implementation monitoring of an evaluation is similar to strategic planning
Defining the evaluation and the evaluation vision
GAP analysis to detect evaluation barriers
Crafting the evaluation implementation plan
Implementing the evaluation plan
Evaluating evaluation performance, reviewing, adjusting, and correcting

28. Tracking Program Implementation AND Evaluation Implementation
Selection of methods
Qualitative measures to track process
Review methods if repeating in cycles to ensure fidelity and reduce drift
MODIFY methods after testing them for fidelity and for changes in needed information
Test
Examine
Modify
Re-Test

29. Next Steps…
Implementation and Evaluation Teams Combined Efforts to map…
HUB Resources
Collaborations and relationships with emphasis on similar initiatives to avoid duplication and to establish mutual support
Data collected for HUB and HUB members/partners
Current measures being used (frequency, data storage, use)
Data systems (EHR, etc.)

30. Next Steps…
Detailed work flow of each HUB via document review and shadowing
HUB understanding of current draft outcomes and collaborating to finalize
Define HUB specific outcomes
Discussion of data driven decision-making practices and needs
How will failures and successes be determined and measured?
What is needed to attribute outcomes to services provided?

31. Next Steps…
Finalize the Adapted Collaborative Care Model (ACCM) process points for tracking
Complete work flow observations to determine process points that fit ACCM
How are problems or issues identified and addressed in work flow?
How are problems or issues identified and addressed that related to patient goals or outcomes?
Finalize white paper and evaluation model based on recent meetings and observations

32. Q & A Questions, comments, sarcastic remarks?
Question that comes to mind later?
Call: