1. Behavioral Health Concepts, Inc.
Performance Improvement Projects 101 Drug MediCal – Bringing Ideas to Successful PIP Concept
Amy McCurry Schwartz, Esq., MHSA
California EQRO Consultant

2. PIPs – 101Background The purpose of PIPs
To assess and improve processes, and thereby outcomes, of the substance use disorder continuum of care.
How is a PIP different from a PM (Performance Measure)?
A PIP focuses on program improvement, a PM reflects current performance of an measure.

3. PIPs – 101 Background Clinical vs. Non-Clinical PIPs
The requirement for DMC programs to do both clinical and non-clinical PIPs comes from the federal regulatory definition of a PIP.
42 CFR (d) defines PIPs as having a “focus on clinical and non-clinical areas.”

4. PIPs – 101 Background Clinical PIPs Might Target
Prevention and care of acute and chronic SUD conditions including relapse
High-volumes of services – such as 180 days or more of residential in a year
High-risk procedures – detox w/mixed drugs
Reducing clients leaving residential detox and not continuing treatment

5. PIPs – 101 Introduction Non-Clinical PIPs Might Target
Coordination of Care across SUD continuum and other treatment or support resources
Appeals, Grievances Processes
Timely Access to Assessment & Treatment
Client Services and experience of care

6. PIPs – 101 Introduction EQR PROTOCOL 3:
OMB Approval No.
EQR PROTOCOL 3:
VALIDATING PERFORMANCE IMPROVEMENT PROJECTS (PIPs)
A Mandatory Protocol for External Quality Reviews (EQR) by the Federal Government for all managed care plans

7. CMS Protocol: Validating Performance Improvement Projects – Key Job of EQRO
10 Protocol Validation Activities each with important steps and processes:
Identify a potential the study topic(s)
Define the study question(s)
Use a Representative and Generalizable study population
Select the study variable(s)
If sampling used, valid approach (not recommended without sophisticated stat capacity) just use one site or program

8. CMS Protocol: Validating Performance Improvement Projects
10 Protocol Steps Continued:
Reliably collect data
Analyze data and interpret study results
Implement intervention/improvement strategies
Plan for “real” improvement
Achieve sustained improvement

9. Step 1: Identify the potential study topic(s)
Look for topics that:
Impact a significant portion of clients
Get client input, data on topic/approach whenever possible
Have a potentially significant impact on client SUD recovery, functional status or satisfaction
Consider high-volume or high-risk conditions of the population served

10. Step 1: Identify the study topic(s)
Data sources that may help:
MediCal eligibility database
DMC encounter/claims data
DMC performance on standardized measures on CalOMS by program
External data – Medicaid FFS, other health plans, local or national public health reports, research review of improvement efforts in this area (be sure to show this data has a direct correlation to the DMC-ODS’s population)
Special studies to examine your study question itself

11. CALEQRO PIP Submission Tool

12. Step 2: Define the Study Question(s)
Make the Question:
Obvious
Simple
Answerable with the data and design you are considering

13. Step 2: Define the Study Question(s)
State the Question in a way that:
Supports the ability to determine whether the intervention has a measurable improvement impact for a clearly defined population

14. Step 2: Define the Study Question(s)
Start small and simple (incremental approaches are easier to manage well) and then if successful you can bring to scale

15. Step 2: Define the Study Question(s)
#1 What’s the problem?
Try to define the problem as systems issues, clinical or process oriented, that relate to individual clients' treatment, outcomes, or satisfaction.
IF the problem is No-Show rates:
The MHP may see this problem as:
Fiscal
Efficiency
Inconvenience
Provider communication/partnership with client

16. Step 2: Define the Study Question(s)
Study questions are measurable and manageable issues that reflect the goal of the PIP.
“To assess and improve processes, and thereby outcomes, of the substance use disorder continuum of care”—it is important to the agency, the client, and to the provision of quality Substance Use health care services.

17. Step 2: Define the Study Question(s):
Don’t write a question that asks:
“Will providing pre-appointment reminder calls for all assessment appointments reduce the DMC’s cost of ASAM Assessment Services?” OR
“Will providing training to all front line staff get them to enter appointments in a more timely manner into the EHR?”

18. Step 2: Define the Study Question(s):
INSTEAD:
Think or ASK? What is keeping clients from their appointments?
What are the barriers to coming to the assessment appointments?
Transportation
Time off work or school during the clinic hours
Childcare
Remembering because it was too far from request
Frame the PIP around these barriers. Provide
interventions that mitigate these barriers.

19. Step 2: Define the Study Question(s)
Vague Study Question Example:
An “implied” study question
“Is a mail intervention reminder adequate to improve SUD medication adherence over time?” or
“Can we evaluate the use of appointment reminders?”

20. Step 2: Define the Study Question(s):
Clearer Study Question Example:
Will providing educational materials to doctors, counselors, and clients increase the number and percentage of clients who participate in a MAT (medication assisted treatment) clinic, and decrease the number and percentage of clients who use counseling services without MAT supports?

21. Step 2: Define the Study Question(s):
Clearer Study Question Example:
“Will reducing the number or incidents of multiple intake/assessment appointments prior to the first treatment session increase the number of clients successfully engaging in treatment?”
Even better if you add a factor by which to increase (i.e. 10%, 5%...)

22. Step 2: Define the Study Question(s):
This step is critical because it sets the framework for data collection and analysis of results.
So take the time to develop a
strong study question!
Consult with your assigned EQRO staff to help refine the question and bring it to concept!

23. CALEQRO PIP Submission Tool

25. Step 3: Use a Representative and Generalizable Study Population
Each PIP must potentially represent the entire SUD Medi-Cal population to which the study indicators apply. For Example, a youth question might be studied only at one program but have potential to improve all such programs.

26. Step 3: Use a Representative and Generalizable Study Population
The DMC must decide whether to review data for the entire population or a single program or unit or set of that population.
It is good to start small and incrementally bring to scale if the intervention works

27. Step 3: Use a Representative and Generalizable Study Population
Reasons to use a unit, set or single program:
Limited resources are available to analyze the data
The intervention(s) and data collection can be very resource intensive and you do not know yet if it is effective

28. Step 3: Use a Representative and Generalizable Study Population
Population Example: “All DMC clients ages 0-18 years with a primary or secondary diagnosis of XXXXXXX as identified through claims data”

29. CALEQRO PIP Submission Tool
Same questions

30. Step 4: Select the Study Indicator(s)
A study indicator is:
A quantitative or qualitative characteristic reflecting a discrete event or continuous status to be measured
Used to track performance and improvement over time

31. Step 4: Select the Study Indicator(s)
Study Indicator “tips”:
The study indicator should be objective and clearly defined
The indicator must be “measurable” – do you have the necessary data (or can you get it)?
client input recommended on these is good and many counties have client QI advisory group.

32. Step 4: Select the Study Indicator(s)
Study Indicator “tips”:
Use an existing indicator generally accepted in the SUD clinical guidelines such as ASAM or NIATX or SAMHSA or HEDIS or AHRQ, etc.
Use an existing, relevant SUD
best practice or guideline to
develop the study indicator (s) to tracking improvements or change.

33. Step 4: Select the Study Indicator(s)
Study Indicators Example:
Clients enrolled in SUD outpatient clinic with 4 visits within 30 days of initial assessment.
Access & engagement indicator
Percentage of clients who complete residential detox and transition to next level of care within 14 days of discharge.
Transitions in care indicator

34. Step 4: Select the Study Indicator(s)
Study Indicators Example:
Percentage of Methadone clients with 9 months or more of regular treatment visits (3 per week average)
Percentage of current Methadone clients able to work at least 20 hours per week.
Improved functioning indicator

36. Step 5: Use Sound Sampling/Target Group Techniques
Proper sampling techniques are necessary to provide valid and reliable (therefore generalizable) information on the quality of care provided.
CAUTION – true statistically valid sampling is difficult. At this point, consider a representative clinic or program that is small and build out unless you have statistical and data capacity.

37. Step 5: Use Sound Sampling Techniques
Probability sampling is based on chance, it removes bias from the selection process.
Examples: simple random sampling
systematic random sampling
Non-probability sampling is based on choice, not chance.
Example: convenience sampling

38. Step 5: Use Sound Sampling Techniques
Probability Sample:
A study of how many clients received a client satisfaction survey after completion of a level of care during a specified year by randomly selecting 100 of the 1,000 of the DMC population.
Non- Probability Sample:
A study of the performance of an outpatient clinic by sampling all the clients that were seen in that SUD office on a specific day.

39. CALEQRO PIP Submission Tool
Discuss why

40. Step 6: Reliably Collect Data
Data must be valid and reliable.
Valid – indication of accuracy
The data measures what
is intended to be measured.
No other data is mixed in.
An example would be good here

41. Step 6: Reliably Collect Data
Example:
Attendance may be a valid measure of engagement but not necessarily a valid measure of satisfaction with the program.
Client satisfaction survey was reliable but discovered that because it wasn’t anonymous (we had them sign them) and were completed the week prior to discharge, response bias could reduce their reliability.

42. Step 6: Reliably Collect Data
Reliable – indication of repeatability of collection method and source.
Produces consistent results

43. Step 6: Reliably Collect Data

44. Step 6: Reliably Collect Data
Develop a data collection plan:
Clearly identify data to be
collected
Identify data sources and how/when the baseline and repeat indicator data will be collected
Some examples that would be SUD

45. Step 6: Reliably Collect Data
Specify who will collect the data and that they are qualified to collect the data
Data collection personnel have the conceptual and organizational skills to abstract the data.
Example helpful

46. Step 6: Reliably Collect Data
Experienced clinical staff, such as SUD certified counselors, should be used to extract the appropriate data from medical records (if this is data source) to support a judgment about whether SUD clinical criteria are met.
In contrast, trained assistants or medical records clerks may collect data if the abstraction involves verifying the presence of a diagnostic test or assessment report, etc. (i.e. is is present or not??)

47. Step 6: Reliably Collect Data
Identify the data collection instruments to be used

48. CALEQRO PIP Submission Tool

50. Step 7: Analyze Data and Interpret Study Results
Use the Data Analysis Plan Up Front
How did the SUD identified group or program perform on the selected indicators?
Many PIPs have baseline data with indicators and periodic tracking of change in indicators over a planned time period

51. Step 7: Analyze Data and Interpret Study Results
Valid interpretations occur when:
Analysis is conducted according to a well developed data analysis plan
Results present numerical data that is accurate, clear and easily understood
Analysis addresses the comparability of initial (or baseline) and repeat measurements, including factors that threaten validity
A conclusion is drawn as to the success of the PIP and planned follow-up activities are described or refinement of the design and the PIP can be extended or continued.

52. CALEQRO PIP Submission Tool
Goal?

53. Step 8: Implement Intervention and Improvement Strategies
Interventions should:
Relate to causes/barriers identified through data analysis – use client input
Issue = No Show Rates
Barrier identified = Transportation
Intervention = Bus and Taxi vouchers, Gas reimbursement

54. Step 8: Implement Intervention and Improvement Strategies
Issue = No Show Rates
Barrier identified = Time off during clinic hours
Offer After-hours SUD clinic
Offer home visit or community
Based counseling option
Barrier identified = Childcare during appointment
Provide childcare (vouchers or on-site)

55. Step 8: Implement Intervention and Improvement Strategies
Interventions should:
Be culturally and linguistically
appropriate
Be “implementable” system-wide
Not be “one time” efforts

56. Step 8: Implement Intervention and Improvement Strategies
An improvement strategy is an intervention designed to change behavior at an institutional, practitioner, or beneficiary level.
Real, sustained improvement results from a continuous cycle of measuring performance and implementing system-wide improvements.
(Plan, Do, Check, Act)
Need examples, this is all too abstract

57. CALEQRO PIP Submission Tool
example

58. Step 9: Plan for “Real” Improvement
When a change in performance occurs, determine whether the change is real, attributable to an event unrelated to the intervention (s), or random chance
Results of the intervention must be “statistically significant” . Consultation available to help measure this.
example

59. CALEQRO PIP Submission Tool

60. Step 10: Achieve Sustained Improvement
If real change has occurred, the project should be able to achieve sustained improvement.
Sustained improvement is demonstrated through repeated measurements over time.
Improvements can then be expanded to other clinics or other programs with data monitoring to insure they are working.
example

61. CMS Protocols: Conducting Performance Improvement Projects and Validating Performance Improvement Projects
Protocols Available Online at: Topics/Quality-of-Care/Quality-of-Care-External-Quality- Review.html Click on “External Quality Review Protocols (2012 Update)”