QUALITY IMPROVEMENT TRAINING: Module #4 How will we know that a change is an improvement? An introduction to QI measurement This work is supported by a grant from the CDC to the Academic Pediatric Association and in-kind contributions from Children’s Mercy Hospital in Kansas City, MO.

Objectives After viewing this segment, you will be able to: 1. Describe what makes up a complete set of measures. 2. Name the 2 reasons to define measures and collect data. 3. List at least 3 important features of data collection for QI. 4. Describe the elements of a run chart

Effective measurement requires a complete set of measures

3 Types of Measures Outcome measures Process measures Balancing measures

Outcome Measure The voice of the patient or other stakeholder, e.g., parent or payer Describes where there’s value in the process Sometimes we use a proxy, for example: HPV vaccination rate Average hemoglobin A1c level

Process Measure Are the steps that are required in order to reach the intended outcome being followed? Describes the workings of the system Example Date for next dose on DC paperwork Our site has a problem with HPV vaccine doses 2 & 3 Adolescent immunization rate (complete)

Balancing Measure Are other parts of the system being disrupted by our changes? Describes unintended negative consequences Example Date for next dose on DC paperwork Our site has a problem with HPV vaccine doses 2 & 3 Adolescent immunization rate (complete) Increased no show rate

Change in the steps of the operation Step XX was giving us difficulty Patient survival rate OR utilization time

Why should you define measures and collect data? Nolan TW. Execution of Strategic Improvement Initiatives to Produce System level results. IHI Innovations Series White Paper. Cambridge, Mass: Institute for Healthcare Improvement; 2007

Use data to investigate the problem and to drive improvement -- Establishing a learning system to increase the chance of producing the intended results

Using Data to Investigate Do we truly have a problem? What exactly are we trying to accomplish? What issues are at the root of our problem? What factors cause the problem more or less than others? Does the data confirm or disprove my theories? How frequently & to whom does the problem occur? What is the baseline level of variation?

Using Data to Drive Improvement To focus our change strategies

Using Data to Drive Improvement To focus our change strategies To see variation over time To determine if an intervention is successful To see if our interventions resulted in improvement

Some important features of data collection for QI Langley, G., Moen, R., Nolan, K., Nolan, T., Norman, C., & Provost, L. (2009) The Improvement Guide: A Practical Approach to Enhancing Organizational Performance, 2 nd Edition. San Francisco: Jossey-Bass.

Measurement Tips Collect data before and after a change Use clear operational definitions for your measures

Operational Definition Establishes a shared understanding of a measure Removes ambiguity Includes: The method of measurement Criteria for judgment Example: Outcome measure-HPV vaccination initiation Population: As of 2/1/2015, pts at least 11 years of age & seen for any visit during the past 2 years (per billing data) Measure: Proportion of the target population that has received at least 1 dose of any HPV vaccine (per EHR extract) (irrespective of allergies, etc.)

Reliability & Validity Reliability: Can you repeat the measurement process multiple times and get the same result? Validity: Does the process really measure what you are trying to measure?

Measurement Tips Collect data before and after a change Use clear operational definitions for your measures Use a reliable and valid data collection system Keep measures to 6 or less Make sure that your data collection is easy Use qualitative data to give your quantitative data depth Document your data collection plan (who, what, when, how)

Displaying data over time helps interpret it.

Constructing a run chart

Traditional Research Approach

Same annual data, more complete story

Same annual data, very different story

4 Elements in a Run Chart: median, goal, time series, & notes To find the median, list your pre-cycle data in numerical order and find the middle value. You need >12 pre-intervention values. The run chart: a simple analytical tool for learning from variation in healthcare processes. Rocco J Perla, Lloyd P Provost, Sandy K Murray

4 Elements in a Run Chart: median, goal, time series, & notes The run chart: a simple analytical tool for learning from variation in healthcare processes. Rocco J Perla, Lloyd P Provost, Sandy K Murray Start standin g orders Reminder sent New interns

Standing orders start Were the results as you predicted? (Your hypothesis) There was less of an increase than we expected What did we learn from this PDSA cycle? The nurses need more training to feel confident with this

Summary 1. A complete set of measures has outcome, process, and balancing measures; include each in your project 2. Define measures and collect data to learn more about your problem and to determine if your change resulted in improvement. 3. Make sure your data collection system is reliable, valid and uses clear operational definitions. 4. A run chart displays measurement changes over time, the goal, the median of the pre-cycle data, and notes on changes made to the process.

The End For more on QI measurement, see IHI. Science of Improvement: How to Improve ntHowtoImprove.aspx