Measurement and Data Display QA Residency 2 Melanie Rathgeber, Merge Consulting

Learning Objectives 1. Identify and explain uses for five different tools to learn from data in QI 2. Create and analyze a run chart 3. Apply best practices for display of data using run charts

Reference:

Tools to Learn from Variation in Data FREQUENCY PLOT PARETO CHART SCATTER PLOT RUN CHART SHEWHART CHART Health Care Data Guide, p. 65

Run Charts and Shewhart/Control Charts 1. Run Chart: a. Makes Performance on Key Measures Visible b. Determine if Change is an Improvement – using Key Measures c. Determine if Improvement is Sustained – using Key Measures 2. Shewhart/Control Charts a. Look for evidence of variation and improvement b. Understand underlying causes of variation We will come back to these ……

Tools to Learn from Variation in Data FREQUENCY PLOT PARETO CHART SCATTER PLOT RUN CHART SHEWHART CHART Health Care Data Guide, p. 65

Other tools to learn from data : Frequency plots, Pareto charts, and Scatterplots Frequency Plot - shows distribution of data - useful to understand variability (how your results are spread out and distributed) for variables like time, patient volume, demand, or number of problems - the variable is a continuous measure

Example: Each day for about 10 months (255 days), a team measures how long it takes to do medication administration on their unit on the night shift (they rounded the number of minutes to the nearest five minutes) Variable = number of minutes taken to do medication administration

Results What does the Frequency Plot show you? Results Day 125 minutes Day 25 minutes Day 340 minutes Day 430 minutes Day 515 minutes Etc ….

Frequency Plot in Excel 1.Set up data as below – might need to use a count function in Excel to get raw data into this format 2. Insert column chart 3. Go to Select Data Source Series is “count of days” (i.e. the frequency) Horizontal axis is your variable – “number of minutes” Count of days Minutes to do med admin (rounded to nearest 5)

5 minutes practice – create a frequency plot in Excel What data do you work with that could be displayed as a frequency plot?

Pareto Chart - shows frequency of categories: -Useful when you are asking “why is this happening?” - Often answers “where is the greatest opportunity for improvement?” -Variable is categorical (not continuous like in frequency plots) -If you want to answer – why does med administration sometimes take l longer than 25 minutes or longer, you could do a chart review to find out reasons and then use a pareto chart to display results. -Helps you focus your QI efforts

1.You do some data collection and list each reason, with the number of times it was the primary reason, in descending order. (Reasons need to add up to 100%) 2. Hide Column B and Insert Column Chart Pareto Chart In Excel Primary reason for med admin delaysPercent Nurses busy with an incoming admission5539% Medication not stocked on unit3928% Med reconciliation not complete for all patients2115% Orders needed to be verified by phone1511% Special evening activity on the unit75% Need to locate code for medication cabinet43% %

Optional – add a Cumulative Percentage line which tells you how many categories are responsible for 80% of the problem. (need to do manually in Excel).

10 minutes practice. As a table, think of a hypothetical example where you could use a Pareto chart in your work Formulate possible reasons and enter your hypothetical data in excel. Remember the reasons need to add up to 100%. Construct a Pareto chart. (If you would like some extra tutoring on how to construct the chart, office hours are available).

Scatterplot Shows relationship between two continuous variables. Will show a cause and effect relationship if one exists. Useful for seeing potential areas for improvement. “If we can decrease X, Y should also go down” “If we can decrease time patients spend in the waiting room, the number of patients leaving without being seen should go down” “If we can decrease time patients spend in the waiting room, the ratings on patient satisfaction should go up.”

Week Average Time in Waiting Room Average Pt Satisfaction Rating Is there a relationship between average wait time in waiting room and patient satisfaction? In Excel, “Insert Scatter”

What about the relationship between number of weeks to get an appointment, and satisfaction? Average Pt Satisfaction Rating Number of weeks to get appointment

Run ChartShows performance of in key measure over time. Look for improvement. Shewhart/Control ChartShows performance of key measure over time. Look for improvement. Understand variation. Frequency PlotShows distribution of a continuous variable (e.g. time) Pareto ChartShows frequency of categories. Helpful to answer “why is this happening – what are the primary reasons?” ScatterplotShows relationship between two continuous variables. If we decrease one variable, will it have an effect on the other?

Practice Look for examples on your peers’ storyboards, where a frequency plot, pareto chart, or scatterplot may be used

Run Charts and Shewhart/Control Charts 1. Run Chart: a. Makes Performance on Key Measures Visible b. Determine if Change is an Improvement – using Key Measures c. Determine if Improvement is Sustained – using Key Measures 2. Shewhart/Control Charts a. Look for evidence of improvement b. Understand underlying causes of variation

Review – Key Measures –Collected over the life time of your project –Provides feedback that changes are having the desired impact –Outcome, process and balancing –Displayed over time (e.g. Run Chart) –Small samples collected frequently –Guideline: between 3-5 measures in total –Balancing measures may be collected more sporadically

Measures from Your Storyboards

Displaying Key Measures over Time - Run Chart -Data displayed in time order -Time is along X axis -Result along Y axis -Centre line = median -One “dot” = one sample of data -Sample size = each “dot” should have the same n

Showing improvement: No improvement. Random fluctuation. Improvement. Trend going up.

How Do We Analyze a Run Chart? Visual analysis first If pattern is not clear, then apply probability based rules The Data Guide, p 3-10

Making the Run Chart in Excel 1.Data will be in two columns. Column A will have dates or times, and Column B will have the result of your data collection. 2. Highlight this data

Making the Run Chart in Excel 3. With your mouse, choose Insert Line Line With Markers

Making the Run Chart in Excel

4. Inserting the Median Line (needed to analyze the run chart): Calculate the Median Value from your Data

Making the Run Chart in Excel 5. Enter the Median in Column C (we want a flat line for the median, i.e. all the same value)

Making the Run Chart in Excel 6. Add this median line to your data. Right click on your run chart. A menu will come up – choose “Select Data” 7. Select Add Series

Making the Run Chart in Excel 7. When you click on Add Series, it will ask you about the data you want to add. It will ask for a Series Name and Series Values. Ignore Series Name Click the button beside Series Values and then highlight the median values in column C. Press Enter and then OK.

8. The Run Chart is done and can be formatted: titles, gridlines, etc. Note – to make the median a flat line with no markers, right click on the line and change the Marker Options. Extra help with formatting - available during office hours.

Making the Run Chart in Excel - Practice

-There are simple rules, based on probability, that are used to determine evidence of improvement in our projects -Interpretation: the rules tell us if there is a non-random pattern in our data. -If we have implemented a change, and we see a non- random pattern (going in the right direction), it is evidence of improvement Analyzing Run Charts

Non-Random Signals on Run Charts A Shift: 6 or more An astronomical data point Too many or too few runs A Trend 5 or more The Data Guide, p 78 Evidence of a non-random signal if one or more of the circumstances depicted by these four rules are on the run chart. The first three rules are violations of random patterns and are based on a probability of less than 5% chance of occurring just by chance with no change.

Original Source: Swed, Frieda S. and Eisenhart, C. (1943) “Tables for Testing Randomness of Grouping in a Sequence of Alternatives.” Annals of Mathematical Statistics. Vol. XIV, pp , Tables II and III. The Data Guide, p 80

Trend?

Trend? note: if we have 2 of the same values in a row – only one contributes to the trend

Shift?

Shift? note: values on median don’t make or break a shift

Interpretation? -There is a signal of a non-random pattern -There is less than 5 % chance that we would see this pattern if something wasn’t going on, i.e. if there wasn’t a real change -There is a signal of a non-random pattern -There is less than 5 % chance that we would see this pattern if something wasn’t going on, i.e. if there wasn’t a real change

Plain Language Interpretation? There is evidence of improvement – the chance we would see a “shift” like this in data if there wasn’t a real change in what we were doing is less than 5%

Two few or too many runs? - 1. bring out the table 2. how many points do we have (not on median?) 3. how many runs do we have (cross median +1) 4. what is the upper and lower limit?

Two few or too many runs? - 1. bring out the table 2. how many points do we have (not on median?) how many runs do we have (cross median +1) 3 4. what is the upper and lower limit? 6-16

Two few runs? Plain language interpretation There is evidence of improvement – our data only crosses the median line twice – three runs. If it was just random variation, we would expect to see more up and down.

What if we had too many runs? Plain language interpretation There is evidence of a non-random pattern. There is a pattern to the way the data rises and falls above and below the median. Something systematically different. Should investigate and maybe plot on separate run charts.

Astronomical Data Point?

QA examples

Data Display Principles

* hypothetical data – illustrative purposes only

SMALL MULTIPLES – all info on one page * hypothetical data – illustrative purposes only

Case Studies --- each table does 2 case studies. What is data telling you? How did you analyze it? What would your next steps be based on these results? Any data display issues?

Case Study 1 The Infection Control team collected data for 25 days in a row. The first week they did a poster campaign and made hand sanitizer available outside every room. The goal of the campaign is to have 95% compliance.

Case Study 2 The VP of Surgery has asked the Ortho Dept to reduce turnaround time. This is the data they collected over 25 days. What would you advise?

Case Study 3 This department has been working to implement the surgical checklist and has an aim that 85% of surgeries will have the checklist completed. You have been working with the team and have been collecting data weekly for the last 25 weeks. What does this data show and what are next steps?

Case Study 4 This department has been working to implement the surgical checklist and has an aim that 85% of surgeries will have the checklist completed. You have been working with the team and have been collecting data weekly for the last 25 weeks. What does this data show and what are next steps?

References BCPSQC Measurement Report Langley GJ, Moen R, Nolan KM, Nolan TW, Norman CL, Provost LP (2009) The Improvement Guide (2nd ed). Provost L, Murray S (2011) The Health Care Data Guide.