Was There Improvement? Hospital Data Average LOS (days) % Complications 2007 (last half) % 2008 (first half) % Difference %
About Metrics Metric is derived from the word measure. Metric is derived from the word measure. Whatever metric you decide to use will depend on the process and whether data can be obtained at steps within that processWhatever metric you decide to use will depend on the process and whether data can be obtained at steps within that process Metric should be meaningful and representative enough to judge work, effort, quality, timeliness, and satisfaction.Metric should be meaningful and representative enough to judge work, effort, quality, timeliness, and satisfaction.
Metrics Healthcare metrics are usually in the form of time, count, proportion, costs and evaluation. Although metrics are made for a number of reasons including control, improvement and compliance, we will focus on improvement.
Example Metrics Time –Length of stay – days –Time in waiting rooms – hours –Time to process a claim –Time spent assembling patients charts
Example Metrics Count (Total or Average) –Number of readmits –Number left without being seen –Number of patients with incomplete lab slips –Number of patients contacted –Number of errors occurring in a procedure –Comparative count data
Example Metrics Proportion –% Complications –% Readmits –% Understaffed Costs –Cost per case or patient –Total costs of procedures –Total cost of salaries –Savings per case
Example Metrics Evaluation –Customer surveys –Customer complaints
Metrics – Measuring Improvement Improvement, as measured with data, is based on comparing for differences in the averages of populations or differences from some standard or target. And on comparing for statistical differences in variation dispersion (distribution) about means.
Metrics – Parametric vs Nonparametric Analysis Tests for differences in averages are either parametric or nonparametric. Parametric tests are considered more powerful in measuring differences in means and can be used to measure differences in the amount of dispersion. Nonparametric tests can be used to compare non-normal distributions.
Metrics – About Parametric Tests Parametric tests are based on measurements that have population distributions that are approximately normal (bell shaped). Large samples averages of data from non- normal distributions are normally distributed and are therefore acceptable for parametric testing (central limit theorem) Small sample averages from non-normal distributions tend to keep approaching normality as the sample size increases.
Parametric (Normal) Distribution
Metrics – Parametric Restrictions for Differences in Means The data must be approximately normally distributed. The variances (measures of normal variation) must be equivalent, at least to an acceptable degree. Note: Parametric testing is more restrictive than the non-normal counterpart.
Transforming Non-normal Data for Parametric Analysis Data from non-normal distributions can be normalized by transformation. The most common is logarithmic transformation. For proportions, the arcsine transformation might be necessary.
Metrics – About Nonparametric Tests Nonparametric tests are suitable for comparing data from non-normal distributions. Nonparametric tests rank the data by order of magnitude and compare the medians rather than the means. Small samples from suspected non-normal population distributions can be compared by nonparametric testing.
Metrics – Nonparametric Tests Since nonparametric testing is distribution free, it is suitable for data that are commonly collected in healthcare that are not normally distributed, such as small sample counts of things, proportions and some measurements. Note: Statistical testing for differences in averages needs to be done, only when distributions overlap and it is not clear if there is truly a difference in the averages.
Guidelines for Choosing Parametric or Nonparametric Sample Size? Badly Skewed? Increase to 50? Transform Data? Mann-Whitney.. (Nonparametric) Normal? T-test (Parametric) Z-Test (Parametric) > 30< 30 Y Transform? N N Y N Y Y N YN
Metrics – Parametric Application Parametric tests –z test for comparing the averages of two large samples (n>30) –t test for comparing the averages of two small samples. –t test for matched pairs. –Minimum requirements for these tests are the average, standard deviation and sample size. Individual tests are preferred in order to confirm normality and some statistical software packages require the individual data for input.
About P Values P-Value is the risk in concluding that a difference exists when in fact it does not. Also known as Type I error or alpha risk. P-Value Risk % Confidence %
Comparing Differences in Means for Large Samples - Minitab Sample N Mean StDev SE Mean Difference = mu (1) - mu (2) Estimate for difference: % CI for difference: ( , ) T-Test of difference = 0 (vs not =): T-Value = 1.97 P-Value = DF = 912 Hospital Data – 6 Month LOS Dif 0.39 Stat>Basic Statistics>2 Sample t Test>Summarized Data
Comparing Differences in Proportions for Large Samples - Minitab Sample X N Sample p Difference = p (1) - p (2) Estimate for difference: % CI for difference: ( , ) Test for difference = 0 (vs not = 0): Z = 1.66 P-Value = Hospital Data – 6 Month % Complications Dif 2.06 Represents a binomial experiment (independent!) Note: Events were only 21 and 12 in such a large sample Stat>Basis Statistics>2 Proportions>Summarized Data
Metrics – Nonparametric Application Nonparametric tests commonly used: –Man-Whitney U-test for two unmatched samples. –Kruskal-Wallis for more than two unmatched samples. –Wilcoxon test for matched pairs. –Minimum requirements for these tests are the individual tests (for ranking). –Restrictions - The probability distributions do not have to be normal but must be similar.
Mann-Whitney Application Drug A Drug B Patients with arthritis have been treated with two different drugs at a clinic based on the preference of physicians. A healthcare worker wants to know if one drug is more effective than the other for pain and chooses patients at random to complete a survey consisting of 10 questions. Each has a scale from 0 to 10, with 0 being without any pain and 10 being extreme pain (i.e. low score has less overall pain). Scores are as follows:
Mann-Whitney Application Results - Minitab Mann-Whitney Test and CI: Drug A, Drug B N Median Drug A Drug B Point estimate for ETA1-ETA2 is Percent CI for ETA1-ETA2 is (-20.00,0.00) W = 60.0 Test of ETA1 = ETA2 vs ETA1 not = ETA2 is significant at The test is significant at (adjusted for ties) Stat>Nonparametrics>Mann-Whitney
Kruskal-Wallis Application A healthcare worker collects the waiting times of five different patients chosen at random from out-patient clinics A, B, C, and D to determine if there are differences between the clinics. A B C D
Kruskal-Wallis Results - Minitab Kruskal-Wallis Test on Time Clinics N Median Ave Rank Z A B C D Overall H = DF = 3 P = H = DF = 3 P = (adjusted for ties) Stat>Nonparametrics>Kruskal-Wallis
Wilcoxon Test for Matched Pairs Application A trial was conducted to evaluate the effects of alcohol on reaction time. Twelve students were chosen at random to consume the same amount of a certain alcoholic beverage. Reaction times (minutes) to complete a complex task was measured before and after the beverage consumption. Results were as follows: A G B H C I D J E K F L Did the alcohol affect the performance?
Wilcoxon Matched Pairs Results - Minitab Wilcoxon Signed Rank Test: C3 Test of median = versus median not = N for Wilcoxon Estimated N Test Statistic P Median C Note: For Mann-Whitney P = Calc>Calculator (C1-C2) Stat>Nonparametrics>1 Sample Wilcoxon (Test Median)
Metrics – Frequencies Chi-square tests for analyzing frequencies are very useful for things that are counted and classified on nominal scales (categories) such as sex, age-group, blood group type, ethnic origin and so on. Tests commonly used are: 1.Test for Homogeneity 2.Goodness of fit 3.Test of association – Contingency tables
Chi-square Tests The Test for Homogeneity determines if various counts are consistent with random variation. Goodness of fit is same as homogeneity test except it compares frequencies against some expected frequency of each according to some rule, theory, etc. Contingency tables are used with two or more rows of observations to test for association between variables.
Chi-square Homogeneity Test Application There are four standard surgical techniques A, B, C and D being used in a particular type surgery. In order to find out if any one is preferred, 160 randomly selected surgeons were asked which technique he or she preferred. Results were as follows: A B C D Is there sufficient evidence to conclude that there is a difference in preferences for the four techniques?
Chi-square Homogeneity Test - Minitab SET C (Observed values) END SET C (Expected values) END LET K1=SUM((C1-C2)**2/C2) PRINT K1 CDF K1 K2; CHISQUARE 1. LET K3=1-K2 PRINT K3 Data Display K (Chi-square statistic) Data Display K (P-value) Edit>Command Line Editor
Chi-square Goodness of Fit Test Application Suppose in the previous example of the test for homogeneity, a study had been conducted in the past that concluded that there was a preference among doctors for technique B according to the following percentages: A B C D 20% 40% 20% 20% Does this latest sample of 160 surgeons indicate a change from these percentages?
Chi-square Goodness of Fit - Minitab SET C (Observed values) END SET C (Expected values calculated from expected percentages) END LET K1=SUM((C1-C2)**2/C2) PRINT K1 CDF K1 K2; CHISQUARE 3. LET K3=1-K2 PRINT K3 Data Display K (Chi-square statistic) Data Display K (P-value) Edit>Command Line Editor
Contingency Table Application In a study of the implications of smoking habits to certain regions, a survey of 252 men from these regions showed the following: Region 1 Region 2 Region 3 Total Non-smokers Smokers Total Is there an association between smoking and regions?
Contingence Table Results - Minitab Expected counts are printed below observed counts Chi-Square contributions are printed below expected counts C10 C11 C12 Total Total Chi-Sq = 7.192, DF = 2, P-Value = Note: Yates Correction not needed. Stat>Tables>Chi-Square (Table in Worksheet) (Observed) (Expected) (Cell Chi-sq) (Non-sm) (Smokers) (same) Total of all cells
Metrics – Sample Size The larger the sample size, the smaller the differences in averages that can be detected. Often, the amount of difference in averages that is desired to be detected is balanced against the size of the sample required for this detection and the amount of error that can be tolerated for this detection.
Metrics – Sample Size However, very small differences in averages can result in large savings in costs, thus making large samples necessary. On the other hand, there are diminishing returns as the sample size becomes larger and larger.
Subjects Not Covered Sample size for detecting differences Sampling methods All restrictions Precision Correlation Regression Analysis of variance