Objectives Understand and appreciate the purpose of indicators Identify criteria for designing, appraising or choosing indicators Apply indicators for research, evaluation or quality improvement.
Background What is an indicator? Succinct measures that describe as much as possible about a system Why use indicators? Understanding – how a system works and how it might be improved (research) Performance monitoring – to assess if the system is performing to at an expected level, compare and improve a system (improvement) Accountability – to inform assessment of effectiveness, efficiency and of responsibility (evaluation)
Considerations Indicators indicate: Cannot capture and reflect the complexity of a system program or service Must be considered within context Encourage explicitness: Help to clarify our understanding of a system Facilitate communication of expectations Numerically based: Rates, ratios etc. Not specifically designed to detect fault: Can be used for interpreting both strengths and weaknesses
Improvement Indicators primarily used to measure systems and outcomes in health care. Goal - improvement Understand how things work leads to understanding of how things can be done better Measurement alone does not lead to improvement.
Anatomy Basic construction How to deconstruct and assess Metadata (indicator) Title, rationale, and how indicator is defined/constructed Data Information that is entered into the indicator
Metadata Metadata will help asses if an indicator is: Important and relevant Able to be populated with reliable data Likely to have desired effect when communicated well
Metadata What is being measured? Why is it being measured? How is this indicator defined? Who/what does it measure? When does it measure? Absolute or proportion? What is the source of data? What is the accuracy/completeness of the data? Any considerations, warnings, caveats? Are specific tests needed to test the meaning of the data?
Statistics Canada – Catalogue no X Statistics Canada – Health Indicators
Definition: Wait time for hip fracture surgery (same/next day) Proportion with surgery same or next day: risk-adjusted proportion of hip fracture patients aged 65 and older who underwent hip fracture surgery on the day of admission or the next day. Wait time for surgery following hip fracture provides a measure of the access to care. While some hip fracture patients need medical treatment to stabilize their condition before surgery, research suggests patients typically benefit from timely surgery in terms of reduced morbidity, mortality, pain, length of stay in hospital, as well as improved rehabilitation. Rates for Quebec are not available due to differences in data collection. Source(s): Canadian Institute for Health Information, Discharge Abstract Database.
Group Activity Deconstruct the Wait Time for Hip Fracture Surgery indicator using the Metadata Assessment Questions in the handout. Identify information that you would need to fully understand the composition of this indicator. Report back.
Data Is the indicator populated with the best available data? Reliable Valid Bias Completeness Error Convenience
Criteria for Good Indicators 1. Importance and relevance 2. Validity 3. Feasibility 4. Meaning 5. Implications Adapted from: The Good Indicators Guide: Understanding how to use and choose indicators. NHS Institute for Innovation and Improvement, 2008
1. Importance and Relevance Does the indicator measure what is relevant in the system? Indicators must relate to objectives of the system Key parts of process and/or outcome should have associated indicators Is there balance? All important system components are covered No over or under-represented components Will the indicators promote consensus? Well formed indicators can be helpful in developing consensus on the objectives of the system
2. Validity Does the indicator actually measure what it is intended to measure? Accuracy and degree to which the indicator actually measures the construct May require validation if existing valid indicators are not available or appropriate Translation validity Face validity Content validity Criterion-related validity Predictive validity Concurrent validity Convergent validity Discriminant validity http ://
3. Feasibility Are reliable data available to populate the indicator? Are data available at appropriate times? Are comparator data similarly available? Can the cost/resource allocation be justified?
4. Meaning Will the indicator be sensitive to detect and demonstrate meaningful changes in the system? (it should not identify random variations) Can high and low indicator values provide appropriate signals to take action? Can the indicator be deconstructed to provide insight about specific results or patterns? Will others accept the indicator as having meaning?
5. Implications What will you do if an indicator suggests further action? Will indicator results induce perverse incentives and unintended uses? Is there sufficient lag time in measurement so as to allow interventions to have an effect?
MacNamara Fallacy The MacNamara Fallacy is named after the Robert McNamara, the US Secretary of Defence in the 1960s who was obsessed with quantifying the Vietnam War in a way that tended to ignore what was truly going on. MacNamara argued that the ratio of Viet Cong losses to US losses was an important measure of effectiveness
MacNamara Fallacy The first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which can’t be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can’t be measured easily really isn’t important. This is blindness. The fourth step is to say that what can’t be easily measured really doesn’t exist. This is suicide. Charles Handy, ‘The Empty Raincoat’, page 219.
Qualitative Indicators Qualitative indicators such as perceptions are important. Qualitative indicators may enable or hamper improvement/change. It is important to balance quantitative with qualitative indicators to provide context
SPICED Indicators Subjective: Informants have a special position or experience that gives them unique insights which may yield a very high return on the investigators time. Participatory: Indicators should be developed together with those best placed to assess them. Interpreted and Communicable: Locally defined indicators may not mean much to other stakeholders, so they often need to be explained. Cross-Checked and Compared: The validity of assessment needs to be cross-checked, by comparing different indicators and progress, and by using different informants, methods, and researchers. Empowering: The process of setting and assessing indicators should be empowering in itself and allow groups and individuals to reflect critically on their changing situation. Diverse and Disaggregated: There should be a deliberate effort to seek out different indicators from a range of groups. This information needs to be recorded in such a way that differences can be assessed over time. Roche (2002)
Program Theory A statement of goals accompanied by the underlying assumptions that guide a system/program/service delivery strategy and are believed to be critical to producing the desired outcomes.
Program Theory Considerations Who are you serving? What are you striving to accomplish for the population that you are serving? What strategies do you believe will help you successfully accomplish your goals?
Logic Model A program logic model links outcomes with program activities … and the theoretical principles of the program” (Kellogg, 2001) Thus, logic models set up both formative and summative evaluation questions Evaluative answers are “useful” when they reduce the risks of making the wrong decision
Types of Evaluation Formative “Improve” Periodic and timely Focus on program activities and outputs Leads to early recommendations for program improvement Summative “Prove” Were resources committed worthwhile Focus on outcomes and impact Measures value of program based on impact * Kellogg logic model development guide
Resources Dental Clinic Coordinator Community Health Director Staff dentist Staff pediatrician Medical providers Money for supplies Activities Training Develop curriculum Two one-hour didactic trainings to medical providers in oral health assessment One-on-one training to medical providers on oral health Outreach Order dental supplies for packets Make up packets Distribute to parents at end of each visit Outputs Training # of two-hour trainings held # of one-on-one trainings held # of medical providers trained Outreach # of parents/children receiving packets Outcomes Medical providers demonstrate accurate oral health assessment, education and prevention activities More children receive high- quality oral health assessment, education and prevention activities during well-child visits Parents/children are more knowledgeable about oral health and caring for children’s teeth Reduced incidence of caries in children at the community health center Program Goal: To improve the oral health of low-income children who receive primary care in a community health center Example Logic Model
Example – Logic Model Linked with Indicators
Indicators Linked with Program Framework
Group Activity Using the DoctorDad logic model: Develop one indicator for an anticipated output Develop one indicator for an anticipated outcome Create a definition and identify sources of data Consider your indicator with respect to Indicator Quality and Planning checklist. Report back.
Variation indicators indicate – variation informs action Objectives: Understanding variation. Know when to investigate. How to respond to variation.
Statistical Process Control Common Cause Variation Everyday and inevitable variation which tends to be small, with observed values close to the average. Special Cause Variation Variation outside the historical experience base. Signal of some important change in the system.
Statistical Process Control Statistical Process Control (SPC) is an effective method of monitoring a process through the use of control charts. Control charts enable the use of objective criteria for distinguishing background variation from events of significance based on statistical techniques.
Statistical Process Control Allows you to determine: That the system is working with an acceptable level of performance and there are no outliers No action needed That the system is working with an acceptable level of performance and there are outliers Address outliers That the system’s average level of performance is not acceptable Address entire system
SPC Run Chart Time ordered presentation of observations A centre line (average or median) of all the observations plotted.
SPC Control Chart three basic components: a centre line, usually the average of all the samples plotted. upper and lower statistical control limits that define the constraints of common cause variations. performance data plotted over time.
SPC Run Chart Example Steps to create a Run Chart Ideally, there should be a minimum of 15 data points. Draw a horizontal line (the x-axis), and label it with the unit of time. Draw a vertical line (the y-axis), and scale it to cover the current data, plus sufficient room to accommodate future data points. Label it with the outcome. Plot the data on the graph in time order and join adjacent points with a solid line. Calculate the mean or median of the data (the centre line) and draw this on the graph.
Run Chart Run chart of the number of red beads drawn across 25 draws
Run Chart – What to look for: Useful Observations – number of observations that do not fall directly on centre line. Run – sequence of one or more consecutive useful observations on the same side of the centre line. Trend – sequence of successive increases or decreases in useful observations
Run Chart Rules Identifying Special Cause Variation Number of Runs If there are too few or too many runs in the process.
Run Chart Rules Identifying Special Cause Variation Number of Runs If there are too few or too many runs in the process.
Run Chart Rules Identifying Special Cause Variation Shift If the number of successive useful observations falling on either side of the centre line is greater than 7.
Run Chart Rules Identifying Special Cause Variation Trend If the number of successive useful observations increasing or decreasing is greater than 7.
Run Chart Rules Identifying Special Cause Variation Zig-Zag If the number of useful observations increasing or decreasing alternately (zig-zag pattern) is greater than 14.
Control Chart Rules Identifying Special Cause Variation Control Limits – 2SD above or below centre line If there is one or more observations beyond the control limits Warning Limits – 3SD above or below centre line If there are two successive observations beyond the control limits Control Limit 2SD Warning Limit 3SD
Example – uniform system underperforming
Control Chart Resource
Addressing Special Cause Variation When you detect a special cause: Control any damage or problems with an immediate, short-term fix. Once a quick fix is in place, search for the cause. Ask people in the process what was different that time. What was out of the ordinary? It might not have been much – an unexpected emergency, a change in schedules, or new employees. Once you have discovered the special cause, you can develop a longer-term remedy. Most special causes have a negative impact on the output of the process and need to be removed. Occasionally, a special cause can have a positive impact depending on the nature of the process. If this is the case, finds ways to capture and integrate it into the system. Avoid these mistakes: Be careful not to view short-term fixes as a permanent solution or the process will never be improved. Changing the process to accommodate the special cause. This usually adds cost and bureaucracy. Blaming individuals. Warn employees to do better. People can only do as well as the system allows.
Addressing Common Cause Variation Common Cause Variation – ideal situation for quality improvement If a process indicator is stable, or in statistical control, does not mean that its results are satisfactory. An indicator may be very consistent, but not meeting an expected outcome Variation can be systematically reduced, even in stable processes, enabling a gradual tightening of control limits, and an overall increase in quality.