Quality Control and Patient Risk Curtis A. Parvin, Ph. D Quality Control and Patient Risk Curtis A. Parvin, Ph.D. Manager of Advanced Statistical Research Quality Systems Division

Laboratory QC and Patient Risk For the clinical laboratory patient risk is related to the reliability of the patient results they produce and report. Laboratory Quality Control principles have been connected to the concept of patient risk for decades. Specify the quality required of a patient result. Determine the magnitude of “critical” out-of-control conditions with unacceptably high probability of producing patient results that fail to meet the required quality. Design QC procedures with adequate power to detect “critical” out-of-control conditions. 2 |

Risk Management A number of risk management guidelines have been published recently. For manufacturers ISO 14971: Medical devices – Application of risk management to medical devices CLSI EP18: Risk management techniques to identify and control laboratory error sources For the laboratory ISO 22367: Medical laboratories – Reduction of error through risk management and continual improvement CLSI EP23: Laboratory quality control based on risk management 3 |

The Risk Management Process Risk Management provides a formal approach to Identify potential failure modes in the lab Rank the identified failure modes in terms of their risk Establish policies and procedures to prevent or reduce (mitigate) the risks Focus on the high ranked risks This is generally a qualitative process Risks are computed as a combination of The probability of occurrence of harm The severity of harm 4 |

Statistical QC and Risk Management The question that we will address today is: How do statistical QC design principles that address questions such as How many QCs should I examine? What QC rule(s) should I use? How frequently should I schedule QC evaluations? fit into the overall risk management process? 5 |

Risk Assessment (EP23 Figure) Hazard Identification Create a process map Identify potential failures in each process step Determine mechanisms in place to prevent or detect a failure Risk Estimation Assess the likelihood or probability of harm for each failure Assess the severity of harm to a patient from each failure Risk Estimation Is the residual risk of harm clinically acceptable Risk Control What control processes are needed to lowerrisk to an acceptable level No Yes The Laboratory’s QCP Compile set of QC process into QCP Review QCP for conformance to regulatory and accreditation requirements Document and implement the set of control processes as the laboratory’s QCP 6 |

Risk Assessment (EP23 Figure) Hazard Identification Create a process map Identify potential failures in each process step Determine mechanisms in place to prevent or detect a failure Risk Estimation Assess the likelihood or probability of harm for each failure Assess the severity of harm to a patient from each failure Risk Estimation Is the residual risk of harm clinically acceptable Risk Control What control processes are needed to lowerrisk to an acceptable level No Yes The Laboratory’s QCP Compile set of QC process into QCP Review QCP for conformance to regulatory and accreditation requirements Document and implement the set of control processes as the laboratory’s QCP 7 |

Probability of Harm Sequence of Events Creating Risk of Harm for a Patient Hazardous Situation P1 P2 P3 P4 P5 P6 Initiating cause Testing process failure Incorrect result generated Incorrect result reported Misdiagnosis Hazardous medical action Patient harmed EP23-A, Figure 6 ISO 14971, Figure H.1 ISO/TS 22367, Figure B.1 Estimates of the probability of harm are desirable For each potential mode of failure Collectively for all possible failure modes 8 |

Estimating Probability of Harm Initiating cause Testing process failure Incorrect result generated Incorrect result reported Misdiagnosis Hazardous medical action Patient harmed P1: The probability that a testing process failure occurs 9 |

P1: Probability of a Failure Mode P1 = how likely is a given failure mode going to occur or more commonly P1 can be the mean time between failures (MTBF) Information regarding MTBF may come from Manufacturer Historical failure data Literature MTBF can be converted to mean number of patient results between failures (MPBF) MTBF = 30 days 150 patient results examined per day (on average) MPBF = 30 * 150 = 4500 patient results 10 |

Estimating Probability of Harm MPBF MPBF P1 P2 P2 P3 P4 P5 P6 Initiating cause Testing process failure Incorrect result generated Incorrect result reported Misdiagnosis Hazardous medical action Patient harmed P2: The probability that a testing process failure generates incorrect patient results 11 |

P2: Probability of Producing an Incorrect (Unreliable) Result The number of unreliable patient results produced during the existence of an out-of-control condition depends on The quality required for patient results The type and magnitude of the out-of-control condition The power to detect the out-of-control condition When (how often) QC evaluations are performed An out-of-control condition may result in many unreliable patient results being produced 12 |

Unreliable Patient Results ISO 15189 Clause 5.6.1: Laboratory QC should assure that patient results meet the quality required for their intended use The quality of a patient result depends on the difference between the correct value and the value reported. If the error in a patient’s result exceeds the allowable total error (TEa) The result is considered unreliable (or incorrect). It creates a hazardous situation for the patient. 13 | 13

Expected Number of Unreliable Patient Results Produced: E(NU) The number of unreliable patient results produced during an out-of-control condition (red asterisks) will depend on The magnitude of the out-of-control condition The power of the QC rule – Detection The number of patient specimens between QC events E(Nu) = Expected number of unreliable patient results generated during an out-of-control condition 14 | 14

Estimating Probability of Harm MPBF E(Nu) E(Nu) P2 P3 P3 P4 P5 P6 Initiating cause Testing process failure Incorrect result generated Incorrect result reported Misdiagnosis Hazardous medical action Patient harmed P3: The probability that incorrect results that are generated will be reported 15 |

P3: Probability of Reporting an Unreliable Patient Result The probability of reporting an unreliable result that leads to an incorrect action depends on The number of unreliable results produced because of an out-of-control condition How and when results are reported The likelihood of identifying and correcting a reported unreliable result before an incorrect action is taken 16 |

Expected Number of Unreliable Patient Results Reported E(Nu) E(Nuf) E(Nuc) The patient results produced during an out-of-control condition can be divided into Results prior to the last accepted QC event (Pre) Results since the last accepted QC event (Post) Bracketed QC: results aren’t reported Immediate reporting: results should be repeated and updated in a timely fashion E(Nuf) = Expected number of unreliable “final” results E(Nuc) = Expected number of unreliable “correctable” results 17 | 17

Estimating Probability of Harm E(Nuf) MPBF E(Nu) E(Nuf) P3 P4 P4 P5 P5 P6 P6 Initiating cause Testing process failure Incorrect result generated Incorrect result reported Misdiagnosis Hazardous medical action Patient harmed P4: The probability that an incorrect result leads to a misdiagnosis P5: The probability that a misdiagnosis leads to a hazardous medical action P6: The probability that a hazardous medical action leads to patient harm P4*P5*P6: The probability than an incorrect result leads to patient harm 18 |

Probability of Patient Harm from an Unreliable Patient Result The probabilities (P4, P5, P6) are associated with activities outside the laboratory Their product equals the probability of patient harm given that an unreliable result has been reported Ph|u = P4*P5*P6 Estimates of the probability of harm given an incorrect result are based on; Medical literature Medical judgment Consultation with clinical colleagues 19 |

Estimating Probability of Harm MPBF E(Nu) E(Nuf) P4 P5 Ph|u P6 Initiating cause Testing process failure Incorrect result generated Incorrect result reported Misdiagnosis Hazardous medical action Patient harmed 20 |

Estimating Probability of Harm: Risk Management + Statistical QC MPBF E(Nu) E(Nuf) Ph|u Initiating cause Testing process failure Incorrect result generated Incorrect result reported Patient harmed 21 |

Harmed Patients from Failure Mode Occurrence Ph|u*E(Nuf) MPBF * * * E(Nuf) E(Nu) E(Nuc) Probability of patient harm is The expected number of patients harmed due to incorrect results from a failure mode: Ph|u* E(Nuf) Divided by the average number of patient results examined between failure mode occurrences: MPBF Probability of Harm = Ph|u* E(Nuf) / MPBF 22 | 22

What Info Do We Need to Predict E(Nu) and E(Nuf)? The quality requirement for the analyte Allowable Total Error (TEa) Test method performance Instrument imprecision and bias QC procedure QC rule(s) used Number of QC levels evaluated at each QC event QC schedule (average number of patients examined for the analyte between QC events) Needed to compute the probability of an unreliable patient result The quality requirement for the analyte Allowable Total Error (TEa) Test method performance Instrument imprecision and bias QC procedure QC rule(s) used Number of QC levels evaluated at each QC event QC schedule (average number of patients examined for the analyte between QC events) The quality requirement for the analyte Allowable Total Error (TEa) Test method performance Instrument imprecision and bias QC procedure QC rule(s) used Number of QC levels evaluated at each QC event QC schedule (average number of patients examined for the analyte between QC events) Needed to compute the expected # of affected patient results 23 |

Probability of an Unreliable Result Due to a Systematic Error Condition 24 | 24

Probability of an Unreliable Result Depends on TEa, Method CV and Bias 25 | 25

Expected # Affected Depends on QC Rule and QC Schedule 26 |

Expected # of Unreliable Patient Results 27 | 27 |

Ways to Reduce (or Manage) Patient Risk Estimate Risk Reduce Risk Rate of Occurrence of Failure Modes Yes: manufacturer info, technical bulletins, product alerts Some: lab environment/processes training Very little: Medical devices Number of Incorrect Results Reported Yes: Statistical QC design & evaluation A lot: Detection in analytical phase A little: Detection in pre- and post- analytical phases Probability of Harm from an Incorrect Result Yes: Medical literature, medical judgment, consultation Very little: Report formatting, education 28 |

Ways to Reduce (or Manage) Patient Risk Manage the risk of unreliable patient results in the worst case Limit the maximum values for E(Nuf) and E(Nuc) Manage the overall expected risk of unreliable patient results Limit the areas under the E(Nuf) and E(Nuc) curves Manage the “defect rate” for unreliable patient results Requires an estimate of the rate at which out-of-control conditions occur 29 |

Summary Risk management Statistical QC The laboratory has two main mechanisms to estimate and reduce the risk of patient harm: Identify as many potential failure modes as possible and seek ways to reduce occurrences of the identified failure modes Risk management Implement QC strategies that minimize the number of incorrect patient results that are reported when a failure mode does occur Statistical QC The combination enables a laboratory to address the full spectrum of patient risk implications for their operations 30 | 30

References Parvin CA. Assessing the impact of the frequency of quality control testing on the quality of reported patient results. Clin Chem 2008;54:2049-54. Parvin CA, Yundt-Pacheco J, Williams M. Analytical assessment in the clinical laboratory: assessing analytical quality goals when the same analyte can be tested on multiple systems is explored. Adv Admin Lab 2011;20(1):28-31. Parvin CA, Yundt-Pacheco J, Williams M. The focus of laboratory quality control: why QC strategies should be designed around the patient, not the instrument. Adv Admin Lab 2011;20(3):48-9. Parvin CA, Yundt-Pacheco J, Williams M. Designing a quality control strategy: in the modern laboratory there are 3 questions that must be answered. Adv Admin Lab 2011;20(5):53-4. Parvin CA, Yundt-Pacheco J, Williams M. The frequency of quality control testing: QC testing by time or number of patient specimens and the implications for patient risk are explored. Adv Admin Lab 2011;20(7):66-9. Parvin CA, Yundt-Pacheco J, Williams M. Recovering from an out-of-control condition: the laboratory must assess the impact and have a corrective action strategy. Adv Admin Lab 2011;20(11):42-4. Parvin CA, Yundt-Pacheco J, Williams M. Sigma metrics, total error budgets, and quality control: Make sure your test system performance and quality control procedures are aligned with your quality goals. Adv Admin Lab 2012;21(1):40-4. 31 |

32 |

33 |

THANK YOU! 34 | 34