Presentation on theme: "Module 6: QC Basic Rules and Charts"— Presentation transcript:
1 Module 6: QC Basic Rules and Charts Analysis, Chartsand Interpretation of RulesQuality Control: Samples, Charts and Rules
2 Learning ObjectivesUpon completion of this module you will be able to:Define: Control materials, Calibration materials, Levey-Jennings Chart, Westgard Single Rule and Multi-Rule Criteria, Systemic Error, & Random ErrorDescribe a quality control sample in terms of assayed versus unassayed and their uses
3 Learning ObjectivesUpon completion of this module you will be able to:Discuss the following Westgard rules:13s12sApply the Westgard rules listed above to decide whether an analytic run is accepted or rejected.List which Westgard rules detect systematic versus random error.22s6x7tQuality Control: Samples, Charts and Rules
4 DefinitionsControl materials: Protein-based materials made or manufactured with a composition similar to patient samples that have or have not been analyzed for concentration so they have expected or no expected values.The International Federation of Clinical Chemistry defines a control solution or control material as a "specimen or solution which is analysed solely for quality control purposes, not for calibration" .Quality Control: Samples, Charts and Rules
5 Definitions Certified reference material (CRM) “A reference material that has one or more values certified by a technically valid procedure and is accompanied by, or is traceable to, a certificate or other document that is issued by a certifying body.” [CLSI] NBS (National Bureau of Standards)Calibrator: purified solution used to adjust electrical output of an instrument and should trace back to a CRM.The International Federation of Clinical Chemistry defines a control solution or control material as a "specimen or solution which is analysed solely for quality control purposes, not for calibration" .Quality Control: Samples, Charts and Rules
6 DefinitionsLevey-Jennings Chart: A visual presentation of daily quality control values plotted on a chart using mean, standard deviation, and +/- 3 Std Dev range criteriaWestgard Single Rule and Multi-Rule Criteria: A set of rules used to evaluate quality control values as acceptable or notThe International Federation of Clinical Chemistry states that calibrators are purified solutions used to adjust the output of an instrument to a particular concentration of an analyte.Quality Control: Samples, Charts and Rules
7 DefinitionsSystemic Errors: Errors in the test system usually caused by a malfunction that affects all testsRandom Errors: Errors that statistically occur unpredictably and do not indicate system malfunction but do indicate an individual test or component malfunction
8 Need for Quality Control Samples Patient results are “unknowns”. Analysis of the patient sample produces a number.But…is the result valid?Patient may have previous result. But…Change from previous may be expectedNo change from previous result may be expectedThe product of a testing process is a numerical result. Unlike a physical product that can be inspected to assess whether it looks good or bad, you can't look at a test result and tell whether it's valid. You could look to see if that patient had the same test measured recently but you don’t know whether to expect it to change because the patient has improved or worsened or if you expect it to stay the same. Quality control samples are like patient samples only they have a known amount of analyte. QC samples (knowns) can be analysed along with patient samples (unknowns) to test the validity of all results in the run.Quality Control: Samples, Charts and Rules
9 Need for Quality Control Samples Quality control samples are “knowns.” They have been made or manufactured with a composition similar to patient samples and have been analyzed for concentration before they are put into use so they have expected results.Therefore, if Quality Control samples are analyzed and expected results are obtained, we can assume our system is operating correctly.orIf Quality Control samples are analyzed and do not give expected results, what should be done?
10 Quality Control Samples Are Preserved to maintain accurate and precise resultsAssayed –provided with mean and s for analytesUnassayed – you must determine mean and sThe purpose of a statistical quality control procedure is to monitor the analytical quality of the measurement during stable operation, detect changes from the stable operation, and eliminate reporting of results with medically important errors. Control samples should be similar to patient specimens. They can be purchased from manufacturers who make them from donated patient samples that have been preserved. It is important to treat the control materials carefully. Attributes of controls are the stability, low vial to vial variability, assayed versus unassayed, and how to reconstitute or treat the control material.Quality Control: Samples, Charts and Rules
11 Why is it advisable to run controls at different levels? Interactive OpportunityWhy is it advisable to run controls at different levels?Gather input from participants.One acceptable response is that running a control that has a result falling in the reference range and at least one control that falls at the medical decision level is value to test the analytical phase.Quality Control: Samples, Charts and Rules
12 Quality Control Sample Use Analyzed solely for quality control purposesNot for calibrationShould be used for qualitative and quantitative testing as known results to check if method to measure patient results is reliable.The International Federation of Clinical Chemistry defines a control solution or control material as a "specimen or solution which is analysed solely for quality control purposes, not for calibration" . Even with assayed controls you should assay the selected control materials under routine operating conditions to characterize the expected measurement variation and establish your own expected values. This usually involves obtaining at least 20 values and calculating the mean and standard deviation. There are a number of pitfalls from using bottle values or other estimates of the means, standard deviations, and control limits without verifying the results first.Quality Control: Samples, Charts and Rules
13 Interactive Opportunity Is it sufficient to use only one calibrator to set the output of an analyzer?Gather input from participants.One acceptable answer is that calibration should initially be with standards that test the lowest, mid and highest level of linearity. Validation of the curve using one calibrator is sufficient followed by analysis of quality control materials.Quality Control: Samples, Charts and Rules
14 Qualitative Versus Quantitative Results What are some examples of qualitative tests in the laboratory?What quality control samples would you use to verify the test is working so that patient results may be reported?Take answers from participants and put on flipchart. Examples: HIV with + and neg controls, RPR or VDRL with reactive and NR controls, ABO with A1 cells (positive for A), B cells (positive for B) O cells or antibody screen cells (negative for A and B), Gram stain (E coli for GNB and Staph for GPC), many others may apply. Most urine testing and hematology testing is quantitative and should be reminded of need for analyzing known control samples or blood films even for manual microscopy to check technician results.
15 QC Results and Expectations for Qualitative Tests Quality Controls will be a known positive and known negative materialThe results will be recorded in a log book with the date, techs initials, and either ‘positive’ or ‘negative’ written in by the techReason for running controls with qualitative tests is to check that the reagent(s) are working as they are supposed to and give accurate resultsDepending on the test, the (+) and (-) controls will be run once a day (usually every morning) or each time a patient specimen is analyzed. When running a urine dipstick test, the controls are usually run only in the morning and recorded; for other tests, HIV, RPR, etc. the (+) and (-) controls must be run along with the patient to ensure reagents have been added correctly. (with the HIV kits, the controls are build in to the test strip)If the (=) and (-) controls do not give the expected result you must discard the patient test and repeat both the controls and the patient.Quality Control: Samples, Charts and Rules
16 QC Results and Expectations for Quantitative Tests Gaussian distributionIn the laboratory, control charts are used to make it simple to compare today's observed value with what is expected based on past history. Assuming a Gaussian or normal bell-shaped distribution, it would be expected that about 68% of the points fall within 1 SD of the mean, 95% within 2 SD of the mean and 99.7% within 3 SD of the mean. A control chart also has the mean and standard deviations indicated but they are shown on the y axis (as if you turned this Gassian distribution on its side.)Quality Control: Samples, Charts and Rules
17 What does the “Internal” Quality Control on a test cartridge indicate? Interactive OpportunityWhat does the “Internal” Quality Control on a test cartridge indicate?In some test systems it indicates only that sufficient sample has reached all necessary parts of the test cartridge. If you add sufficient water you would get the indication for + and -. Some qualitative immunoassay methods such as rapid HIV tests have internal quality control to test for the ability of the test strip to detect a positive result.Quality Control: Samples, Charts and Rules
18 QC Sample Analysis as an Alarm System QC sample result should:Indicate when the test system is not working properlyNot indicate a false alarmIt should be 100% reliableQuality control is run to see if you can get the right answer for a known sample. The right answer is actually a range of values that are calculated from the mean and standard deviation of past results. That mean and control limits can be shown on a control chart to make it simple to plot new control measurements and see how they compare with the expected range of values. From the technologist’s standpoint, the objectives of the control procedure are simply to "alert me when the method has a problem" and "don't alert me when the method is working okay." These are "true alarm" and "false alarm" situations. Technologists want to know about real problems, but can't afford to waste time when the method's working okay.Quality Control: Samples, Charts and Rules
19 Common QC Sample Rules Single Rule: Accept the analytical run if both quality control results fall within mean +/- 2 standard deviations range.Reject the analytical run and troubleshoot the problem if one or both quality control results are outside +/- 2 standard deviations from the mean.Quality Control: Samples, Charts and Rules
20 QC Sample Result Outside of Expected Range QC value < +2 SD or > -2 SD from the meanExpected most of the timeWith 95% confidence. This means 95% of acceptable results are in this range. 5% could still be acceptable i.e. false alarmQC value > + 3 SD or < - 3 SD from the meanUsually indicates a problem and should be followed up onIt would be very unexpected (0.3% chance) to observe a control value greater than 3 SD from the mean and such an observation usually indicates there is a problem with the method. It is somewhat unexpected to observe a control value greater than 2 SD from the mean, but this will happen at least 5% of the time when analysing 1 control per run, so it may indicate a real problem or it may be a false alarm. It is very common (32% chance) to see individual values beyond 1 SD from the mean, therefore this control limit is of no value for making a judgment about method performance based on a single control value.Quality Control: Samples, Charts and Rules
21 Historical Use of QC Sample Results Shewart developed statistical tools to monitor qualityLevey and Jennings – quality standards for the laboratoryWestgard - established rules for monitoring quality controlWalter A. Shewhart was a statistician at Bell Telephone Laboratories who developed the scientific basis for statistical process control. He set up limits within which the results must lie and found that deviations in the results of a routine process outside such limits indicate that the routine has broken down. Levey and Jennings introduced statistical control methods in clinical laboratories, calculating the average and range (maximum difference), then plotting the average and the range on two different control charts. Levey and Jennings also proposed making duplicate measurements on a patient specimen. Others suggested using a stable reference sample and plotting answers directly on a control chart, the Levey-Jennings chart.Quality Control: Samples, Charts and Rules
22 Levey-Jennings ChartA Control chart is a graphical method for displaying control results and evaluating whether a measurement procedure is in-control or out-of-control. Control results are plotted versus time or sequential run number; lines are generally drawn from point to point to accent any trends, systematic shifts, and random excursions.Control limits are lines drawn on a control chart to provide graphical criteria for assessing whether a measurement procedure is in-control or out-of-control.Quality Control: Samples, Charts and Rules
24 Quality Control Result Limits Mean is the central line.One standard deviation above = + 1 sOne standard dev below = - 1s2 standard dev above = + 2s2 standard dev below = -2s3 standard dev above = + 3s3 standard dev below = -3sThese control limits are usually calculated from the mean and standard deviation (SD, or s) determined for a given control material. Typically the interpretation is based on a specified number of results or points exceeding a certain control limit when in-control patient test results are reported. When out-of-control, the run is rejected and no test results can be reported.Quality Control: Samples, Charts and Rules
25 Calculating Confidence Limits on the Chart For example, Control 1 has a mean of 200 and a standard deviation of 4 mg/dL.What is the 95% confidence limit or mean + 2 s limit?The upper control limit, mean + 2s would be:*4 = 208 mg/dLThe mean – 2s, lower control limit would be:*4 = 192 mg/dL95% confidence limit is mg/dLThis should be a practice problem for participants to set 95% confidence limits. Click to get answers after participants have offered up their answers.Quality Control: Samples, Charts and Rules
26 Quality Control Sample Rules Decision CriteriaIn control versus out of controlControl limitsMean + 2s95% limits5% false rejectionIf 1 result exceeds 2s but not 3s and no other shifts or trends, it is usually a warning but not a reason to reject the results.Control rule means decision criteria for judging whether an analytical run is in-control or out-of-control. It is defined by the number of control measurements, and control limits, often specified as the mean plus or minus a multiple of the standard deviation (s) or sometimes by a specified probability for false rejection. Mean +/- 2s give 95% change of correct decision but 5% false rejection.Quality Control: Samples, Charts and Rules
27 QC Sample Measured with each Analytical Run How long is a run?ManualAutomated methodRun, analytical run, or run length refer to the interval, which could be a period of time or group of samples, for which a decision on control status is to be made. Each batch is a run for a manual procedure. 2 controls should be run with each batch. 24 hours is usually the longest run time for automated chemistry methods; 2 controls should be run at least once every 24 hr. 8 hours is the longest run time for automated hematology methods; 3 controls should be run at least once every 8 hr.Quality Control: Samples, Charts and Rules
28 QC Sample Analysis Procedure Laboratory Quality Control Policy (Quality Manual)stating your laboratory will not report results withoutappropriate QC.Quality control Analysis SOPHow many control samplesHow often runHow documentedHow problems are solvedWhat Westgard rules are used.You should prepare written guidelines to define the QC procedure in detail including how many controls to run per analyte, how often they are run, how to document results and problems and how to solve the problems. This written document is important for teaching laboratory techs the QC procedure and establishing a uniform practice. It is also necessary for meeting regulatory requirements.Quality Control: Samples, Charts and Rules
29 Lab Results Need to Be... Accurate Precise Quality Control Rules should help to assess accuracy and precision.After verification with QC Rules, lab results also need to be reported timely.
30 Accuracy Closeness of the measured result to the true value Your control values may show precision but be inaccurateInstrument can repeatedly get the same value, but it’s not the right valueThis diagram is imprecise, but accurate
31 PrecisionPrecision is: The reproducibility or closeness of results to each otherThe degree of fluctuation on repeated measurements is indicative of the “precision” of the assay.The closeness of measurements to the true value is indicative of the “accuracy” of the assay.This is inaccurate, but is precise
32 Interactive Opportunity How can we check for accuracy and precision?Can you have precision without accuracy?Can you have accuracy without precision?STRESS to participants that Quality Control:Should detect random errors (Imprecision)Should detect systematic errors (Inaccuracy)Shift (6 results that settle around new mean)Trend (7 results upward or downward )Quality Control: Samples, Charts and Rules
33 Possible Causes for Random Errors Can be due to variations in line voltage, pipetting, dispensers, contamination, volume dispensed, bubbles in lines of reagents, etc*…A random error can be either positive or negativeThe direction and exact magnitude of the random error cannot be exactly predictedRandom Errors affect reproducibility or precision of a test system*NOTE: These variations should be detected during morning instrument start-up and maintenance*NOTE: Pipettes with poor accuracy can cause systematic errors also.*NOTE: When you see a random error on QC results, the technologist should look for the possible causes listed in the slide to begin the troubleshooting before re-running controls.
34 Random Errors and Possible Causes 1 3s Rule Violation:Usually indicates a Random ErrorWhen reviewing statistical data remember when using +/- 3 SD, there is a 0.3% chance of getting a control value outside the +/- 3 SD rangeBefore running patient specimens you must troubleshoot AND then rerun the control and the values must be within +/- 2SDCheck that controls are in date and well mixed**NOTE: Remind participants to include detailed steps to correctly reconstitute control material in their policy statement and procedure for Section 5 (Equipment) in their Quality ManualQuality Control: Samples, Charts and Rules
35 Westgard Rules Random error detected by 1 3s This means that 1 QC value exceeds -3sCould be below -3s or above + 3s13s refers to a control rule that is commonly used with a Levey-Jennings chart when the control limits are set as the mean plus 3s and the mean minus 3s. A run is rejected when a single control measurement exceeds the mean plus 3s or the mean minus 3s control limit and it indicates a random error. This indicates the need to stop and look for the cause of the random error, solving the problem before reporting the patient results.How can you determine it is random error on the first violation? May be a sudden shift or trend?Quality Control: Samples, Charts and Rules
36 Possible Causes for Random Errors 2 2s Rule Violation:Usually indicates a Systematic ErrorRemember, the first time single control value is > or < 2 SD it serves as a warning and you can run and report out your patient valuesBUT, the second sequential time the control value is > or < 2 SD, you must trouble shoot and find the reason for the error BEFORE you can run patient specimens and report out patient valuesCheck that controls are in date and well mixed*NOTE: 12s is only a warning while 22s rule violation within 20 days signals the run cannot be accepted and the test must undergo troubleshooting to detect the error BEFORE patient specimens may be analyzed.Quality Control: Samples, Charts and Rules
37 Westgard Rules Systematic error detected by 2 2s Could be below -2s or above + 2s
38 Systematic Errors Systematic Errors include bias/shifts, & trends These errors affect accuracy of the test system.Usually 6x and 7t Rule ViolationsCan be due to calibration lot changes, temperature changes in incubator unit, light source deterioration, electronics, reagent lot changes, etc.Systematic errors are always in one direction on a L J chart.
39 Bias/Shift: 6x Rule Violation A systematic difference or systematic error between an observed value and some measure of the truth.6x Rule violations:Usually indicate an error with your test systemOn day 7, after 6 days of one or more control values above or below the mean*, you must trouble shoot and find/correct error before running patient specimens*Note the values of the 6 days are within +/- 2sdFYI:Generally used to describe the inaccuracy of a method relative to a comparative method in a method comparison experiment.It also has a specific meaning in the statistical t-test, where bias equals the difference between the mean values of the two methods being compared or the average of all the differences between the paired sample values.
40 Westgard Rules Systematic Error detected with 6x: shift 6 results are on the same side of the meanIf a laboratory uses this rule, on the 6th run in which the result settles around a new mean such as shown here with 6 results below mean and above -2s, a shift has occurred. This indicates the need to stop and look for the cause of the systematic error, solving the problem before reporting the patient results. //// OR, does troubleshooting this have any medical relevance for this particular test? Discuss Platelet counts vs K+ levelsQuality Control: Samples, Charts and Rules
41 Trend: 7t Rule Violation Systematic error when a series of control values are treated as a time series7t Rule violations:Usually indicate an error with your systemOn day 8, after 7 days of one or more control values moving either upward or downward, you must trouble shoot and find/correct error before running patient specimens
42 Westgard Rules Systematic error detected with 7t: trend 7 results falling in a pattern, up or downOn the 7th run when an upward or downward trend is discovered even within the 95% limits of mean +/- 2 standard deviations, a systematic error has occurred. This indicates the need to stop and look for the cause of the systematic error, solving the problem before reporting the patient results.Quality Control: Samples, Charts and Rules
44 Westgard Rules: Single or Multi Rule QC Single rule QC Procedure uses a single criterion or single set of control limits, such as a Levey-Jennings chart with control limits set as either the mean plus or minus 2 standard deviations (2s) or the mean plus or minus 3s. There is more than 1 single rule but each rule is assessed individually.Multi-rule QCUses a combination of decision criteria, or control rules, to decide if an analytical run is in-control or out-of-controlUses 5 different control rules to judge acceptability of an analytical runNow for a more technical description. Multirule QC uses a combination of decision criteria, or control rules, to decide whether an analytical run is in-control or out-of-control. The well-known Westgard multirule QC procedure uses 5 different control rules to judge the acceptability of an analytical run. By comparison, a single-rule QC procedure uses a single criterion or single set of control limits, such as a Levey-Jennings chart with control limits set as either the mean plus or minus 2 standard deviations (2s) or the mean plus or minus 3s. "Westgard rules" are generally used with 2 or 4 control measurements per run, which means they are appropriate when two different control materials are measured 1 or 2 times per material, which is the case in many chemistry applications. Some alternative control rules are more suitable when three control materials are analyzed, which is common for applications in hematology, coagulation, and immunoassays.
45 Westgard RulesMany laboratories follow the Westgard Single-Rule policy.The rules are assessed individually to check for random and systematic errors.
46 Westgard Multirule System These rules work best together in a multi-rule system with automated analyzersRules evaluated in a certain orderStarts with the warning rule (12s ) that triggers the need to look for other violations:The first time a result exceeds mean + 2s or mean -2s but is within 3 SD it is a warning to check back for shifts or trends or other rule violations. It should indicate the need to verify this occurs only rarely (5% of the time or no more than once in 20 days) and that a 6 x or 7t aren’t also present. A multirule system means that many rules are used to verify patient results are correct, ruling out both random errors, such as 1 3s and systematic errors such as 2 2s, 6x, 7t or even 4 1s rule violations.Quality Control: Samples, Charts and Rules
47 Westgard MultirulesIf 1 control exceeds 2 s (1 2s rule is violated) this decision tree says this is a warning to look for other rule violations such as the 2nd control exceeding 2 s (2 2s) or R4s to decide whether to accept or reject the run. Rejection means that you should turn to the procedure or instrument operating manual to solve the problem before reporting patient results.This decision tree helps to decide whether to reject or accept a run. If you answer yes, it means that rule has been violated.Quality Control: Samples, Charts and Rules
48 Key Points Describe a control in terms of Assayed (mean and s provided)Unassayed (mean and s not provided)Use (validation of run not calibration)Calculate 95% limits for quality control samplesDetect errors in quality control using 95% and Westgard rulesQuality Control: Samples, Charts and Rules
49 Key PointsLevey-Jennings chart include values and markings on the x (days or time) and y axis (concentration mean and s)Quality Control: Samples, Charts and Rules
50 Key Points Westgard rules increase ability to detect systemic errors. Quality Control: Samples, Charts and Rules
51 Key Points Westgard Rules increase ability to detect random errors. Quality Control: Samples, Charts and Rules
52 Key Points Westgard Rule, 1 2s is provided as a Warning Quality Control: Samples, Charts and Rules
53 ReferencesShewhart WA. Economic Control of Quality of Manufactured Product. New York; D. Van Hostrand Company, Inc., 1931.Levey S, Jennings ER. The use of control charts in the clinical laboratory. Am J Clin Pathol 1950;20:Henry RJ, Segalove M. The running of standards in clinical chemistry and the use of the control chart. J Clin Pathol 1952;27:Westgard JO, Groth T, Aronsson T, Falk H, deVerdier C-H. Performance characteristics of rules for internal quality control: probabilities for false rejection and error detection. Clin Chem 1977;23:Quality Control: Samples, Charts and Rules
54 ReferencesWestgard JO, Barry PL, Hunt MR, Groth T. A multi-rule Shewhart chart for quality control in clinical chemistry. Clin Chem 1981;27:Westgard JO, Barry PL. Cost-Effective Quality Control: Managing the Quality and Productivity of Analytical Processes. Washington, DC:AACC PressQuality Control: Samples, Charts and Rules