Presentation is loading. Please wait.

Presentation is loading. Please wait.

Method comparison Chemometrics Department of Chemical Pathology, University of Pretoria, Dr R Delport 2003.

Similar presentations


Presentation on theme: "Method comparison Chemometrics Department of Chemical Pathology, University of Pretoria, Dr R Delport 2003."— Presentation transcript:

1 Method comparison Chemometrics Department of Chemical Pathology, University of Pretoria, Dr R Delport 2003

2 Performance characteristics that are taken into account: Precision, Accuracy, Interference, Working Range, and Detection Limit. http://www.westgard.com/lesson21.htm Chemometrics

3 To carry out a good method validation study, you need to do the following: Define a quality requirement for the test in the form of the amount of error that is allowable, preferably an allowable total error, Select appropriate experiments to reveal the expected types of analytical errors, Collect the necessary experimental data, Perform statistical calculations on the data to estimate the size of analytical errors, Compare the observed errors with the defined allowable error, and Judge the acceptability of the observed method performance. http://www.westgard.com/lesson21.htm Chemometrics

4 An experimental plan can be formulated by: Recognizing the types of errors that need to be assessed for this test and method, Identifying the appropriate experiments and the amount of data needed to estimate those types of errors, then Organizing these experiments to perform the quick and easy ones first and the ones taking more time and effort last. http://www.westgard.com/lesson21.htm Chemometrics

5 Types of errors Imprecision or random errors, Inaccuracy, bias, or systematic errors, which can be of two types Constant systematic error or Proportional systematic error. http://www.westgard.com/lesson21.htm Chemometrics

6 The dashed line in the middle of the figure represents ideal method performance where the test method and the comparative method give exactly the same results. Note that these results will also be subject to the random error of the method, therefore the actual data points would scatter about the line as illustrated in the figure. The range of this scatter above and below the line provides some idea of the amount of random error that is present. The bottom line in the figure shows the effect of a proportional systematic error, where the magnitude of the error increases as the test result gets higher. The top line shows the effect of a constant systematic error, where the whole line is shifted up and all results are high by the same amount. http://www.westgard.com/lesson21.htm Chemometrics

7 Experiments for estimating analytical errors http://www.westgard.com/lesson21.htm Chemometrics

8 Description of different experiments. Replication experiment Provides information about random error Is performed by making measurements on a series of aliquots of the same test samples within a specified period of time, usually within an analytical run, within a day, or over a period of a month. Preliminary experiment involves determining within-run imprecision. Final experiment requires at least 20 working days to provide estimate of the total imprecision, which includes within and between run components. http://www.westgard.com/lesson21.htm Chemometrics Experiments for estimating analytical errors

9 Interference experiment Provides information about the constant systematic error caused by the lack of specificity of the method. One test sample is prepared by adding the suspected material to a sample containing the analyte. A second aliquot of the original sample is diluted by the same amount with solvent, then both samples are analyzed by the test method and the difference determined. http://www.westgard.com/lesson21.htm Chemometrics Description of different experiments. Experiments for estimating analytical errors

10 Recovery experiment Provides information about the proportional systematic error caused by a competitive reaction. Test sample is prepared by adding a standard solution of the analyte being tested to an aliquot of a patient specimen. A baseline sample is prepared by adding an equal amount of the solvent used for the standard solution to a second aliquot of the same patient specimen. The two samples are then analyzed by the test method and the amount recovered is compared to the amount added. http://www.westgard.com/lesson21.htm Chemometrics Description of different experiments. Experiments for estimating analytical errors

11 Comparison of methods experiment Is primarily used to estimate the average systematic error observed with real patient samples. Can also reveal the constant or proportional nature of that error. A series of patient specimens are collected and analyzed by both the test method and a comparative analytical method. The results are compared to determine the differences between the methods, which are the analytical errors between the methods. http://www.westgard.com/lesson21.htm Chemometrics Description of different experiments. Experiments for estimating analytical errors

12 Get the method working and establish an operating protocol. Set up the instrument, prepare the reagents, calibrate the methods, and obtain results from test samples. Check the standards and be sure the method is properly calibrated, otherwise calibration errors will show up throughout the experimental studies. http://www.westgard.com/lesson21.htm Chemometrics Walking tour of the plan. Experiments for estimating analytical errors

13 Determine the working range. The working range will vary from test to test and must be defined as part of the specifications for the method. Check by analyzing a series of solutions, in duplicate or triplicate, covering the concentrations range of interest. If detection limit is a critical characteristic, it may be assessed at this time or in the next phase of preliminary experiments. http://www.westgard.com/lesson21.htm Chemometrics Walking tour of the plan. Experiments for estimating analytical errors

14 Determine within-run imprecision, recovery, and interference. The replication experiment might include 20 samples of two or three materials whose concentrations closely match the medical decision levels of interest for the tests. Interference experiments should test common problems such as hemolysis, lipemia, and high bilirubin. Recovery experiments assess whether there are any competitive reactions due to the matrix or other materials in the native specimens. http://www.westgard.com/lesson21.htm Chemometrics Walking tour of the plan. Experiments for estimating analytical errors

15 Comparison of methods experiment. Fresh patient specimens, and stored specimens. Minimum of 40 well-chosen patient samples should be tested over a minimum of 5 working days. Distributed one-third in the low to low-normal range, one- third in the normal range, and one-third in the high abnormal range. Method acceptability should be judged on the basis of the sizes of the random, systematic, and total analytical errors. http://www.westgard.com/lesson21.htm Chemometrics Walking tour of the plan. Experiments for estimating analytical errors

16 Comparison of methods experiment. Used as initial graph if the two methods are expected to show one-to-one agreement, Displays the difference between the test minus comparative results on the y-axis versus the comparative result on the x-axis, such as shown in the accompanying figure. Chemometrics Walking tour of the plan. Experiments for estimating analytical errors http://www.westgard.com/lesson21.htm Difference plot

17 Comparison of methods experiment. Used as initial graph if the two methods are expected to show one-to-one agreement, Displays the difference between the test minus comparative results on the y-axis versus the comparative result on the x-axis, such as shown in the accompanying figure. Differences should scatter around the line of zero differences, half being above and half being below. Repeat measurements if indicated. Chemometrics Walking tour of the plan. Experiments for estimating analytical errors http://www.westgard.com/lesson21.htm Difference plot

18 Comparison of methods experiment. The correlation coefficient is a statistic that is almost always calculated and reported to describe the results from a comparison of methods study. The new or "test" method values are plotted on y-axis and comparison values on x-axis. Chemometrics Walking tour of the plan. Experiments for estimating analytical errors http://www.westgard.com/lesson21.htm Correlation coefficient

19 Chemometrics http://www.westgard.com/lesson15.htm A value of 1.000 indicates perfect correlation between the results of two methods. Other statistics (such as slope, intercept, and standard deviation of the residuals) can also be calculated from the same data to estimate the size of errors occurring between the methods. Comparison of methods experiment. Walking tour of the plan. Experiments for estimating analytical errors

20 Random error, RE, or imprecision Can be either positive or negative. Direction and exact magnitude cannot be predicted. Imprecision is quantitated by calculating the standard deviation (SD) from the results of a set of replicate measurements. Chemometrics http://www.westgard.com/lesson15.htm Analytical errors Random error

21 Random error, RE, or imprecision Can be either positive or negative. Direction and exact magnitude cannot be predicted. Imprecision is quantitated by calculating the standard deviation (SD) from the results of a set of replicate measurements. As the SD often increases as the concentration increases, the coefficient of variation (CV) is calculated to express the SD as a percentage of the mean concentration from the replication study. Maximum size of a random error is commonly expressed as a 2 SD or 3 SD estimate to help understand the potential size of the error that might occur. Chemometrics http://www.westgard.com/lesson15.htm Analytical errors Random error

22 Systematic error, SE, or inaccuracy Is always in one direction. A systematic shift displaces the mean of the distribution from its original value. In contrast to random errors that may be either negative or positive and whose direction can not be predicted, systematic errors are in one direction and cause all the test results to be either high or low. Chemometrics http://www.westgard.com/lesson15.htm Analytical errors Systematic error

23 Systematic error, SE, or inaccuracy How high or how low can be described by the bias, which is calculated as the average difference, or the difference between averages, between the value by the "test" method and a "comparative" method in a comparison of methods experiment. Alternatively, the expected systematic difference may be predicted from the equation of the line that best fits the graphical display of test method values on the y-axis vs comparative method values on the x-axis. Chemometrics http://www.westgard.com/lesson15.htm Analytical errors Systematic error

24 Systematic error, SE, or inaccuracy How high or how low can be described by the bias, which is calculated as the average difference, or the difference between averages, between the value by the "test" method and a "comparative" method in a comparison of methods experiment. Alternatively, the expected systematic difference may be predicted from the equation of the line that best fits the graphical display of test method values on the y-axis vs comparative method values on the x-axis. SE may stay the same over a range of concentrations, in which case it can also be called constant error, or it may change as concentration changes, in which case it can be called proportional error. Chemometrics http://www.westgard.com/lesson15.htm Analytical errors Systematic error

25 Total Error, TE, It is the net or combined effect of random and systematic errors, as shown in the accompanying figure. Chemometrics http://www.westgard.com/lesson15.htm Analytical errors Total error

26 Total Error, TE, It is the net or combined effect of random and systematic errors, as shown in the accompanying figure. It represents a "worst-case" situation, or just how far wrong a test result might be due to both random and systematic errors. Because laboratories typically only make a single measurement for each test, that measurement can be in error by the expected SE, or bias, plus 2 or 3 SD, depending on how you quantitate the effect of RE. Chemometrics http://www.westgard.com/lesson15.htm Analytical errors Total error

27 Chemometrics

28

29


Download ppt "Method comparison Chemometrics Department of Chemical Pathology, University of Pretoria, Dr R Delport 2003."

Similar presentations


Ads by Google