1. Federaal Agentschap voor de Veiligheid van de Voedselketen 1 Validation of a Quantitative Analytical Procedure – Accuracy (total error) profile Dr. Jacques O. DE BEER Workshop IPH 27th April 2007 Scientific Institute of Public Health - Brussels (Belgium)

2. 2 Method Validation – General Concepts Different regulations relating to GLP, GMP, GCP (OECD, EU) Normative or Regulatory documents (ISO 17025, ICH, EMEA, FDA, dir. 2002/657/EG)  both suggest that analytical procedures have to comply to certain acceptance criteria. This request imposes that these procedures are to be validated. - Some documents define the validation criteria - No proposals on experimental approaches !! - Limited to general concepts !!

3. 3 Introduction - Definition Method Validation: is the confirmation by examination and the provision of objective evidence that the particular requirements for a specific intended use are fulfilled. EN ISO/IEC 17025 § 5.4.5.1 Methods need to be validated or revalidated:  before introduction into routine application  whenever conditions change for which the method has been validated (e.g. Instrument with different characteristics)  whenever the method is modified and modifications are outside original scope of the method.

4. 4 European and International regulatory bodies and their guidelines on different aspects of QA BodyFull nameGuidance on EurachemFocus for Analytical Chemistry in Europe Method validation CITACCooperation of International Traceability in Analytical Chemistry Proficiency testing Quality Assurance EAEuropean Cooperation for Accreditation Accreditation CENEuropean Committee for Normalization Standardization IUPACInternational Union of Pure & Applied Chem. Method validation ISOInternational Standardization Organisation Standardisation AOAC ILAC Association of Official Analytical Chemists International Laboratory Accreditation Cooperat. Internal qual. Control Proficiency testing Accreditation FDAUS Food and Drug Administration Method validation USPUnited States Pharmacopoeia Method validation ICHInternational Conference on Harmonization Method validation

5. 5 Objectives of an analytical procedure Able to quantify as accurately as possible each unknown quantity to be determined. returned result x“true value µ T ”acceptance limit λ After analysis: the difference between returned result x and the unknown “true value µ T ” be small or < acceptance limit λ: - < x - µ T < λ   x - µ T  < λ (eq.1)  λ  λ : depends on objective of analytical procedure e.g. 1-2 % on bulk, 5 % on pharmaceuticals, 15 % for biological samples  previously defined

6. 6 Objectives of an analytical procedure Analytical procedures characterized by (cfr. def.):  M  “true bias”  M = systematic error (unknown) σ² M  “true precision” σ² M = random error measured by a standard deviation or variance (unknown) biasprecision Estimates of bias and precision obtained by experiments during the validation Reliability of these estimates depends on adequacy of experiments on known samples (Valid. Stds), experimental design, number of experiments These estimates  an intermediary but obligatory step to evaluate if procedure is likely or not to quantify with sufficient accuracy the unknown quantities; not objectives per se

7. 7 Bias = +7 % RSD = 3% Bias = +1 % RSD = 8% Bias = 0 % RSD = 20% Bias = +7 % RSD = 12% Procedure 1 Procedure 2 Procedure 3Procedure 4 Examples of procedures having the same acceptance limits = ± 15%

8. 8 Objectives of an analytical procedure Figure Figure: 4 different (hypothetical) methods giving the distribution of 95% of the measures Each method has a true bias  M, a true precision σ² M, a common acceptance limit λ (= 15%  bioanalytical procedure):  Procedure 3  Procedure 3: negligible bias (0%); unsatisfactory precision (20% CV); too many measures beyond +/- 15% of the true value; does not fulfill objective  Procedure 4  Procedure 4: bias (7%); precision (12%); important proportion outside acceptance limits; does not fulfill objective; but both < 15%: required by Washington Conf.  Procedures 1 and 2  Procedures 1 and 2: fulfill (valid): at least 95% of results inside acceptance limits

9. 9 Objectives of an analytical procedure Procedure 1 Procedure 1: presents a bias (+ 7%), but is very precise (3% CV) Procedure 2 Procedure 2: presents a negligible bias (+ 1%), but is less precise (8% CV) FIRST CONCLUSION: Differences between these two procedures don’t matter since results are never too far from true values of the sample to quantify. Quality of results is far more important than the intrinsic characteristic properties of procedure in terms of bias or precision.

10. 10 Objectives of an analytical procedure To develop a procedure without bias and error  considerable cost; not acceptable strategy Analyst has to take minimal risks, compatible with the analytical objectives (within reasonable time!!) acceptance limits (± ): Set up acceptable maximum proportion of measurements that might be outside acceptance limits (± ): 5%20 % risk  e.g. 5% or 20 % of measurements outside (± ) as maximum risk. “true bias”  M “true precision”  ² M  inside triangles (next fig.)  space of acceptable procedures characterized by “true bias”  M and a “true precision”  ² M proportion depends on objectives!!!  Acceptable procedures: 95, 80, 66% of measurements within ± 15% limits (recommendations Washington Conference)  proportion depends on objectives!!!

11. 11 True bias (%) True precision (%) -10-50105 0 5 15 20 66% 95% 80% Proc.3 Proc.4 Proc.1 Proc.2 (0,20) (7,12) (1,8) (7,3) % measurements within ± 15% bias-precision limits

12. 12 Objectives of an analytical procedure Interior triangle = Interior triangle = area of all analytical procedures of which 95% of result X should be included within acceptance limits (± ), set according constraints of analytical domain 2 other triangles 2 other triangles: proportions of 80% and 66% of measurements included within ± (accept. limits) true bias =0true precision = 15%  procedure with true bias =0 ; true precision = 15% : only 66% will fall within acceptance limits (± ) true bias =0true precision = 8%  procedure with true bias =0 ; true precision = 8% : 95% will fall within acceptance limits (± )

13. 13 Objectives of an analytical procedure procedures 1 and 2 Figure: procedures 1 and 2 located inside region of acceptance this region guarantees that at least resp. 95% and 80% of the results are within acceptance limits (± ) procedures 3 and 4 for the same risk of the measurements outside acceptance limits, procedures 3 and 4 not considered as valid procedures 3 and 4 for more important risk, procedures 3 and 4 could be valid.

14. 14 Objectives of an analytical procedure FURTHER CONCLUSION: Procedure qualified as acceptable if: (x)(µ T ) it “guarantees” that the difference between every sample measurement (x) and its “true value” (µ T ) is inside the predefined acceptance limits (±  P(  x - µ T  < )   eq. 2) In equation: P(  x - µ T  < )   eq. 2)   = proportion of measurements inside acceptance limits = acceptance limit, fixed a priori according objectives of the method risk of an analytical procedure Expected proportion of measurements falling outside the acceptance limits  risk of an analytical procedure

15. 15 Objective of the validation What ?  x - µ T  < acceptable limit to give to the laboratories as well to the regulatory bodies “guarantees” that every single measurement performed in routine is close enough to the unknown “true value” of the sample:  x - µ T  < acceptable limit Objective of validation: not simply to obtain estimates of bias and precision; it is to evaluate these guarantees and risks These estimates of bias and precision are required to evaluate risks

16. 16 Objective of the validation 2 basic notions With respect to this objective, 2 basic notions should be considered: “close enough” “close enough” (eq. 1) meaning that routine measure will be less than the acceptance limit λ from its unknown “true value” “guaranteed”, “guaranteed”, (eq. 2) meaning that it is very likely that analysis result will be close enough to the true unknown value.

17. 17 Objective of the validation decision tools are needed giving “guarantees” that future measurements are reasonably inside acceptance limits

18. 18 Decision rules Current position with respect to the decision rules used in the phase of validation  most of them based on use of the null hypothesis: H 0 : bias = 0  H 0 : relative bias = 0 %  H 0 : recovery = 100 % - Bias = x - µ T - Relative bias = 100 (x - µ T )/µ T - Recovery = 100 x/µ T average bias  A procedure wrongly declared adequate when the 95% C.I. of the average bias includes 0  Test inadequate in validation context of analytical procedures rejection criterion of Student t-test  Test inadequate in validation context of analytical procedures because decision based on computation of rejection criterion of Student t-test

19. 19 True bias (%) True precision (%) -10-5 0 105 0 5 15 20 Proc.3 Proc.4 Proc.1 Proc.2 Test based on H 0 = bias = 0 (7,3) (0,20) (1,8) -1515 (7,12) PROCEDURES VALID NOT VALID

20. 20 Decision rules According to the decision rule based on the null hypothesis H 0 in fig.: procedures 2, 3 and 4 are valid and procedure 1 is rejected outside triangle But: procedure 1 shows reduced bias (+ 7%) and a small RSD (3%)  outside triangle: rejected !! procedure 3 has high RSD (20%), procedure 4 has bias of 7% and RSD of 12%  accepted !! method accepted  bad precision  large C.I.  contains 0 as bias value  method accepted method rejected  good precision  small C.I.  may not contain 0 as bias value  method rejected null hypothesis H 0 inadequate in analyt. validation

21. 21 True bias (%) True precision (%) -10-5 0 105 0 5 15 20 Proc.3 Proc.4 Proc.1 Proc.2 Test based on acceptance limits (± 15%) β ≥ 80% -1515 (0,20) (7,12) (1,8) (7,3) PROCEDURES NOT VALID

22. 22 Decision rules According to the decision rule based on use of acceptance limits  triangle in fig. with acceptible valid procedures Triangle in fig. ≥a priori chosen proportion P(  x - µ T  < )   (eq. 2) Triangle in fig. corresponds to procedures with measurement proportion inside acceptance limits (±λ) ≥ a priori chosen proportion (e.g. 80%) as given by equation: P(  x - µ T  < )   (eq. 2) good precision  more sensible decision rule: procedures with good precision  accepted bad precision bad precision  rejected Biased procedure  small variance: acceptable !! Procedure with higher variance  needs small bias

23. 23 Decision rules: Accuracy profile easy and visual decision rule:accuracy profile easy and visual decision rule: use of the accuracy profile within the acceptance limits (± ) Accuracy profile Accuracy profile constructed from the β-expectation intervals on the expected measurements: - allows to decide on capability of analytical procedure to give results inside ± dosage interval fixed risk - describes dosage interval (range) in which the procedure is able to quantify with known accuracy and a fixed risk at the end of the validation risk of 5% included in acceptance limits e.g. risk of 5%  “guarantee” that 95/100 future measurements will be included in acceptance limits, fixed according requirements (1-2 % on bulk, 5 % on pharmaceut., 15 % in bioanalysis)

24. 24 Decision rules Accuracy profile acceptance limits Accuracy profile by concentration level (C1, C2,...) obtained by computing β-expectation tolerance interval  allows evaluating the proportion of expected measurements inside acceptance limits This interval This interval is obtained from available validated estimates of the bias and precision of the procedure (by concentration level) This interval This interval of measurements expected within level  (= proportion of measurements inside ) has  -expectation confidence limits

25. 25 Decision rules concentration level j If for each concentration level j β-expectation tolerance interval are included within acceptance limits  method accepted! Tolerance interval calculation: Tolerance interval calculation: - what matters is: the guarantee of the results, expected in the future by the same analytical procedure in routine - estimation of µ j,  ² B,j,  ² W,j at every conc. j are used to estimate the expected proportion of observations within the predifined acceptance limits [-  i.e.: E µ,  P[  x - µ T  < ]  M,  M  

26. 26 Calculation of β-expectation tolerance interval estimated bias (mean added concentrations minus mean calculated concentrations) j = conc. level these statistical parameters (trueness, within/between precision) might be calculated for each concentration level from validation standards.

27. 27 Calculation of the interval in which a proportion β of all samples with a certain real concentration is observed (method of Mee): β expectation tolerance interval: Calculation of β-expectation tolerance interval ISO 5725-2: calculation of within and between variance

28. 28 = degrees of freedom (Satterthwaite): p = number of series (days) n = number of replicates per series Calculation of β-expectation tolerance interval Q t = β quantile of the Student’s t-distribution with ν degrees of freedom

29. 29 interval representing in % the region containing β % of analysis results for a certain concentration level j : Calculation of β-expectation tolerance interval after rearrangement:

30. 30 Interval consists of two terms: % bias +/- coefficient of variation for intermediate precision = expression of method accuracy method is accurate for this concentration level if obtained tolerance interval is included within acceptance limits [-λ,λ] Calculation of β-expectation tolerance interval

31. 31 Accuracy profile bias (%) concentration +  C1C2C3C4 LLQULQ RANGE mean relat. bias acceptance limits dosage interval 0 bias limits of confidence

32. 32 Decision rules Estimates of bias and variance are essential to compute evaluation of the expected proportion of measurements within acceptance limits Accuracy profile obtained by connecting the lower or upper limits of confidence (cfr. fig) If a subsection (concentration range) falls outside the acceptance limits  new limits of quantification be defined and a new dosage interval (Upper and Lower Limits of Quantification)

33. 33 Decision rules (conclusion) Accuracy profile Accuracy profile represents limits “ULQ and LLQ” in agreement with definition of criterion: LLQ LLQ = smallest quantity of the substance that can be measured with defined accuracy Accuracy profile Accuracy profile as single decision tool:  Allows reconciling the objectives of the procedure and those of the validation  Allows to visually grasp the capacity of the procedure to fulfill its analytical objective

34. 34 Validation Protocols: Life Cycle Validation has to be considered as an element intervening after the development of a new analytical procedure Objective of procedure = to be used in routine 2 objectives Usage in routine must be coupled with a quality control (QC) of which the 2 objectives are: unknown samples  the validity of the found results on the unknown samples continuity of the performances  the assessment of the continuity of the performances of the procedure at the time of its exploitation

35. 35 Protocols in validation phase Main objectives in validation phase: - demonstrate specificity/selectivity - validate the response function (or calibration model used in routine) - estimate precision (repeatability and intermediate precision), trueness, accuracy - validate the quantitation limits, validate the range (dosage interval); cfr. accuracy profile! - assess linearity of the analytical procedure (results directly proportional to concentration in the sample – cfr. definitions)

36. 36 Protocols in validation phase  calibration standards concentration levelsrepetitions by level  preparation of calibration standards (CS) with fixed number of concentration levels and repetitions by level  validation standards independent samples  preparation of the validation standards (VS) in the matrix; are independent samples  between-series variance. VS prepared and treated independenly as future samples  essential for good estimation of between-series variance. intermediate precision different daysequipmentdifferent operators To estimate intermediate precision, VS analyzed on different days, equipment and by different operators.  Validation phase is ultimate stage before exploitation;  allows to estimate procedure’s performances in the expected experimental conditions   allows to check procedure’s capability to quantify unknown sample

37. 37 Protocols in validation phase matrix effect. Question whether or not presence of a matrix effect. no matrix effect validation protocolsV1 V2 If no matrix effect, question is which concentration levels will be used for calibration  apply described validation protocols (V1 and V2) matrix effectprotocol V5 Evidence of matrix effect: apply protocol V5 doubtprotocols V3 and V4 In case of doubt: apply protocols V3 and V4 according to calibration levels (cfr.Table) types of standards (CS and VS), concentration levels? Which types of standards (CS and VS), concentration levels? VS similate future samples VS prepared in matrix and independent; must similate future samples

38. 38 Choise of number of CS and VS depending on selected protocol standards conc. levels Protocol Protocol (no matrix, doubt, matrix) V1V2V3V4V5 CS. calibration out matrix Low Mid High 2 2(*) 2 2(-) 2 2(*) 2 2(-) 2 CS. calibration in matrix Low Mid High Addit. 2 2(*) 2 2(-) 2 2(-) 2 2(+) VS. validation in matrix Low Mid High 333333 333333 333333 333333 333333 Minimum number of series 33333 Total number of experiments3345(39)3963(51)45(39)

39. 39 Description of protocol V1 Calibration standards Validation standards (1) Additional validation standards (linearity ICH) Series 1Series 2Series 3 Conc R

40. 40 Description of protocol V2 Calibration standards Validation standards (1) Additional validation standards (linearity ICH) Series 1Series 2Series 3 ()) ) ( ( R Conc

41. 41 Description of protocol V3 Calibration Standards without matrix Validation standards (1) Additional validation standards (linearity ICH) Series 1Series 2Series 3 R Conc Calibration Standards within matrix

42. 42 Possible concentation levels by type of procedure (e.g. 6 comparative procedures) single chemical substance 1. Determination of single chemical substance; reference available or determination of active ingredient in a pharmaceutical speciality (matrix) available synthesis impurity concentration levels > LOQ 2. Determination of available synthesis impurity in an active substance or pharmaceutical speciality (matrix) at concentration levels > LOQ available synthesis impurity around impurity limit 3. Determination of available synthesis impurity in an active substance or pharmaceutical speciality (matrix) around impurity limit (impurity limit > LOQ) chemical substance and one of its non-available impurities 4. Simultaneous determination of chemical substance and one of its non-available impurities in this substance or pharmaceutical speciality (use substance as tracer to allowed maximum concentration of impurity)

43. 43 Possible concentation levels by type of procedure (e.g. 6 comparative procedures) active substance for measuring dissolution kinetics 5. Determination of active substance for measuring dissolution kinetics for a dry dosage form (matrix) active ingredient and its metabolites in plasma 6. Determination of active ingredient and its metabolites in plasma (drugs), drug residues,... WHICH CONCENTRATION LEVELS ?  cfr. TABLE

44. 44 Examples of possible concentration levels by type of procedure Procedure 123456 Calibration standards Low Mid High addition 100% (120%) LOQ (½ C max ) (C max ) 80% LA 100% LA (120% LA) LOQ/LA (50%) 120% C min (50%) 120% LOQ ½ C max C max X Validation standards Low Mid High 80% 100% 120% LOQ ½ C max C max 80% LA 100% LA 120% LA LOQ/LA 50% 120% C min (50%) 120% LOQ ½ C max C max LA = admitted limit; C max = max. conc.; C min = min. conc.

45. 45 Protocols in validation phase relationship between response Y and concentration Xcalibration standards response function Identify relationship between response Y and concentration X using calibration standards (response function). Regression modelsaccuracy profiles Regression models are fitted, accuracy profiles calculated, one model selected  decision about validity of the procedure of interest. procedure type fixed method objectives Model: depends on  procedure type (pharmaceutical, bio-analytical, immuno-assay)  fixed method objectives Linear regression (origin or not) envisaged. Mathematical transformations applied on X and Y Quadratic regression may be useful

46. 46 Protocols in validation phase VS concentrations by  calibration curve equations Back-calculation of estimated VS concentrations by series by  calibration curve equations estimation of trueness and precision For each concentration level  estimation of trueness and precision limitsforaccuracy  calculation of limits for accuracy : cfr. CI j (bias) (include large proportion of results) accuracy profile for each fitted model  accuracy profile for each fitted model visual decision tool Accuracy profile  visual decision tool to evaluate capability of the method  if not within pre-fixed acceptance limits: - restrict dosis range  new limits of quantification - extend acceptance limits (possible??)

47. 47 A C E B D F Bias (%) 15 -15 123450012345 Concentration ACCURACY PROFILES with same VALIDATION PROTOCOL (0.01 – 5.0 ng/ml) quadratic regression weighed linear regression linear regression linear regression throug 0 linear regression on square root transformed data linear regression on log transformed data

48. 48 Protocols in validation phase Figure: Accuracy profiles Figure: Accuracy profiles for validation of dosing procedure of chemical substance in biological matrix. Protocol V5 Protocol V5 applied + some concentration levels Essentially low levels  good estimation of LOQ response functions 2 of 6 response functions (A: quadratic regress. + B: weighed regression) answer objective: acceptance limits ± 15%  accuracy profile allows to decide about method capability: Quantifiable dosing range with known accuracy: 0.01 – 5.0 ng/ml at risk  = 5%

49. 49 CONCLUSIONS harmonized approach Lack of generalisation between different validation protocols  harmonized approach review objectives of the validation objectives of the analytical procedure Proposal to review objectives of the validation according to objectives of the analytical procedure diagnosis rulesdecision rules Distinction between diagnosis rules and decision rules not simply to obtain estimates of bias and precision Objectives of validation not simply to obtain estimates of bias and precision but also: risks or confidences close enough to unknown true value To evaluate risks or confidences that any single measurement is close enough to unknown true value Trueness, precision, linearity,..., no longer sufficient to make these guarantees.

50. 50 CONCLUSIONS Adapted decision tool  accuracy profile of the analytical procedure, based on: –  -expectation tolerance interval at each concentration level –concept of total error (bias + standard deviation) Allows to bring together objectives of the procedure and those of validation Allows to visually grasp the capacity of the procedure  to fulfil its objectives  to control risk associated with its use in routine

51. 51References C. Hartmann et al., An analysis of the Washington Conference Report on bioanalytical method validation J. Pharm. Biomed. Anal., 12(11) (1994) 1337-1343 Ph. Hubert et al., The SFSTP guide on the validation of chromatographic methods for drug bioanalysis: from the Washington Conference to the laboratory. Anal. Chim. Acta, 391 (1999) 135-148 P. Chiap et al., Validation of an automated method for the liquid chromatographic determination of atenolol in plasma: application of a new validation protocol. Anal. Chim. Acta, 391 (1999) 227-238

52. 52 References B. Boulanger et al., An analysis of the SFSTP guide on validation of chromatographic bioanalytical methods: progress and limitations. J. Pharm. Biomed. Anal., 32 (2003) 753-765 Ph. Hubert et al., Validation des procédures analytiques quantitatives. Harmonisation des démarches. STP Pharma Pratiques, 13(3) (2003) 101-138 Ph. Hubert et al., Harmonization of strategies for the validation of quantitative analytical procedures. A SFSTP proposal – part I J. Pharm. Biomed. Anal., 36 (2004) 579-586

53. 53 Ph. Hubert et al., Validation des procédures analytiques quantitatives. Harmonisation des démarches. Partie II - Statstiques STP Pharma Pratiques, 16(1) (2006) 28 – 58 Ph. Hubert et al., Validation des procédures analytiques quantitatives. Harmonisation des démarches. Partie III – Exemples d’application STP Pharma Pratiques, 16(2) (2006) 87 – 121 M. Feinberg et al., New advances in method validation and measurement uncertainty aimed at improving the quality of chemical data Anal. Bioanal. Chem 380 (2004) 502-514 M. Feinberg et al., A global approach to method validation and measurement uncertainty Accred. Qual. Assur 11 (2006) 3-9 References