Claude Beigel, PhD. Exposure Assessment Senior Scientist Research Triangle Park, USA Practical session metabolites Part II: goodness of fit and decision making

1 Recommended Tools for Assessing Goodness of Fit Same recommended approach as for parent substance Combination of visual assessment and statistical tests Visual assessment, although bringing some level of subjectivity, is necessary to discern between normal data variability (scattering) and systematic model deviation, this is not done by the statistical tests

2 Recommended Tools for Assessing Goodness of Fit Visual Assessment Visual check of model description of measured data and distribution of residuals (plot of residuals, Predicted - Observed) Systematic deviation indicates kinetic model may not be appropriate (unless deviation can be attributed to experimental artifacts) Example 8.2 of report, SFO-SFO fit

3 Recommended Tools for Assessing Goodness of Fit Visual Assessment Residuals should be randomly distributed on vertical axis Example 8.2 of report, SFO-FOMC fit

4 Recommended Tools for Assessing Goodness of Fit Statistical Indices Chi 2 ( 2 ) statistical test Minimum error percentage to pass 2 test at a 5% significance level –Needs to be performed for each substance individually, to avoid that good fit of the main substances (parent and/or major metabolite) overshadows goodness of fit of more minor substances (weighting issue) –Calculated from fitted versus observed substance data (use of average values for replicates is recommended) –Degrees of freedom for the substance defined as number of substance data points used in 2 test minus number of estimated parameters for the substance Do not count replicates if averages used, excludes data points set to 0 (metabolite at time-0) or not counted (<LOD/LOQ) Metabolite parameters defined as metabolite formation fraction and degradation rate parameters (dependent of kinetic model used)

5 Recommended Tools for Assessing Goodness of Fit Statistical Indices One-sided t-test for evaluating uncertainty of rate constant parameters To determine whether rate is significantly different from 0 –If p < 0.05, parameter is considered significantly different than zero –If p between 0.05 and 0.1, weight of evidence should be considered Especially important for metabolites that do not show a clear decline Because parameters in parent + metabolite fits (formation and degradation parameters) can be highly correlated, the t-test is performed at final step (all parameters fitted together) –Degrees of freedom defined as number of data points (including replicates) minus number of fitted parameters

6 Recommended Tools for Assessing Goodness of Fit Data Handling / Methodology Basic data handling Paste ModelMaker output (integration table) in Excel spreadsheet –Extract fitted values corresponding to measured times for each substance –Average replicates if necessary (an automated Excel spreadsheet may be created for that purpose, but not available yet) Minimum 2 error % for metabolites may be calculated using Parent degradation kinetics.xls file Paste measured Vs. fitted values in Chi2 all models worksheet, update number of parameters cell and click calculate

7 Recommended Tools for Assessing Goodness of Fit Data Handling / Methodology Residuals may be plotted in Parent degradation kinetics.xls file Valid only for 1- or 2-replicate data sets (if more, needs to be done manually) Paste measured Vs. fitted values (all replicates) in SFO no-reps or SFO 2-reps worksheet t-test for rate constant parameters may be performed using provided t-test.xls file For each rate constant parameter, enter parameter estimate and standard error, number of data points and number of parameters estimated

8 Hands-on Example 1 Exercise 1 Open ModelMaker file for example 1 From result table, extract fitted value for each sampling time, write down in output tables for parent, metabolite1 and metabolite2 Enter values in Metabolitesexamplesoutput.xls, averages are calculated automatically

9 Hands-on Example 1 Visual assessment Check ModelMaker plot of fit and answer following questions for each substance –Does fitted line adequately describe data, are there obvious over- or under- predictions (including day-0) Plot residuals for each substance and answer following questions –Do residuals show distinct pattern, are most of the points above or below 0- line, what is the magnitude? Statistical indices Calculate minimum 2 error percentage for each substance Perform t-test for all rate constant parameters (parent and metabolites) and record P-value and conclusion

10 Hands-on Example 1 Visual Assessment GraphAssessment / Remarks Parent Overall fit Residuals Metabolite1 Overall fit Residuals Metabolite2 Overall fit Residuals

11 Hands-on Example 1 Statistical Indices 2 -test Relevant Parameters Estimated (y/n) Number of Parameters Minimum 2 Error Percentage Parent Pini kP Metabolite1 ffM1 kM1 Metabolite2 ffM2 kM2 t-test Estimated Value Standard Error Number of Data Points Number of Estimated Parameters P-valueConclusion kP kM1 kM2

12 Hands-on Example 2, Parent FOMC Exercise 2 Open ModelMaker file for example 2, parent FOMC From result table, extract fitted value for each sampling time, write down in output tables for parent and metabolite Enter values in Metabolitesexamplesoutput.xls, averages are calculated automatically Perform visual assessment and calculate statistical indices

13 Hands-on Example 2, parent FOMC Visual Assessment GraphAssessment / Remarks Parent Overall fit Residuals Metabolite Overall fit Residuals

14 Hands-on Example 2, parent FOMC Statistical Indices 2 -test Relevant Parameters Estimated (y/n) Number of Parameters Minimum 2 Error Percentage Parent Pini P Metabolite ffM kM t-test Estimated Value Standard Error Number of Data Points Number of Estimated Parameters P-valueConclusion kM

15 Hands-on Example 2, parent DFOP Exercise 3 Open ModelMaker file for example 2, parent DFOP From result table, extract fitted value for each sampling time, write down in output tables for parent and metabolite Enter values in Metabolitesexamplesoutput.xls, averages are calculated automatically Perform visual assessment and calculate statistical indices

16 Hands-on Example 2, parent DFOP Visual Assessment GraphAssessment / Remarks Parent Overall fit Residuals Metabolite Overall fit Residuals

17 Hands-on Example 2, parent DFOP Statistical Indices 2 -test Relevant Parameters Estimated (y/n) Number of Parameters Minimum 2 Error Percentage Parent Pini g k1 k2 Metabolite ffM kM t-test Estimated Value Standard Error Number of Data Points Number of Estimated Parameters P-valueConclusion k1 k2 kM

18 Hands-on Example 2, Metabolite Decline Exercise 4 Open ModelMaker file for example 2, metabolite decline From result table, extract fitted value for each sampling time, write down in output tables for parent and metabolite Enter values in Metabolitesexamplesoutput.xls, averages are calculated automatically Perform visual assessment and calculate statistical indices

19 Hands-on Example 2, Metabolite Decline Visual & Statistical Assessment GraphAssessment / Remarks Metabolite Decline Overall fit Residuals 2 -test Relevant Parameters Estimated (y/n) Number of Parameters Minimum 2 Error Percentage Metabolite Mmax kM t-test Estimated Value Standard Error Number of Data Points Number of Estimated Parameters P-valueConclusion kM

20 Decision Making Trigger Endpoints Based on visual assessment and statistical indices, is SFO model acceptable for the metabolite (in combination with best-fit model for parent)? Yes use SFO DT50/90 endpoints No and clear decline of metabolite, use FOMC model for metabolite (in combination with best-fit model for parent) –If FOMC acceptable based on visual assessment and statistical indices, use FOMC DT50/90 endpoints –If not, model decline of metabolite with best-fit model and use decline DT50/90 as conservative endpoints No and no apparent decline of metabolite –Assess relevance of study with regard to metabolite –Check other studies –Study with metabolite may be needed

21 Decision Making Modeling Endpoints Based on visual assessment and statistical indices, is SFO model acceptable for the metabolite (in combination with appropriate model for parent)? Yes use modeling endpoints for metabolite No and clear decline –If formation fraction estimate is reliable, use with decline rate constant as conservative endpoints –If not, use formation fraction of 1 with decline rate constant as conservative endpoints –If metabolite biphasic, use appropriate higher-Tier approach (e.g. DFOP, PEARL) –If terminal metabolite and biphasic, use FOMC DT90/3.32 as half-life No and no apparent decline of metabolite –Assess relevance of study with regard to metabolite –Check other studies –Study with metabolite may be needed

22 Hands-on Examples Determine appropriate trigger and modeling endpoints for Example 1 metabolites 1 and 2 and Example 2 metabolite