1. Evaluating the Efficacy and Equivalence of Phytosanitary Risk Reduction Measures Mark Powell U.S. Department of Agriculture, Office of Risk Assessment and Cost-Benefit Analysis Washington, DC, USA International Plant Health Risk Analysis Workshop October 24-28, 2005 Niagara Falls, Ontario, Canada

2. Powell - Efficacy and Equivalence2 Motivation Efficacy - ISPM 2 (Guidelines for Pest Risk Analysis) Article 3.2 (Efficacy and Impact of the Options) - Calls for consideration of several factors in evaluating phytosanitary risk management options, including their “biological effectiveness.” Equivalence - SPS Agreement (Article 4): “Members shall accept the sanitary or phytosanitary measures of other Members as equivalent, even if these measures differ from their own or from those used by other Members trading in the same product, if the exporting Member objectively demonstrates to the importing Member that its measures achieve the importing Member's appropriate level of sanitary or phytosanitary protection.”

3. Powell - Efficacy and Equivalence3 Outline Describe three analytical methods to recurrent challenges faced by plant health risk analysts in evaluating the efficacy and equivalence of phytosanitary risk reduction measures: 1. Define the desired level of treatment efficacy as a performance standard 2. Combine information from a series of replicated treatment trials 3. Account for uncertainty in control mortality

4. Powell - Efficacy and Equivalence4 Define the desired level of treatment efficacy as a performance standard Process standards: –Zero survivors out of ~30,000 treated –Zero survivors out of 90,000 + treated Performance standards: –95% confidence of 1 x 10 -4 survival (0.0001) –95% confidence of 3.2 x 10 -5 survival (Probit 9) Couey & Chew. 1986. Confidence Limits and Sample Size in Quarantine Research. J. Econ. Entomol. 79: 887-890.

5. Powell - Efficacy and Equivalence5 Binomial Distribution Random number of successes (s) of n independent trials: s ~ Binomial(n,p) where: f(x) = Binomial probability mass function F(x) = Binomial cumulative distribution function p = probability of survival (define “success” = survivor) x = number of survivors (s) n = sample size C = confidence level (more precisely, prob(s>x|n,p))

6. Powell - Efficacy and Equivalence6 Calculate Sample Size from Binomial Distribution Special case where s = 0, simplifies to: C = prob(x>0|n,p) =1-(1-p) n n=[log(1-C)/log(1-p)] Set p = 1 x 10 -4 and C = 0.95: solve for n = 29,956 (~30,000) Set p = 3.2 x 10 -5 and C = 0.95: solve for n = 93,615 (90,000 + )

7. Powell - Efficacy and Equivalence7 Poisson Approximation to Binomial where: f(x) = Poisson probability mass function F(x) = Poisson cumulative distribution function x = number of survivors (s) C = confidence level (prob(s>x|λ)) λ ≈ np (for large n, small p, where p is binomial probability)

8. Powell - Efficacy and Equivalence8 Calculate Sample Size from Look-up Table C = 0.95Set p at desired level sλ=npSolve for n 03.00 14.74 26.30 37.75 49.15 510.51 611.84 713.15 814.43 915.71 1016.96 Source: Couey & Chew (1986, Table 1)

9. Powell - Efficacy and Equivalence9 C = 0.95p = 0.0001 sλ=npn 03.0030,000 14.7447,400 26.3063,000 37.7577,500 49.1591,500 510.51105,100 611.84118,400 713.15131,500 814.43144,300 915.71157,100 1016.96169,600 Calculate Sample Size from Look-up Table

10. Powell - Efficacy and Equivalence10 C = 0.95p = 0.0001 sλ=npn 03.0030,000 14.7447,400 26.3063,000 37.7577,500 49.1591,500 510.51105,100 611.84118,400 713.15131,500 814.43144,300 915.71157,100 1016.96169,600 Equivalent Treatment Efficacy Illustrates that different combinations of s and n are equivalent in terms of demonstrating the efficacy of phytosanitary treatment.

11. Powell - Efficacy and Equivalence11 Spreadsheet Snapshot Where: n 1 = sample size from look-up table for p = 0.0001 and C 1 = 0.95; n 2 = precise sample size for p = 0.0001, C 2 = 0.9500. C is calculated in Excel © as: C = 1-BINOMDIST(s, n, p, 1). Sample size (n 2 ) is calculated numerically to desired precision using the Excel © add-in tool Solver ©. (Calculated values confirmed using SAS ©, where C = 1-CDF(‘BINOMIAL’,s, p, n).) Spreadsheet model provides greater numerical precision than look-up table for calculating sample size requirements (n), although need to consider measurement error for large counts. Provides greater flexibility to calculate sample size (n) required for any desired treatment efficacy (p) and confidence level (C), where C = p(s>x|n,p). sn1n1 C1C1 n2n2 C2C2 n 1 -n 2 030,0000.950229,9560.950044 147,4000.949847,4250.9500-25 263,0000.950262,9650.950035

12. Powell - Efficacy and Equivalence12 Combining Confidence Levels in Assessing Treatment Efficacy One or more initial rounds of testing may be analyzed to determine how much more data is required to meet a defined treatment performance standard. Combining confidence levels. If an overall confidence level C is required, and the first round of testing provides C 1, then the level of confidence C 2 required for the second round is: C 2 = (C-C 1 )/(1-C 1 ) (Source: Cannon, R.M. 2002. Demonstrating disease freedom – combining confidence levels. Preventive Veterinary Medicine. 52: 227-249.) Example: Design standard is 95% confidence that survival is no more than p. An initial round of testing provides 86% confidence that survival is no more than p. Applying the equation for combining confidence levels, the second round of testing has to provide 64% confidence that the survival is no more than p. Won’t find that value in a sample size look-up table. (Note: this equation also useful for pest prevalence surveillance.)

13. Powell - Efficacy and Equivalence13 Beta Distribution - Uncertainty about Binomial Probability (p) Recall that the binomial distribution applies to the random variable s: s ~ Binomial(n,p); where n & p are fixed (C = 1- F(x) = p(s>x|n,p)). To assess the efficacy and equivalence of phytosanitary treatments, we’d like to draw inferences about p, where p = prob(plant pest survival). p is a distribution of values, not a point estimate, with 0≤p≤1. Given observed data (n & s), the binomial probability (p): p ~ Beta(α,β) where: μ = α/(α+β) σ 2 = αβ/[(α+β) 2 (α+β+1)]

14. Powell - Efficacy and Equivalence14 Beta Parameter Estimation Different methods for estimating the parameters of Beta(α,β) Method 1: p prior ~ Uniform(0,1) α est = s+1 β est = n-s+1 For n = s = 0 (no data), Beta(1,1) ~ Uniform(0,1) Advantage: can handle case where s = 0 Disadvantage: estimate of binomial probability (p) is biased toward 0.5. (Probability of s in “future” trials is 50-50.)

15. Powell - Efficacy and Equivalence15 Method 2: Method of Matching Moments (MoMM) Disadvantage: can’t handle case where s = 0 Advantage: unbiased estimate of binomial probability (p) Beta Parameter Estimation

16. Powell - Efficacy and Equivalence16 Beta – Uniform Prior n 1 = sample size from look-up table for p = 0.0001 (1E-04) and C = 0.95 s = number of survivors, α est = s+1, β est = n-s+1 p u = BETAINV(C, α, β) = 95% confidence level for p (prob(survival)) n1n1 30,00047,40063,00077,50091,500105,100 s012345 α est 123456 β est 30,00147,40062,99977,49891,497105,096 mean est 3.33E-054.22E-054.76E-055.16E-055.46E-055.71E-05 var est 1.11E-098.90E-107.56E-106.66E-105.97E-105.43E-10 pupu 9.98E-051.00E-049.99E-051.00E-04

17. Powell - Efficacy and Equivalence17 Beta – Uniform Prior n 1 = sample size from look-up table for p = 0.0001 (1E-04) and C = 0.95 s = number of survivors, α est = s+1, β est = n-s+1 p u = BETAINV(C, α, β) = 95% confidence level for p (prob(survival)) n1n1 30,00047,40063,00077,50091,500105,100 s012345 α est 123456 β est 30,00147,40062,99977,49891,497105,096 mean est 3.33E-054.22E-054.76E-055.16E-055.46E-055.71E-05 var est 1.11E-098.90E-107.56E-106.66E-105.97E-105.43E-10 pupu 9.98E-051.00E-049.99E-051.00E-04 Equivalent to the results calculated from the sample size look-up table.

18. Powell - Efficacy and Equivalence18 Beta – MoMM n2n2 30,000 47,40063,00077,50091,500 s012345 Sample pn/a3.33E-054.22E-054.76E-055.16E-055.46E-05 Sample s 2 n/a1.11E-098.9E-107.56E-106.66E-105.97E-10 α est 1.0000*0.99991.99992.99993.99994.9999 β est 30,001*29,99747,39662,99577,49491,493 pupu 9.98E-059.99E-051.00E-049.99E-051.00E-04 n 2 = sample size s = number of survivors, α* = s+1, β* = n-s+1 p u = BETAINV(C, α, β) = 95% confidence level for p (survival)

19. Powell - Efficacy and Equivalence19 Beta – MoMM n2n2 30,000 47,40063,00077,50091,500 s012345 Sample pn/a3.33E-054.22E-054.76E-055.16E-055.46E-05 Sample s 2 n/a1.11E-098.9E-107.56E-106.66E-105.97E-10 α est 1.0000*0.99991.99992.99993.99994.9999 β est 30,001*29,99747,39662,99577,49491,493 pupu 9.98E-059.99E-051.00E-049.99E-051.00E-04 n 2 = sample size s = number of survivors, α* = s+1, β* = n-s+1 p u = BETAINV(C, α, β) = 95% confidence level for p (survival) MoMM yields unbiased estimate of prob(survival)

20. Powell - Efficacy and Equivalence20 Comparison of Results For n = 47,400 s = 1 (s/n = 2 x 10 -5 ) Uniform prior ~ Beta(2, 47400) MoMM ~ Beta(1, 47397) μ = 2 x 10 -5 μ = 4 x 10 -5

21. Powell - Efficacy and Equivalence21 Comparison of Results Let n = 30,000 s = 0 Uniform prior ~ Beta(1, 30001) Let n = 30,000 s= 1 MoMM ~ Beta(0.9999, 29997) 95%ile = 0.0001

22. Powell - Efficacy and Equivalence22 Simulation to Confirm MoMM Results p = s/n s~binomial(n,p) For SAS © users - DATA; DO I=1 TO 1E5; /* 1E5 = no. iterations */ P =(RANBIN(SEED,N,S/N))/N; OUTPUT; END; /* SEED = random number seed */ RUN; PROC UNIVARIATE; HISTOGRAM P; RUN; SN95%ILE(P) 1300001.00E-04 2474001.05E-04 3630009.52E-05 4775001.03E-04 5915009.84E-05

23. Powell - Efficacy and Equivalence23 Combine information from a series of replicated trials Bayes Rule implies: Likelihood function (in Excel © ) = BINOMDIST(s,n,p,0). Beginning with a suitable starting prior (e.g., p prior1 = uniform(0,1)), we can use Bayes rule to sequentially update the estimated treatment effect (p = prob(survival)) from a series of replicated trials. The posterior distribution of p after considering results of the i th trial becomes the prior distribution of p for considering the results of the (i+1) th trial.

24. Powell - Efficacy and Equivalence24 Sequential Updating TRIAL 1TRIAL 2TRIAL 3 survivors (s)s000 sample size (n)n10000 p (prop. survivors)priorlikproductposterior1likproductposterior2likproductposterior3 1.000E-070.00019.990E-019.990E-059.996E-049.990E-019.986E-041.998E-039.990E-011.996E-032.996E-03 2.000E-070.00019.980E-019.980E-059.986E-049.980E-019.967E-041.994E-039.980E-011.990E-032.987E-03 3.000E-070.00019.970E-019.970E-059.976E-049.970E-019.947E-041.990E-039.970E-011.984E-032.978E-03 4.000E-070.00019.960E-019.960E-059.967E-049.960E-019.927E-041.986E-039.960E-011.978E-032.969E-03 5.000E-070.00019.950E-019.950E-059.957E-049.950E-019.907E-041.982E-039.950E-011.972E-032.960E-03 6.000E-070.00019.940E-019.940E-059.947E-049.940E-019.887E-041.978E-039.940E-011.966E-032.951E-03 7.000E-070.00019.930E-019.930E-059.937E-049.930E-019.867E-041.974E-039.930E-011.960E-032.942E-03 8.000E-070.00019.920E-019.920E-059.927E-049.920E-019.848E-041.970E-039.920E-011.955E-032.933E-03 9.000E-070.00019.910E-019.910E-059.917E-049.910E-019.828E-041.966E-039.910E-011.949E-032.925E-03 1.000E-060.00019.900E-019.900E-059.907E-049.900E-019.808E-041.962E-039.900E-011.943E-032.916E-03... 1.000E-030.00014.517E-054.517E-094.520E-084.517E-052.042E-124.086E-124.517E-051.846E-162.770E-16 sum9.994E-021sum4.998E-011sum6.663E-011 Illustrative Example: 3 Replicate Trials of n = 10,000 with s = 0 survivors

25. Powell - Efficacy and Equivalence25 Sequential Updating TRIAL 1 TRIAL 2TRIAL 3 survivors (s)s 0 00 sample size (n)n 10000 p (prop. survivors)prior lik productposterior1likproductposterior2likproductposterior3 1.000E-070.0001 9.990E-01 9.990E-059.996E-049.990E-019.986E-041.998E-039.990E-011.996E-032.996E-03 2.000E-070.0001 9.980E-01 9.980E-059.986E-049.980E-019.967E-041.994E-039.980E-011.990E-032.987E-03 3.000E-070.0001 9.970E-01 9.970E-059.976E-049.970E-019.947E-041.990E-039.970E-011.984E-032.978E-03 4.000E-070.0001 9.960E-01 9.960E-059.967E-049.960E-019.927E-041.986E-039.960E-011.978E-032.969E-03 5.000E-070.0001 9.950E-01 9.950E-059.957E-049.950E-019.907E-041.982E-039.950E-011.972E-032.960E-03 6.000E-070.0001 9.940E-01 9.940E-059.947E-049.940E-019.887E-041.978E-039.940E-011.966E-032.951E-03 7.000E-070.0001 9.930E-01 9.930E-059.937E-049.930E-019.867E-041.974E-039.930E-011.960E-032.942E-03 8.000E-070.0001 9.920E-01 9.920E-059.927E-049.920E-019.848E-041.970E-039.920E-011.955E-032.933E-03 9.000E-070.0001 9.910E-01 9.910E-059.917E-049.910E-019.828E-041.966E-039.910E-011.949E-032.925E-03 1.000E-060.0001 9.900E-01 9.900E-059.907E-049.900E-019.808E-041.962E-039.900E-011.943E-032.916E-03... 1.000E-030.0001 4.517E-05 4.517E-094.520E-084.517E-052.042E-124.086E-124.517E-051.846E-162.770E-16 sum9.994E-021sum4.998E-011sum6.663E-011 Initial prior – all values of p are equally likely

26. Powell - Efficacy and Equivalence26 Sequential Updating TRIAL 1 TRIAL 2TRIAL 3 survivors (s)s0 00 sample size (n)n10000 p (prop. survivors)priorlikproductposterior1 likproductposterior2likproductposterior3 1.000E-070.00019.990E-019.990E-05 9.996E-04 9.990E-019.986E-041.998E-039.990E-011.996E-032.996E-03 2.000E-070.00019.980E-019.980E-05 9.986E-04 9.980E-019.967E-041.994E-039.980E-011.990E-032.987E-03 3.000E-070.00019.970E-019.970E-05 9.976E-04 9.970E-019.947E-041.990E-039.970E-011.984E-032.978E-03 4.000E-070.00019.960E-019.960E-05 9.967E-04 9.960E-019.927E-041.986E-039.960E-011.978E-032.969E-03 5.000E-070.00019.950E-019.950E-05 9.957E-04 9.950E-019.907E-041.982E-039.950E-011.972E-032.960E-03 6.000E-070.00019.940E-019.940E-05 9.947E-04 9.940E-019.887E-041.978E-039.940E-011.966E-032.951E-03 7.000E-070.00019.930E-019.930E-05 9.937E-04 9.930E-019.867E-041.974E-039.930E-011.960E-032.942E-03 8.000E-070.00019.920E-019.920E-05 9.927E-04 9.920E-019.848E-041.970E-039.920E-011.955E-032.933E-03 9.000E-070.00019.910E-019.910E-05 9.917E-04 9.910E-019.828E-041.966E-039.910E-011.949E-032.925E-03 1.000E-060.00019.900E-019.900E-05 9.907E-04 9.900E-019.808E-041.962E-039.900E-011.943E-032.916E-03... 1.000E-030.00014.517E-054.517E-09 4.520E-08 4.517E-052.042E-124.086E-124.517E-051.846E-162.770E-16 sum9.994E-021 sum4.998E-011sum6.663E-011 Update the estimated distribution of p based a series of trials, considering the new information sequentially.

27. Powell - Efficacy and Equivalence27 Sequential Updating TRIAL 1 TRIAL 2 TRIAL 3 survivors (s)s 0 0 0 sample size (n)n 10000 p (prop. survivors)priorlikproductposterior1likproductposterior2 likproductposterior3 1.000E-070.00019.990E-019.990E-059.996E-049.990E-019.986E-04 1.998E-03 9.990E-011.996E-032.996E-03 2.000E-070.00019.980E-019.980E-059.986E-049.980E-019.967E-04 1.994E-03 9.980E-011.990E-032.987E-03 3.000E-070.00019.970E-019.970E-059.976E-049.970E-019.947E-04 1.990E-03 9.970E-011.984E-032.978E-03 4.000E-070.00019.960E-019.960E-059.967E-049.960E-019.927E-04 1.986E-03 9.960E-011.978E-032.969E-03 5.000E-070.00019.950E-019.950E-059.957E-049.950E-019.907E-04 1.982E-03 9.950E-011.972E-032.960E-03 6.000E-070.00019.940E-019.940E-059.947E-049.940E-019.887E-04 1.978E-03 9.940E-011.966E-032.951E-03 7.000E-070.00019.930E-019.930E-059.937E-049.930E-019.867E-04 1.974E-03 9.930E-011.960E-032.942E-03 8.000E-070.00019.920E-019.920E-059.927E-049.920E-019.848E-04 1.970E-03 9.920E-011.955E-032.933E-03 9.000E-070.00019.910E-019.910E-059.917E-049.910E-019.828E-04 1.966E-03 9.910E-011.949E-032.925E-03 1.000E-060.00019.900E-019.900E-059.907E-049.900E-019.808E-04 1.962E-03 9.900E-011.943E-032.916E-03... 1.000E-030.00014.517E-054.517E-094.520E-084.517E-052.042E-12 4.086E-12 4.517E-051.846E-162.770E-16 sum9.994E-021sum4.998E-011 sum6.663E-011

28. Powell - Efficacy and Equivalence28 Sequential Updating TRIAL 1TRIAL 2 TRIAL 3 survivors (s)s 00 0 sample size (n)n 10000 p (prop. survivors)priorlikproductposterior1likproductposterior2likproductposterior3 1.000E-070.00019.990E-019.990E-059.996E-049.990E-019.986E-041.998E-039.990E-011.996E-03 2.996E-03 2.000E-070.00019.980E-019.980E-059.986E-049.980E-019.967E-041.994E-039.980E-011.990E-03 2.987E-03 3.000E-070.00019.970E-019.970E-059.976E-049.970E-019.947E-041.990E-039.970E-011.984E-03 2.978E-03 4.000E-070.00019.960E-019.960E-059.967E-049.960E-019.927E-041.986E-039.960E-011.978E-03 2.969E-03 5.000E-070.00019.950E-019.950E-059.957E-049.950E-019.907E-041.982E-039.950E-011.972E-03 2.960E-03 6.000E-070.00019.940E-019.940E-059.947E-049.940E-019.887E-041.978E-039.940E-011.966E-03 2.951E-03 7.000E-070.00019.930E-019.930E-059.937E-049.930E-019.867E-041.974E-039.930E-011.960E-03 2.942E-03 8.000E-070.00019.920E-019.920E-059.927E-049.920E-019.848E-041.970E-039.920E-011.955E-03 2.933E-03 9.000E-070.00019.910E-019.910E-059.917E-049.910E-019.828E-041.966E-039.910E-011.949E-03 2.925E-03 1.000E-060.00019.900E-019.900E-059.907E-049.900E-019.808E-041.962E-039.900E-011.943E-03 2.916E-03... 1.000E-030.00014.517E-054.517E-094.520E-084.517E-052.042E-124.086E-124.517E-051.846E-16 2.770E-16 sum9.994E-021sum4.998E-011sum6.663E-011

29. Powell - Efficacy and Equivalence29 Sequential Updating Spreadsheet Sequential UpdatingTRIAL 1TRIAL 2TRIAL 3 survivors (s)s000 sample size (n)n10000 p (prop. survivors)priorlikproductposterior1likproductposterior2likproductposterior3 1.000E-070.00019.990E-019.990E-059.996E-049.990E-019.986E-041.998E-039.990E-011.996E-032.996E-03 2.000E-070.00019.980E-019.980E-059.986E-049.980E-019.967E-041.994E-039.980E-011.990E-032.987E-03 3.000E-070.00019.970E-019.970E-059.976E-049.970E-019.947E-041.990E-039.970E-011.984E-032.978E-03 4.000E-070.00019.960E-019.960E-059.967E-049.960E-019.927E-041.986E-039.960E-011.978E-032.969E-03 5.000E-070.00019.950E-019.950E-059.957E-049.950E-019.907E-041.982E-039.950E-011.972E-032.960E-03 6.000E-070.00019.940E-019.940E-059.947E-049.940E-019.887E-041.978E-039.940E-011.966E-032.951E-03 7.000E-070.00019.930E-019.930E-059.937E-049.930E-019.867E-041.974E-039.930E-011.960E-032.942E-03 8.000E-070.00019.920E-019.920E-059.927E-049.920E-019.848E-041.970E-039.920E-011.955E-032.933E-03 9.000E-070.00019.910E-019.910E-059.917E-049.910E-019.828E-041.966E-039.910E-011.949E-032.925E-03 1.000E-060.00019.900E-019.900E-059.907E-049.900E-019.808E-041.962E-039.900E-011.943E-032.916E-03 1.000E-030.00014.517E-054.517E-094.520E-084.517E-052.042E-124.086E-124.517E-051.846E-162.770E-16 sum9.994E-021sum4.998E-011sum6.663E-011 Zoom in to take a closer look

30. Powell - Efficacy and Equivalence30 Sequential Updating Spreadsheet BDCFE 3 4 5 6 7 8 10007 Sum(E7:E10006) Sequential UpdatingTRIAL 1 survivors (s)s0 sample size (n)n10000 p (prop. survivors)priorlikproductposterior1 1.000E-070.0001=BINOMDIST(D$4,D$5,$B7,0)=C7*D7=E7/$E$10007 2.000E-070.0001=BINOMDIST(D$4,D$5,$B8,0)=C8*D8=E8/$E$10007 Discretized values of p in very small increments (1 x 10 -7 )

31. Powell - Efficacy and Equivalence31 Sequential Updating Spreadsheet BDCFE 3 4 5 6 7 8 10007 Sum(E7:E10006) Sequential UpdatingTRIAL 1 survivors (s)s0 sample size (n)n10000 p (prop. survivors)priorlikproductposterior1 1.000E-070.0001=BINOMDIST(D$4,D$5,$B7,0)=C7*D7=E7/$E$10007 2.000E-070.0001=BINOMDIST(D$4,D$5,$B8,0)=C8*D8=E8/$E$10007 Discretized values of p in very small increments (1 x 10 -7 ) Initial prior is that all values of p are equally likely

32. Powell - Efficacy and Equivalence32 Sequential Updating Spreadsheet BDCFE 3 4 5 6 7 8 10007 Sum(E7:E10006) Sequential UpdatingTRIAL 1 survivors (s)s0 sample size (n)n10000 p (prop. survivors)priorlikproductposterior1 1.000E-070.0001=BINOMDIST(D$4,D$5,$B7,0)=C7*D7=E7/$E$10007 2.000E-070.0001=BINOMDIST(D$4,D$5,$B8,0)=C8*D8=E8/$E$10007 Likelihood function in Excel © : BINOMDIST(s,n,p,0)

33. Powell - Efficacy and Equivalence33 Sequential Updating Spreadsheet BDCFE 3 4 5 6 7 8 10007 Sum(E7:E10006) Sequential UpdatingTRIAL 1 survivors (s)s0 sample size (n)n10000 p (prop. survivors)priorlikproductposterior1 0.00000010.0001=BINOMDIST(D$4,D$5,$B7,0)=C7*D7=E7/$E$10007 0.00000020.0001=BINOMDIST(D$4,D$5,$B8,0)=C8*D8=E8/$E$10007 s = D4 Likelihood function in Excel © : BINOMDIST(s,n,p,0)

34. Powell - Efficacy and Equivalence34 Sequential Updating Spreadsheet BDCFE 3 4 5 6 7 8 10007 Sum(E7:E10006) Sequential UpdatingTRIAL 1 survivors (s)s0 sample size (n)n10000 p (prop. survivors)priorlikproductposterior1 0.00000010.0001=BINOMDIST(D$4,D$5,$B7,0)=C7*D7=E7/$E$10007 0.00000020.0001=BINOMDIST(D$4,D$5,$B8,0)=C8*D8=E8/$E$10007 s = D4 n = D5 Likelihood function in Excel © : BINOMDIST(s,n,p,0)

35. Powell - Efficacy and Equivalence35 Sequential Updating Spreadsheet BDCFE 3 4 5 6 7 8 10007 Sum(E7:E10006) Sequential UpdatingTRIAL 1 survivors (s)s0 sample size (n)n10000 p (prop. survivors)priorlikproductposterior1 0.00000010.0001=BINOMDIST(D$4,D$5,$B7,0)=C7*D7=E7/$E$10007 0.00000020.0001=BINOMDIST(D$4,D$5,$B8,0)=C8*D8=E8/$E$10007 s = D4 n = D5 p values in Col B Likelihood function in Excel © : BINOMDIST(s,n,p,0)

36. Powell - Efficacy and Equivalence36 Sequential Updating Spreadsheet BDCFE 3 4 5 6 7 8 10007 Sum(E7:E10006) Sequential UpdatingTRIAL 1 survivors (s)s0 sample size (n)n10000 p (prop. survivors)priorlikproductposterior1 0.00000010.0001=BINOMDIST(D$4,D$5,$B7,0)=C7*D7=E7/$E$10007 0.00000020.0001=BINOMDIST(D$4,D$5,$B8,0)=C8*D8=E8/$E$10007 posterior is proportional to prior * lik

37. Powell - Efficacy and Equivalence37 Sequential Updating Spreadsheet BDCFE 3 4 5 6 7 8 10007 Sum(E7:E10006) Sequential UpdatingTRIAL 1 survivors (s)s0 sample size (n)n10000 p (prop. survivors)priorlikproductposterior1 0.00000010.0001=BINOMDIST(D$4,D$5,$B7,0)=C7*D7=E7/$E$10007 0.00000020.0001=BINOMDIST(D$4,D$5,$B8,0)=C8*D8=E8/$E$10007 posterior is proportional to prior * lik normalized posterior sums to 1

38. Powell - Efficacy and Equivalence38 Sequential Updating Spreadsheet BDCFE 3 4 5 6 7 8 10007 Sum(E7:E10006) Sequential UpdatingTRIAL 1 survivors (s)s0 sample size (n)n10000 p (prop. survivors)priorlikproductposterior1 0.00000010.0001=BINOMDIST(D$4,D$5,$B7,0)=C7*D7=E7/$E$10007 0.00000020.0001=BINOMDIST(D$4,D$5,$B8,0)=C8*D8=E8/$E$10007 posterior1 based on TRIAL 1 becomes prior for updating based on TRIAL 2 Prior2

39. Powell - Efficacy and Equivalence39 Sequential Updating

40. Powell - Efficacy and Equivalence40 Sequential Updating

41. Powell - Efficacy and Equivalence41 Sequential Updating

42. Powell - Efficacy and Equivalence42 Sequential Updating

43. Powell - Efficacy and Equivalence43 Comparison of Sequential Updating v. Beta Distribution Fit to Cumulative Data Sequential: 3 replicate trials of n =10,000 with s = 0 Cumulative: n = 30,000; s = 0; s ~ Beta(s+1,n-s+1) = Beta (1, 30001)

44. Powell - Efficacy and Equivalence44 Combine information from a series of replicated trials As evidence accumulates, the influence of the prior diminishes, and the data dominate the posterior distribution. Conveniently, when the cumulative sample size is large (n ~ 10,000), we can obtain a good approximation of the posterior distribution of p by simply analyzing the combined data from replicate trials to estimate the Beta distribution parameters: p posterior → p ~ Beta(α est, β est ) where: n = ∑n i s = ∑s i i = 1 to j trials

45. Powell - Efficacy and Equivalence45 Account for uncertainty in control mortality Abbott’s Formula to correct for control mortality: Conventionally, p control treated as a deterministic value (point estimate). Fails to account for uncertainty in control mortality. Source: Abbott, W.S. 1925. A method of computing the effectiveness of an insecticide. J Econ Entomol 18: 265-267.

46. Powell - Efficacy and Equivalence46 Account for uncertainty in control mortality p control ~ Beta(α 1, β 1 ) p treat ~ Beta(α 2, β 2 ) p(survival, corrected for control mortality and accounting for uncertainty):

47. Powell - Efficacy and Equivalence47 Account for uncertainty in control mortality: empirical example Ex: 90% survival in control trial with n = 100 Control trial Treatment trial n10033,500 s900 sample p0.9n/a sample s 2 0.0009n/a α est 88.21 β est 9.833,501 p corrected estimated by Monte Carlo Simulation, using @RISK © Summary of Results: 95th %ile = 9.95 x 10 -5 Extra treatment trial sample size ~ 3,500 required to demonstrate p ≤ 0.0001 with 95% confidence.

48. Powell - Efficacy and Equivalence48 Disclaimers The opinions expressed herein are the views of the author and do not necessarily reflect the official policy or position of the United States Department of Agriculture. Reference herein to any specific commercial products, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government.