Decision-Theoretic Views on Switching Between Superiority and Non-Inferiority Testing. Peter Westfall Director, Center for Advanced Analytics and Business Intelligence Texas Tech University

Background MCP2002 Conference in Bethesda, MD, August 2002 J. Biopharm. Stat. special issue, to appear Articles: –Ng,T.-H. “Issues of simultaneous tests for non-inferiority and superiority” –Comment by G. Pennello –Comment by W. Maurer –Rejoinder by T.-H. Ng

Ng’s Arguments No problem with control of Type I errors in switching from N.I. to Sup. Tests However, it seems “sloppy”: –Loss of power in replication when there are two options –It will allow “too many” drugs to be called “superior” that are not really superior.

Westfall interjects for the next few slides Why does switching allow control of Type I errors? Three views: –Closed Testing –Partitioning Principle –Confidence Intervals

Closed Testing Method(s) Form the closure of the family by including all intersection hypotheses. Test every member of the closed family by a (suitable)  -level test. (Here,  refers to comparison-wise error rate). A hypothesis can be rejected provided that –its corresponding test is significant at level  and –every other hypothesis in the family that implies it is rejected by its  level test. Note: Closed testing is more powerful than (e.g.) Bonferroni.

Control of FWE with Closed Tests Suppose H 0j 1,..., H 0j m all are true (unknown to you which ones). You can reject one or more of these only when you reject the intersection H 0j 1 ...  H 0j m Thus, P(reject at least one of H 0j 1,..., H 0j m | H 0j 1,..., H 0j m all are true)  P(reject H 0j 1 ...  H 0j m | H 0j 1,..., H 0j m all are true) = 

Closed Testing – Multiple Endpoints H 0 :  1 =  2 =  3 =  4 =0 H 0 :  1 =  2 =  3 =0H 0 :  1 =  2 =  4 =0H 0 :  1 =  3 =  4 =0H 0 :  2 =  3 =  4 =0 H 0 :  1 =  2 =0H 0 :  1 =  3 =0 H 0 :  1 =  4 =0 H 0 :  2 =  3 =0 H 0 :  2 =  4 =0 H 0 :  3 =  4 =0 H 0 :  1 =0 p = H 0 :  2 =0 p = H 0 :  3 =0 p = H 0 :  4 =0 p = Where  j = mean difference, treatment -control, endpoint j.

Closed Testing – Superiority and Non-Inferiority     (Null: Inf.; Alt: Non-Inf)    (Null: not sup.; Alt: sup.) Note: The intersection of the non-inferiority hypothesis and the superiority hypothesis is equal to the non-inferiority hypothesis     (Null: Inf.; Alt: Non-Inf) Intersection of the two nulls

Why there is no penalty from the closed testing standpoint Reject     only if –     is rejected, and –     is rejected. (no additional penalty) Reject     only if –     is rejected, and –     is rejected. (no additional penalty) So both can be tested at 0.05; sequence is irrelevant.

Why there is no need for multiplicity adjustment: The Partitioning View Partitioning principle: –Partition the parameter space into disjoint subsets of interest –Test each subset using an  -level test. –Since the parameter may lie in only one subset, no multiplicity adjustment is needed. Benefits –Can (rarely) be more powerful than closure –Confidence set equivalence (invert the tests)

Partitioning Null Sets            You may test both without multiplicity adjustment, since only one can be true. LFC for    is   ; the LFC for    is  Exactly equivalent to closed testing.

Confidence Interval Viewpoint Contruct a 1-  lower confidence bound on , call it d L. If d L > 0, conclude superiority. If d L >    conclude non-inferiority. The testing and interval approaches are essentially equivalent, with possible minor differences where tests and intervals do not coincide (eg, binomial tests).

Back to Ng Ng’s Loss Function Approach Ng does not disagree with the Type I error control. However, he is concerned from a decision-theoretic standpoint So he compares the “Loss” when allowing testing of: –Only one, pre-defined hypothesis –Both hypotheses

Ng’s Example Situation 1: Company tests only one hypothesis, based on their preliminary assessment. Situation 2: Company tests both hypotheses, regardless of preliminary assessment,

Further Development of Ng Out of the “next 2000” products, –1000 are truly equally efficacious as A.C. –1000 are truly superior to A.C. Suppose further that the company either –Makes perfect preliminary assessments, or –Makes correct assessments 80% of the time

Perfect Classification; One Test Only

80% Correct Classification; One Test Only

No Classification; Both Tests Performed Ng’s concern: “Too many” Type I errors.

Westfall’s generalization of Ng Three – decision problem: –Superiority –Non-Inferiority –NS (“Inferiority”) Usual “Test both” strategy: –Claim Sup if 1.96 < Z –Claim NonInf if 1.96 –  0 < Z < 1.96 –Claim NS if Z < 1.96 –  0

Further Development Assume  0 = 3.24 (  90% power to detect non-inf.). True States of Nature –Inferiority:  < –Non-Inf: <  < 0 –Sup: 0 < 

Loss Function 0L12L13 L210L23 L31L320 Inf (  -3.24) NonInf (-3.24 <   0) Sup (0 <  ) Nature Claim NSNonInfSup

Prior Distribution – Normal + Equivalence Spike

Westfall’s Extension Compare –Ng’s recommendation to “preclassify” drugs according to Non-Inf or Sup, and –The “test both” recommendation Use % increase over minimum loss as a criteria. The comparison will depend on prior and loss!

Probability of Selecting “NonInf” Test Probit function, anchors are P(NonInf|  =0) = p s ; P(NonInf|  =3.24) = 1-p s. Ng suggests p s =.80.

Summary of Priors and Losses  ~ p  {I(  =0)} + (1-p)  N(  ; m , s 2 ) (3 parms) P(Select NonInf |  ) =  (a + b  ), where a,b determined by p s (1 parm) (only for Ng) Loss matrix (5 parms) Total: 3 or 4 prior parameters and 5 loss parameters. Not too bad!!!

Baseline Model  ~ (.2)  {I(  =0)} + (.8)  N(  ; 1, 4 2 ) P(Select NonInf |  ) =  (  ) (p s =.8) Loss matrix: (An attempt to quantify “Loss to patient population”) Inf (  -3.24) NonInf (-3.24 <   0) Sup (0 <  ) NS/InfNonInfSup Nature Claim

Consequence of Baseline Model Optimal decisions (standard decision theory; see eg Berger’s book): –Classify to NS when z < –Classify to NonInf when < z < 2.20 –Classify to Sup when 2.20 < z Ordinary rule: Cutpoints are -1.28, 1.96

Loss Matrix – Select and test only the NonInf hypothesis Inf (  -3.24) NonInf (-3.24 <   0) Sup (0 <  ) Z< < Z < <Z Nature Outcome

Loss Matrix – Select and test only the Sup hypothesis Inf (  -3.24) NonInf (-3.24 <   0) Sup (0 <  ) Z< < Z < <Z Nature Outcome

Deviation from Baseline: Effect of p

Deviation from Baseline: Effect of m

Deviation from Baseline: Effect of s

Deviation from Baseline: Effect of Correct Selection, p s

Changing the Loss Function c1c1 090 c  20c  10 0 Inf (  -3.24) NonInf (-3.24 <   0) Sup (0 <  ) NS/InfNonInfSup Nature Claim Multiply lower left by c; c>0

Deviation from Baseline: Effect of c

Conclusions The simultaneous testing procedure is generally more efficient (less loss) than Ng’s method, except: –When Type II errors are not costly –When a large % of products are equivalent A sidelight: The optimal rule itself is worth considering: –Thresholds for Non-Inf are more liberal, which allows a more stringent definition of non- inferiority margin