1. Covering Principle to Address Multiplicity in Hypothesis Testing
Huajiang Li, Ph.D. Avanir Pharmaceuticals
Hong Zhou , Ph.D. Arkansas State University

2. A question
How to deal with situations when constrains exist among hypotheses?
Two types of constrains correspond to parameter space and sample space
In ANOVA H1: µ1=µ2=µ3, H2: µ1=µ2, H3: µ1=µ3, H4: µ2=µ3 show constrains in parameter space
In clinical trials hypotheses on primary and secondary endpoints, or on high, medium or low doses form constrains in sample space
We propose a Covering Principle to handle constrains in sample space

3. A Motiving Example: Serial GatekeepingH1 R1 R2 H2 R3 H3
(a) Flow Chart of Decisions
(b) Venn Diagram of Rejection Regions

4. Outlines Covering Principle Application to Gate-Keeping problems
Concepts and Notations
Dominance relation
Application to Gate-Keeping problems
Comparison to graphical methods
Simulation
Conclusions

5. Concepts and Definitions
Individual hypothesis test
Null hypothesis H0 vs. alternative hypothesis Ha
Parameter 𝜃 and parameter space Ω
Sample data X and sample space S
Test function and rejection region
Type I error: probability of false rejection
Level-α test

6. Concepts and Definitions (cont’)
Multiple hypothesis test
Multiple null hypotheses Hi, i=1, 2, …, n
Multiple hypothesis test 𝜙 = { 𝜙 𝑖 , i=1, 2, …, n}
Each 𝜙 𝑖 has a corresponding rejection region Ri
False rejection: 𝜃 Hi, but Hi is rejected
Type I error/familywise error rate(FWER): probability of rejecting any true null hypothesis
Power: probability of rejecting any false null hypothesis

7. Dominance Relation
Definition: Denote 𝑵= 𝟏,𝟐, …, 𝒏 as index set of family of null hypothesis H1,H2,…,Hn and corresponding rejection region R1,R2,…,Rn. If there exist index set 𝑰⊂𝑵 and 𝑱⊂𝑵, 𝑰∩𝑱=∅,such that 𝒊∈𝑰 𝑹 𝒊 ⊆ 𝒋∈𝑱 𝑹 𝒋 , we say 𝑰 is dominated by 𝑱, or equivalently 𝑱 dominates 𝑰, denoted by 𝑰≺ 𝑱.
Dominance relation defines a partial order on subsets of null hypotheses
Index set S is said to satisfy a relation 𝑰≺ 𝑱 if 𝑱⊆𝑺 and 𝑰∩𝑺≠∅
We say S satisfies a relation set D if S satisfies at least one relation in D

8. Covering Principle Covering Principle: Suppose ∅≠𝐼⊂𝑁,∅≠𝐽⊂𝑁, 𝐼∩𝐽=∅, and
∪ 𝑖∈𝐼 𝑅 𝑖 ⊆ ∪ 𝑗∈𝐽 𝑅 𝑗
Denote: 𝜙 𝑗 = 𝜙 𝑖 𝑗 :𝑖∈𝑁\j , 𝑗∈𝐽
𝜙 𝐼 = 𝜙 𝑖 𝐼 :𝑖∈𝑁\I
Define: 𝜓 𝑖 = min 𝑚𝑖𝑛 𝑗∈𝐽 𝜙 𝑖 𝑗 , 𝜙 𝑖 𝐼 , 𝑖∈𝑁\I min 𝑚𝑖𝑛 𝑗∈𝐽 𝜙 𝑖 𝑗 , 𝑚𝑎𝑥 𝑗∈𝐽 𝜓 𝑗 , 𝑖∈𝐼
If 𝜙 𝑗 ∈ Φ 𝛼 𝑁\j , ∀𝑗∈𝐽, and 𝜙 𝐼 ∈ Φ 𝛼 𝑁\I ,
then 𝜓 𝑖 :𝑖∈𝑁 ∈ Φ 𝛼 𝑁 .

9. Explanation of Covering Principle
𝐼∩𝐽=∅, and ∪ 𝑖∈𝐼 𝑅 𝑖 ⊆ ∪ 𝑗∈𝐽 𝑅 𝑗 : Sets 𝐼 and 𝐽 are disjoint
𝜙 𝑗 = 𝜙 𝑖 𝑗 :𝑖∈𝑁\j , 𝑗∈𝐽 𝜙 𝐼 = 𝜙 𝑖 𝐼 :𝑖∈𝑁\I
Decompose set N into | 𝐽 |+1 subsets
| 𝐽 | Subsets: 𝑖:𝑖∈𝑁\j , 𝑗∈𝐽 and one subset: 𝑖:𝑖∈𝑁\I
Define: 𝜓 𝑖 = min 𝑚𝑖𝑛 𝑗∈𝐽 𝜙 𝑖 𝑗 , 𝜙 𝑖 𝐼 , 𝑖∈𝑁\I min 𝑚𝑖𝑛 𝑗∈𝐽 𝜙 𝑖 𝑗 , 𝑚𝑎𝑥 𝑗∈𝐽 𝜓 𝑗 , 𝑖∈𝐼
Decision rules: for an individual hypothesis 𝐻 𝑖 ,
(a) If 𝑖∈𝑁\I, then it will be rejected as long as it is rejected in all subsets which contain it.
(b) If 𝑖∈𝐼, then at least one of its dominant hypothesis must be rejected first, and 𝐻 𝑖 it is also rejected in all subsets which contain it.
If 𝜙 𝑗 ∈ Φ 𝛼 𝑁\j , ∀𝑗∈𝐽, and 𝜙 𝐼 ∈ Φ 𝛼 𝑁\I , then 𝜓 𝑖 :𝑖∈𝑁 ∈ Φ 𝛼 𝑁 .
The multiple test based on covering principle will strongly control familywise error rate if every subset can strongly control the familywise error rate.

10. Using Covering Principle
Step 1: Analyze the order of testing hypotheses and construct the dominance relations among hypotheses
Step 2: Decompose the whole family of n hypotheses into a few subsets
Step 3: Test each hypothesis Hi in each subset in which Hi is contained. Different 𝛼-level multiple hypotheses testing procedures can be used for different subsets.
Step 4: Consolidate results from all decomposed subsets to make conclusions on each hypothesis
A dominant or a non-constraint hypothesis Hi will be rejected if Hi is rejected in all decomposed subsets in which Hi is contained
A dominated hypothesis Hi will be rejected if at least one of its dominant hypotheses is rejected first. In addition, Hi must be also rejected in all subsets in which Hi is contained.

11. Gatekeeping Problems
Divide whole hypothesis family into several ordered non-overlapped sub-families F1,…,Fn. Fi serves as gatekeeper of Fi+1, i.e. if Fi is not rejected (at least one or all hypotheses are rejected) then Fi+1 will be accepted automatically
Serial and parallel gatekeeping procedures are very popular in clinical trials
Various procedures including graphical approach, chain procedure, multistage procedure and mixture procedure were proposed to address general gatekeeping problems
Gatekeeping is a special case of dominance relations

12. Application to Parallel gate-keeping (1)
Rejection region R 4 ⊆ R 1 ∪ R 2 ∪ R 3
Dominance relation {𝟒} ≺ {𝟏,𝟐,𝟑}
Test sub-families {H1,H2,H3}, {H1,H2,H4}, {H1,H3,H4}, {H2,H3,H4}
Reject H1 if it is rejected in all sub-family tests; Similarly for H2 or H3
Reject H4 if it is rejected in all sub-family tests AND either H1 or H2 or H3 is rejected R1 H1 H2 H3 R4 R2 R3 H4

13. Application to Parallel gate-keeping (2)
Rejection region R 3 ∪R 4 ⊆ R 1 ∪ R 2
Dominance relation {𝟑,𝟒} ≺ {𝟏,𝟐}
Test sub-families {H1, H2}, {H1, H3, H4}, {H2, H3, H4}
Reject H1 if it is rejected in all sub-family tests; Similarly for H2
Reject H3 if it is rejected in all sub-family tests AND either H1 or H2 is rejected; Similarly for H4 H1 H2 R1 R2 R3 R4 H3 H4

14. Application to Mixed gate-keeping (1)
Rejection region R 4 ⊆ R 3 ⊆ R 1 ∪ R 2
Dominance relation {𝟒} ≺ {𝟑} ≺ {𝟏,𝟐}
Test sub-families {H1, H2}, {H1, H3}, {H2, H3}, {H1, H4}, {H2, H4}
Reject H1 if it is rejected in all sub-family tests; Similarly for H2
Reject H3 if it is rejected in all sub-family tests AND either H1 or H2 is rejected
Reject H4 if it is rejected in all sub-family tests AND H3 is rejected H1 H2 R1 R2 R3 R4 H3 H4

15. Application to Mixed gate-keeping (2)
Rejection region R 5 ⊆ R 3 ⊆ R 1 , R 6 ⊆ R 4 ⊆ R 2
Dominance relation {𝟓} ≺ {𝟑} ≺ {𝟏}, {𝟔} ≺ {𝟒} ≺ {𝟐}
Test sub-families {H1, H2}, {H1, H4}, {H1, H6}, {H3, H4}, {H3, H6}, {H5, H6}, {H2, H3}, {H2, H5}, {H4, H5}
Reject H1 if it is rejected in all sub-family tests; Similarly for H2
Reject H3 if it is rejected in all sub-family tests AND its upper level hypothesis is rejected; Similarly for H4 , H5, H6 H1 H2 R1 R2 R3 R4 R5 R6 H3 H4 H5 H6

16. Comparison to Graphical Methods
Suppose we want to test two doses of a drug to see whether each is clinically effective
Primary Hypotheses: 𝐻 1 and 𝐻 2
Secondary Hypotheses: 𝐻 3 and 𝐻 4
Tertiary Hypotheses: 𝐻 5 and 𝐻 6
The drug can be claimed clinically effective if at least one primary hypothesis is rejected;
The secondary and tertiary hypotheses cannot be considered if its upper level hypothesis isn’t rejected
We want to test whether this drug is effective at a significance level of 𝛼=0.05 by using Graphical Method and Covering Principle Method H1 H2 H3 H4 H5 H6
Low
dose
High
dose

17. Graphical Method: Strategy A
Each hypotheses is assigned an initial significance level
When a hypothesis is rejected, it’s significance level is transferred to the other hypotheses
Alpha
0.05
Hypothesis
P-value H1
0.0409 H2
0.0003 H3
0.0184 H4
0.0162 H5
0.0139 H6
0.0484
Most common method

18. Graphical Method A (Cont’)
𝐻 2 and 𝐻 4 are rejected
Hypothesis
P-value H1
0.0409 H2
0.0003 H3
0.0184 H4
0.0162 H5
0.0139 H6
0.0484

19. Graphical Method: Strategy B
Only 𝐻 21 is rejected
Alpha
0.05 H1
0.0409 H2
0.0003 H3
0.0184 H4
0.0162 H5
0.0139 H6
0.0484

20. Covering Principle Method
Decomposition with Covering Principle:
Decompose six hypotheses: 𝐻 1 , 𝐻 2 , 𝐻 3 , 𝐻 4 , 𝐻 5 , 𝐻 6 into 8 subsets: {𝟏,𝟐}, {𝟏,𝟒}, {𝟏,𝟔}, 𝟑,𝟒 , {𝟑,𝟔}, 𝟓,𝟔 , 𝟐,𝟑 , {𝟐,𝟓}, 𝟒,𝟓
Dominance relations:
𝑅 5 ⊆ 𝑅 3 ⊆ 𝑅 1
𝑅 6 ⊆ 𝑅 4 ⊆ 𝑅 2 H1 H2 H3 H4 R1 R2 R3 R4 R5 R6 H5 H6
Low
dose
High
dose

21. Illustration of Decomposition
1,2,3,4,5,6
1,2,4,5,6
1,2,4,6
1,2,6
1,2
1,6
1,4,6
1,4
2,4,5,6
2,3,4,5,6
2,3,4,6
2,3,6
2,3
3,6
3,4,6
3,4
2,5,6
2,5
5,6
4,5,6
4,5
𝟔∝2
𝟒∝𝟐
𝟓∝𝟏
𝟔∝𝟒
𝟔∝𝟐
𝟒∝2
𝟑∝𝟏
𝟔∝4
𝟓∝3
𝟔∝𝟐
𝟒∝𝟐
𝟔∝4

22. Testing Subsets with Same Level of Importance
For the subsets ( 𝐻 1 , 𝐻 2 ), ( 𝐻 3 , 𝐻 4 ) and ( 𝐻 5 , 𝐻 6 ) test using Holm’s Method.
Subset
𝐻 1
𝐻 2
𝐻 3
𝐻 4
𝐻 5
𝐻 6
( 𝐻 1 , 𝐻 2 )
Reject -
( 𝐻 3 , 𝐻 4 )
( 𝐻 5 , 𝐻 6 )
Subsets containing hypotheses of the same significance
Primary-Primary Secondary-Secondary
H1 and H2 are rejected in this subset H1
0.0409 H2
0.0003 H3
0.0184 H4
0.0162 H5
0.0139 H6
0.0484

23. Testing Subsets with Different Levels of Importance
For the subsets ( 𝐻 1 , 𝐻 4 ), ( 𝐻 1 , 𝐻 6 ), ( 𝐻 2 , 𝐻 3 ), ( 𝐻 2 , 𝐻 5 ), ( 𝐻 3 , 𝐻 6 ), ( 𝐻 4 , 𝐻 5 ) test using the Fixed sequence method.
Subset
𝐻 1
𝐻 2
𝐻 3
𝐻 4
𝐻 5
𝐻 6
( 𝐻 1 , 𝐻 4 )
Reject -
( 𝐻 1 , 𝐻 6 )
( 𝐻 2 , 𝐻 3 )
( 𝐻 2 , 𝐻 5 )
( 𝐻 3 , 𝐻 6 )
( 𝐻 4 , 𝐻 5 ) H1
0.0409 H2
0.0003 H3
0.0184 H4
0.0162 H5
0.0139 H6
0.0484

24. Graphical Method vs Covering Principle
Hypotheses are rejected overall if they’re rejected in all subsets they’re contained in
In this scenario, the Holms/Fallback Method using the Covering Principle is more powerful than the traditional Graphical Method.
Hypothesis
Holm/Fallback
Graph A
Graph B
𝐻 1
Reject
𝐻 2
𝐻 3
𝐻 4
𝐻 5
𝐻 6

25. Simulation Study Methods Studied The Covering Principle using:
Holm’s Method for all subsets
Holm’s Method for subsets with hypotheses of the same importance and the Fixed Testing Sequence Procedure for hypotheses of different levels of importance
Holm’s Method for subsets with hypotheses of the same importance and the Fallback Procedure for hypotheses of different levels of importance
Graphical Methods: Strategies A and B

26. Randomizing p-values
Using R, p-values for each hypothesis were generated based on multivariate normal distribution with given means and common correlation

27. Covering Principle Using Holm’s Procedure/Fixed vs Graph A

28. Covering Principle Using Holm’s Procedure/Fixed vs Graph B

29. Holm/Fallback (0.9,0.1) vs Graph A

30. Simulation Results
Our simulation shows that multiple tests based on covering principle are more powerful than graphical method when primary objective are concerned in various scenarios
Covering principle is flexible to tailor the special needs of clinical trials by using different multiple test procedures for different subsets and increase the power

31. Summary and Conclusion
We proposed the covering principle and showed it can be applied to various multiple test scenarios (parallel and mixed gate-keeping) in clinical trials
It is a principle rather than a specific testing procedure. We can use any valid multiple testing procedures on subfamilies after decomposition
It performs sample space analysis using relations of union of rejection regions in sample space in contrast to closure/partitioning principle using intersection of hypotheses in parameter space

32. Selected references
MARCUS, R., PERITZ, E. and GABRIEL, K.R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika – 660.
FINNER, H. and STRASSBURGER, K. (2002). The partitioning principle: a powerful tool in multiple decision theory. Ann. Statist – 1213.
SONNEMANN, E. (2008). General Solutions to Multiple Testing Problems. Biometrical Journal – 656.
DMITRIENKO, ET AL. (2010). Multiple Testing Problems in Pharmaceutical Statistics. CRC Press.
Li, H.J. and Zhou, H. A new approach to address multiplicity in hypothesis testing. (re-submitted and under review)

33. Thank you!