1. Single-Step Simes Test Procedures for Multiple Testing
Matthew Hudson MS, Prosoft Clinical
Joshua Naranjo PhD, Western Michigan University
Dror Rom PhD, Prosoft Clinical

2. Simes Procedure for Multiple Tests of Significance (1986)
Assume m independent hypotheses and let P(1), …, P(m) be the order p-values.
Global Null: H0 = {H1, …, Hm}
Reject H0 if P(j) ≤ jα/m, for any j = 1, …, m
Simes also holds for certain positively correlated tests in the one- sided case.

3. Definitions Type 1 Error: The rejection of a true null hypothesis.
Family-wise error (FWE) rate: The probability of making at least one type 1 error.

4. Definitions from Hochberg and Tamhane (1987)
A testing procedure is said to control the FWE rate in the weak sense if the probability of making at least one type 1 error is less than or equal to α under the global null hypothesis.
A testing procedure is said to control the FWE rate in the strong sense if the probability of making at least one type 1 error is less than or equal to α under all configurations of true and false hypotheses.

5. Limitation of Simes: Simes test does not make any inference about the individual hypotheses The Hochberg procedure (1988) is a commonly used Simes based procedures for testing individual hypotheses.

6. Assume m hypotheses and that Simes’s test has a type 1 error rate ≤ α, then the Hochberg procedures has strong control of the family- wise error (FWE) rate as defined by Hochberg and Tamhane (1987).
Hochberg’s Procedure (1988)
For j=m, m-1, …, 1, if P(j) ≤ α/(m-j+1), then reject all H(j') where j' ≤ j

7. Example: Assume a study with m=3 hypotheses yielding p-values of 0
Example: Assume a study with m=3 hypotheses yielding p-values of 0.031, 0.032, and Hochberg critical values: j
P-value CV
CV (α=0.05) 3
P(3) α
0.0500 2
P(2)
α/2
0.0250 1
P(1)
α/3
0.0167

8. Results for Example: P-values: 0. 031, 0. 032, 0
Results for Example: P-values: 0.031, 0.032, Simes test rejects since P(2) = ≤ 2α/3 = , therefore proceed to the Hochberg. Hochberg: No hypothesis rejected.

9. Summary and Limitations:
Hochberg’s procedure is a Simes global test based procedure for testing individual hypotheses that is easy to implement.
All Simes based procedures for making inferences on individual hypotheses are conservative as they may not reject any individual hypothesis when Simes test rejects the global null hypothesis.

10. Extended single-step Simes testing procedure
Consider m test statistics for testing hypotheses H1, .., Hm, then if P(j) ≤ jα/m for any j=1, …, m, then reject:
H(1)
Any Hi such that Pi ≤ α/m
All Hi if P(m) ≤ α

11. Extended single-step Simes Results for Example: P-values: 0. 031, 0
Extended single-step Simes Results for Example: P-values: 0.031, 0.032, Simes global test rejects since P(2) = ≤ 2α/3 = , therefore proceed to the extended single-step Simes. Extended single-step Simes: Reject H(1) Hochberg procedure does not reject any individual hypothesis, but the extended single-step Simes does.

12. Proof of the Extended Single-Step Simes Procedure for m=3 Independent Normal Test Statistics Scenario 1: All three hypotheses are true. This is the global null, so type 1 error is ≤ α by Simes test. Scenario 2: Only one hypothesis is true. The probability to reject the one true hypothesis is ≤ α since no hypothesis can be rejected unless the p-value is ≤ α. Scenario 3: Two of the three hypotheses are true. Without loss of generality, assume H1 and H2 are true, then by the closure principle of Marcus et al. (1976), the procedure will have strong control of the FWE rate if 𝑃 𝑅𝑒𝑗𝑒𝑐𝑡 𝐻 1 ∩ 𝐻 2 ≤ α.

13. Figure 1: Simes Rejection Region

14. Figure 2: Extended Single-Step Simes Rejection of 𝐻 1 ∩ 𝐻 2

15. Figure 3: Extended Single-Step Simes Rejection of 𝐻 1 ∩ 𝐻 2 with Hochberg Rejection Overlay

16. Figure 3: Extended Single-Step Simes Rejection of 𝐻 1 ∩ 𝐻 2 with Hochberg Rejection Overlay for Testing H1 and H2 only.

17. Figure 4: Difference Between Rejection Regions of H1 for Extended Simple-Step Simes and Hochberg for a fixed α < P2 ≤ 1 (P-values)

18. Figure 5: Difference Between Rejection Regions of H1 for Extended Simple-Step Simes and Hochberg for a fixed α < P2 ≤ 1 (Z values) (One-Sided Testing)

19. The following can be shown using the monotone likelihood
ratio property:
𝑥 13 𝑥 𝑥 1 𝑥 25 𝑓 0,0 𝑥 1 , 𝑥 2 𝑑𝑥 2 𝑑𝑥 𝑥 11 𝑥 𝑥 21 𝑥 1 𝑓 0,0 𝑥 1 , 𝑥 2 𝑑𝑥 2 𝑑𝑥 ≤ 𝑥 13 𝑥 𝑥 1 𝑥 25 𝑓 0 ,𝜇 𝑥 1 , 𝑥 2 𝑑𝑥 2 𝑑𝑥 𝑥 11 𝑥 𝑥 21 𝑥 1 𝑓 0 ,𝜇 𝑥 1 , 𝑥 2 𝑑𝑥 2 𝑑𝑥 1
The left side is the ratio of the two areas under the null, which is equal to 1.
1 ≤ 𝑥 13 𝑥 𝑥 1 𝑥 25 𝑓 0 ,𝜇 𝑥 1 , 𝑥 2 𝑑𝑥 2 𝑑𝑥 𝑥 11 𝑥 𝑥 21 𝑥 1 𝑓 0 ,𝜇 𝑥 1 , 𝑥 2 𝑑𝑥 2 𝑑𝑥 1

20. The right side is the area of the Hochberg region over the area of the extended single-step Simes procedure when μ1 = 0 and μ3 > 0. The Hochberg procedure has a type 1 error rate ≤ α. Since this ratio is ≥ 1, the 𝑃 𝑅𝑒𝑗𝑒𝑐𝑡 𝐻 1 ∩ 𝐻 2 ≤ α for the extended single-step Simes since it never exceeds that of the Hochberg procedure. Therefore, by the closure principle, the extended single-step Simes procedure has strong control of the family-wise error rate for three independent normal statistics.

21. Simulations (10M): Family-Wise Error Rates for Three Independent Normals with One False Hypothesis
Mean of False
Hypothesis
Simes Global Rejects
Extended Simes
FWE
Hochberg
Hommel
2.8
0.7602
0.0353
0.0468
0.0474
2.9
0.7900
0.0352
0.0473
0.0478
3.0
0.8170
0.0350
0.0475
0.0479

22. Simulations (10M): Power for Three Independent Normals with One False Hypothesis
Mean of False
Hypothesis
Minimum
Hypotheses
Rejected
Extended Simes
Hochberg
Hommel
2.8 1
0.7602
0.7583
0.7597 2
0.0260
0.0403 3
0.0022
2.9
0.7900
0.7883
0.0270
0.0417
3.0
0.8170
0.8154
0.8165
0.0279
0.0427
0.0023

23. Simulations (10M): Three Correlated Normals with One False Hypothesis Power to reject at least one hypothesis.
Mean of False
Hypothesis ρ
Extended Simes
FWE
Power
Hochberg
Hommel
2.9
0.1
0.0352
0.7872
0.7858
0.7868
0.3
0.0351
0.7829
0.7821
0.7827
0.5
0.0348
0.7809
0.7806
0.7808
0.7
0.0346
0.7799
0.9
0.0360

24. Simulations (10M): Ten Correlated (ρ=0
Simulations (10M): Ten Correlated (ρ=0.3) Normals Power to reject at least one hypothesis.
Mean of False
Hypothesis(es) #
False
Extended Simes
FWE
Power
Hochberg
Hommel
2.9 1
0.0405
0.6340
0.6318
0.6324 2
0.0359
0.8271
0.8179
0.8203 3
0.0318
0.9063
0.8952
0.8984 4
0.0276
0.9443
0.9336
0.9369

25. References
Hochberg, Y. (1988, December). A Sharper Bonferroni Procedure for Multiple Tests of Significance. Biometrika, 75(4),
Hochberg Y, Rom DM. Extensions of multiple testing procedures based on Simes’ test. Journal of Statistical Planning and Inference, 1995; 48: 141–52.
Hochberg, Y., & Tamhane, A. (1987). Multiple Comparison Procedures. Wiley.
Hommel, G. (1988, June). A Stagewise Rejective Multiple Test Procedure Based on a Modified Bonferroni Test. Biometrika, 75(2),
Marcus, R., Peritz, E., & Gabriel, K. R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika, 63(3),
Samuel-Cahn, E. (1996). Is the Simes improved Bonferroni procedure conservative?. Biometrika, 83, 928–933.
Sarkar, S.K. (1998). SOME PROBABILITY INEQUALITIES FOR ORDERED MTP2 RANDOM VARIABLES: A PROOF OF THE SIMES CONJECTURE. The Annals of Statistics, 26(2),
Sarkar, S. K., & Chang, C.-K. (1997, December). The Simes Method for Multiple Hypothesis Testing With Positively Dependent Test Statistics. Journal of the American Statistical Association, 92(440),
Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika, 73,

26. Hommel’s Procedure (1988)
Compute: 𝑗=𝑚𝑎𝑥 𝑖∈ 1, …, 𝑚 : 𝑃 𝑚−𝑖+𝑘 > 𝑘𝛼 𝑖 𝑓𝑜𝑟 𝑘=1, …, 𝑖
If the maximum does not exist, then reject all Hi, i=1, …, m.
Otherwise reject all Hi with Pi ≤ α/j.

27. Hommel critical values:k
P-value CV
CV (α=0.05) 1
P(3) α
0.0500 2
P(2)
α/2
0.0250 3
P(1)
α/3
0.0167
2α/3
0.0333

28. Results for Example: P-values: 0. 031, 0. 032, 0
Results for Example: P-values: 0.031, 0.032, Hommel: Procedure yields j=2. Therefore, reject hypotheses with p-values ≤ α/2 = No hypothesis rejected.