1. A nearly optimal sequential testing approach to permutation-based association testing Julian Hecker, Ingo Ruczinski, Brent Coull, Christoph Lange
Introduction
Permutation-based approaches address statistical issues in genetic association studies as: small sample sizes, rare genetic variants, design imbalances or non-normal phenotypes
Significance levels of interest are usually very small  permutation-based approaches are computational intensive
Adaptive permutation strategies in existing genetic association analysis tools (e.g. PLINK Chang et al. 2015) use heuristics to stop early if the p-value is obviously non-significant
Sequential testing approach
𝑝 true, unknown association p-value
sequence 𝑥 1 , 𝑥 2 ,… where 𝑥 1 =1 iff permuted statistic more extreme, 0 otherwise
We introduce a small indifference region and consider the hypotheses
𝐻 1 :𝑝≤ 𝑝 1 and 𝐻 2 :𝑝≥ 𝑝 2 = 𝑝 1 +𝑑
(e.g. 𝑝 1 =5∗ 10 −8 and 𝑑= 10 −8 )

2. A nearly optimal sequential testing approach to permutation-based association testing Julian Hecker, Ingo Ruczinski, Brent Coull, Christoph Lange
Objects Pre-specified error probabilities 𝛼 1 , 𝛼 2 (e.g. 𝛼 1 = 𝛼 2 = 10 −10 ). Define (Pavlov 1991) 𝜏 𝑖 𝛼 𝑖 ≔min{𝑛: 𝜋 𝑛 / sup 𝜃∈ 𝐷 𝑖 𝑝 𝑛 𝜃, 𝑥 𝑛 ≥ 𝛼 𝑖 −1 } for 𝑖=1,2, where 𝐷 1 = 0, 𝑝 1 , 𝐷 2 = 𝑝 2 ,1 , 𝑥 𝑛 = 𝑥 1 ,…, 𝑥 𝑛 , 𝑝 𝑛 𝜃, 𝑥 𝑛 = 𝑖=1 𝑛 𝑝( 𝜃, 𝑥 𝑖 ), 𝜋 𝑛 ≔ 𝑖=1 𝑛 𝑝( 𝜃 𝑖−1 , 𝑥 𝑖 ) and 𝜽 𝒊−𝟏 ≔ 𝒌=𝟏 𝒊−𝟏 𝒙 𝒌 + 𝟏 𝟐 𝒊 . 𝑝(𝜃,𝑥) Bernoulli density with parameter 𝜃.
Decision procedure STr
If 𝜏 1 𝛼 1 ≤ 𝜏 2 ( 𝛼 2 ), we set 𝜕=2 and 𝑁= 𝜏 1 ( 𝛼 1 ). If 𝜏 1 𝛼 1 > 𝜏 2 ( 𝛼 2 ), we set 𝜕=1 and 𝑁= 𝜏 2 ( 𝛼 2 ).

3. A nearly optimal sequential testing approach to permutation-based association testing Julian Hecker, Ingo Ruczinski, Brent Coull, Christoph Lange
Theorem (Pavlov 1991, Tartakovsky 2014) 1.) 𝑃 𝜃 𝛿=2 ≤ 𝛼 1 for 𝜃∈ 𝐷 1 and 𝑃 𝜃 𝛿=1 ≤ 𝛼 2 for 𝜃∈ 𝐷 2 2.) Let 𝐾 𝑡 1 , 𝑡 2 ,𝛼 be the class of all decision rules ( 𝑁 ′ , 𝜕 ′ ) such that 𝑃 𝜃 𝛿′=2 ≤ 𝑡 1 𝛼 for 𝜃∈ 𝐷 1 and 𝑃 𝜃 𝛿′=1 ≤ 𝑡 2 𝛼 for 𝜃∈ 𝐷 2 , then 𝐸 𝜃 𝑁 inf 𝑁 ′ , 𝜕 ′ ∈𝐾 𝑡 1 , 𝑡 2 ,𝛼 𝐸 𝜃 𝑁 ′ =1+𝑜 1 as 𝛼→0 for all 𝜃∈ 0,1 .
Error probabilities are strictly controlled
Approaches theoretical minimum number of expected permutations if error level goes to zero

4. A nearly optimal sequential testing approach to permutation-based association testing Julian Hecker, Ingo Ruczinski, Brent Coull, Christoph Lange
Comparison with confidence interval based approach
𝑝 empirical estimate of p-value after 𝑛 permutations
( 𝑝 − 𝑐 𝛼 𝑆𝐸, 𝑝 +𝑐 𝛼 𝑆𝐸) corresponding 1−𝛼 confidence interval
CI-based rule (CIr)
choose 𝜕=1 if 𝑝 + 𝑐 𝛼 𝑆𝐸≤ 𝑝 2 , set 𝜕=2 if 𝑝 − 𝑐 𝛼 𝑆𝐸≥ 𝑝 1
Ratio STr/CIr ratio overall number of permutations needed for 12,045,191 single variant tests in LD (simulated)
Red scenario:
Type 1 error at least for CIr
𝒑 𝟏 / 𝒑 𝟐
𝜶 𝟏 / 𝜶 𝟐
𝜶 CIr
ratio STr/CIr
1e-09/2e-09
1e-10/1e-10
1e-10
12.62
5e-08/6e-08
6.39
9e-04/1e-03
1e-10/4e-03
1.11

5. A nearly optimal sequential testing approach to permutation-based association testing Julian Hecker, Ingo Ruczinski, Brent Coull, Christoph Lange
Our approach…
… can be applied to arbitrary association test statistics.
… controls the error probabilities rigorously and approaches the theoretical minimum in terms of expected number of permutations.
… is easy to implement and no additional computational effort is required.
… is an alternative approach to sequential Monte Carlo hypothesis testing (Gandy 2009).
Example code available at:
Contact:
Julian Hecker