1. New Proposals for Multiple Test Procedures, Applied to Gene Expression Array Data Siegfried Kropf, Otto von Guericke University Magdeburg in cooperation with Jürgen Läuter, Magdeburg Peter H. Westfall, Lubbok, USA Markus Eszlinger, Knut Krohn, Leipzig IZBI-Workshop, November 11, 2002, Leipzig

2. 11.11.2002 Leipzig2 Otto von Guericke 1602-2002

3. 11.11.2002 Leipzig3 Example 1 6 quadrants, each 98 genes (double spotted): 588 genes + housekeeping genes Atlas Human Cancer 1.2 Array Example data – gene expression arrays

4. 11.11.2002 Leipzig4 applied to 6 patients with nodules in thyroid gland 3 hot, 3 cold, here not distinguished, + surrounding logarithmic transformation, double spots averaged correction with housekeeping gene at position i5a  distribution can hardly be checked with n = 6, standard deviation of the genes is not too different:

5. 11.11.2002 Leipzig5 Example 2 30 patients with nodules in thyroids 15 hot nodules, 15 cold nodules tissue samples of nodules and surrounding analysed with Affymetrix ® Gene Chips Signal log ratio nodule vs. surrounding from each patient for each of 12.625 genes outlier catching by additional logistic transformation approximately multivariate normal distribution “similar” variances for all genes, expectation 0 if unaffected

6. 11.11.2002 Leipzig6 Why familywise error rate? Discussion: non-statistical assessment – unadjusted statistical assessment – false discovery rate – familywise error rate familywise error rate is rather high claim, growing with dimension of array (in contrast to false discovery rate) if possible, however, then highest degree of security for the positive results of this one trial trials mostly with small or moderate samples sizes, not enough to rule out effects in case of non-significance, therefore at least the positive results should be as sure as possible results for FWE could at least be given in addition to other versions

7. 11.11.2002 Leipzig7 Procedure with data-driven ordering of hypotheses Starting point: Two well known procedures for MCPs controlling the FWE Testing with a-priori ordered hypotheses (without  -adjustment) Bonferroni-Holm (data dependent order, with adjustment) In analysis of high-dimensional gene expression arrays both not applicable/optimal.  We are looking for a method with data dependent ordering of hypotheses but without  -adjustment.

8. 11.11.2002 Leipzig8 New proposal (Kropf, 2000; Kropf and Läuter, 2002) Consider one-sample situation first: data matrix from n iid p-dimensional normal data vectors Aim: test of the local hypotheses H i :  i = 0 at the strong FWE . Procedure I: sort variables for decreasing values of, in that order carry out the unadjusted one-sample t tests for the variables as long as significance is attained.

9. 11.11.2002 Leipzig9 Remark: In order to yield an efficient order of variables, the variances of the variables should be approximately equal because with we have. Thus, approximately equal variances are important for a high power of the procedure.

10. 11.11.2002 Leipzig10 That alone would, however, only ensure the multiple error rate under the global hypothesis (regardless of variances). Additionally, we have special criteria for ordering and special tests – the non-null variables do not confuse the behaviour of the null variables, only conservative influence. Proof that the procedure keeps the FWE (draft): The univariate t tests with the single variables are considered as special cases of the stabilized multivariate tests with scores z j = d´x (j). The weight vectors are

11. 11.11.2002 Leipzig11 Example I: Comparison nodules vs. surrounding (3 hot and 3 cold nodules together  one-sample test vs. 0) # locally sign. genes: 33 # sign. genes Westfall-Young: 0 # sign. genes Holm‘s proc.: 0 # sign. genes Procedure I: 10 Quadrant A only (98 genes, 2 spots aver., corrected with housek. genes) gene no. sum of squares unadjusted P-value 56 0.6020 0.0017 71 0.5481 0.0390 1 0.5334 0.0012 78 0.4551 0.0075 88 0.3629 0.0363 29 0.3573 0.0036 19 0.3456 0.0263 10 0.3454 0.0043 57 0.3451 0.0092 54 0.3430 0.0460 31 0.3052 0.0594 8 0.3023 0.0462.................

12. 11.11.2002 Leipzig12 Example II: 15 cold nodules vs. surrounding (one-sample problem) The present procedure stops already after the 2nd gene. The basic trend for sums of squares is present, but the procedure is sensitive to disturbances. It should be smoothed (see below, hybridisation with Bonferroni / Holm).... 0.2169199 5 · 10 -5 81358 1 · 10 -6 81047 0.0465186 0.3462575 5 · 10 -3 35684 0.388483 2 · 10 -4 65672 2 · 10 -4 67461 P valuegeneno. For comparison: without any adjustment: 1064 Bonferroni /Holm: 1 (gene 8104) Westfall / Young: 0

13. 11.11.2002 Leipzig13 Simulation experiments guided by example I: n = 6,...,33 cases, p = 98 variables, normally distributed, variance 1, pairwise correlation 0.5, expectation 0 for 88 var‘s, other 10 var‘s: sample size n Average # of significant genes in Monte Carlo replications Simulation experiments

14. 11.11.2002 Leipzig14 Extensions: Other test problems: –particularly comparison of two/more independent samples ordering by sums of squares, i.e., related to the variablewise total mean of all samples, then two-sample t tests or one-way ANOVA. Other subsets of variables (e.g., pairs of variables)  Kropf, Läuter (2002) „Distribution-free“ version possible

15. 11.11.2002 Leipzig15 Weighted procedure (Procedure II) In notation of the one-sample problem (Westfall, Kropf, Finos, 2002) Calculate the P-values p i (i = 1, …, p) for the usual unadjusted one-sample t test for each of the p variables. For each variable, determine the sums of squares values and the weights for fixed   0. Calculate the weighted P-values q i = p i / g i and order the variables for increasing values of them. Then the hypothesis H (j) for the jth ordered variable is rejected iff S i : ith ordered variable and all subsequent ones.

16. 11.11.2002 Leipzig16 How does this procedure fit to the others above? Procedure II utilises ideas from Bonferroni/Holm (fixed weights) as well as from Procedure II (data-driven through w ii ).  = 0, g i = w ii 0 = 1 : Then the procedure is identical to usual unweighted Bonferroni / Holm.   : According to Westfall and Krishen (2001), the influence of the weights totally rules out the P-values from Bonferroni- Holm, critical function converges to that of Procedure I. Intermediate values of  : both parts are present, „power- assump- tion“ of equal variances only important for part of Procedure I. In an application,  has to be fixed in advance!

17. 11.11.2002 Leipzig17 Example 2 again Cold nodules vs. surrounding unadjusted 1064, Westf./Y. 0.25 1 4 16 64  Is the choice of genes stable? B/H Pr. I 

18. 11.11.2002 Leipzig18 Example 2, cont. hot nodules vs. surrounding.25 1 4 16 64  hot vs. cold nodules B/H Pr. I B/H Pr. I unadjusted 2597, Westf./Y. 93unadjusted 1290, Westf./Y. 2.25 1 4 16 64 

19. 11.11.2002 Leipzig19 Simulation experiments with weighted procedure guided by example II p = 12.625 variables, n = 4, 6, 8, 12, 16, 20, 30, 50, 100 number of significant genes 10, 100, 1000 pairwise correlation coefficient 0 or 0.5 heterogeneity of variances in 5 levels influence of pairwise correlation on optimal choice of  small, also number of significant genes not so important sample size is influential (and known in practice) heterogeneity of variances is important, too, but not known in practice; estimation through w ii ensures only weak FWE control.

20. 11.11.2002 Leipzig20 Summary A new technique for multiple testing with data-dependent ordering of hypotheses is proposed. It keeps the FWE in the strong sense for arbitrary multivariate normal data. In order to provide a high power, the variables should have approximately equal variances. The proposal is advantageous in very small samples of high-dimensional data. The method is sensitive to disturbances. Westfall‘s proposal of the weighted procedure establishes a link of the above procedure and the Bonferroni-Holm method and smoothes out for these disturbances. The weighted procedure is a real alternative to existing analysis techni- ques for microarray data, problem of suitable choice of .

21. 11.11.2002 Leipzig21 References Fang, K.-T. and Zhang, Y.-T., 1990: General Multivariate Analysis. Science Press Beijing and Springer-Verlag Berlin Heidelberg. Kropf, S., 2000: Hochdimensionale multivariate Verfahren in der medizinischen Statistik. Shaker Verlag, Aachen. Kropf, S., and Läuter, J., 2002: Multiple Tests for Different Sets of Variables Using a Data- Driven Ordering of Hypotheses, with an Application to Gene Expression Data. Biometrical Journal 44, no. 7. Läuter J., 1996: Exact t and F Tests for Analysing Studies with Multiple End­points. Biometrics 52, 964-970. Läuter, J., Glimm, E., and Kropf, S., 1998: Multivariate Tests Based on Left-Spherically Distributed Linear Scores. Annals of Statistics 26, 1972-1988. Erratum: Annals of Statistics 27, 1441. Westfall, P.H., Kropf, S., and Finos, L., 2002: Weighted FWE-controlling methods in high- dimensional situations. Submitted for IMS Philadelphia companion volume. Westfall, P.H. and Krishen, A. (2001): Optimally weighted, fixed sequence, and gatekeeping multiple testing procedures. Journal of Statistical Planning and Inference 99, 25-40. Westfall, P.H. and Young, S.S., 1993: Resampling-Based Multiple Testing. John Wiley & Sons, New York.