1. Significance Tests P-values and Q-values

2. Outline Statistical significance in multiple testing Statistical significance in multiple testing Empirical distribution of test statistics Empirical distribution of test statistics Family-wide p-values Family-wide p-values Correlation and p-values Correlation and p-values False discovery rates False discovery rates

3. Tests and Test Statistics T-test is fairly robust to skew, but not robust to outliers – “thick tails” of distribution T-test is fairly robust to skew, but not robust to outliers – “thick tails” of distribution Non-parametric tests are robust, but lose too much ability to detect differences (power) Non-parametric tests are robust, but lose too much ability to detect differences (power) Robust tests can be useful Robust tests can be useful Permutation tests are simple and easy to program Permutation tests are simple and easy to program Some authors use: Some authors use: rather than To reduce numbers of low fold-changes in highly signficant scores

4. Distribution of test statistics Quantile plots of t-statistics: left: random distn; right: experiment

5. Distribution of Set of p-values

6. Multiple comparisons Suppose 10,000 genes on a chip Suppose 10,000 genes on a chip None actually differentially expressed None actually differentially expressed Each gene has a 5% chance of exceeding the threshold score for a p-value of.05 Each gene has a 5% chance of exceeding the threshold score for a p-value of.05 Type I error definition Type I error definition On average, 500 genes should exceed.05 threshold ‘by chance’ On average, 500 genes should exceed.05 threshold ‘by chance’

7. Family-Wide Error Rate ‘Corrected’ p-value: ‘Corrected’ p-value: Probability of finding a single false positive among all N tests Probability of finding a single false positive among all N tests Normally all tests at same threshold Normally all tests at same threshold Simplest correction (Bonferroni) Simplest correction (Bonferroni) p i * = Np i, (if Np i < 1, otherwise 1) p i * = Np i, (if Np i < 1, otherwise 1) Fairly close to true false positive rate in simulations of independent tests Fairly close to true false positive rate in simulations of independent tests Too conservative in practice! Too conservative in practice!

8. P-Values from Correlated Genes.5.3.9.7.03.1.4.9.05.6.8.4.2.9 Null distribution from independent genes.5.3.9.5.3.9.5.3.9.5.3.9.5.3.9 Null distribution from perfectly correlated genes Rows: genes; columns: samples; entries: p-values from randomized distribution.5.3.9.45.2.95.65.25.8.4.35.75.5.4.85 Null distribution from highly correlated genes

9. The Effect of Correlation If all genes are uncorrelated, Sidak is exact If all genes are uncorrelated, Sidak is exact If all genes were perfectly correlated If all genes were perfectly correlated p-values for one are p-values for all p-values for one are p-values for all No multiple-comparisons correction needed No multiple-comparisons correction needed Typical gene data is highly correlated Typical gene data is highly correlated First eigenvalue of SVD may be more than half the variance First eigenvalue of SVD may be more than half the variance More sensitive tests possible if we can generate joint null distribution of p-values More sensitive tests possible if we can generate joint null distribution of p-values

10. Re-formulating the Question Independent: ~5% of genes exceed.05 threshold, all the time Independent: ~5% of genes exceed.05 threshold, all the time Perfectly Correlated: all genes exceed.05 threshold ~5% of the time Perfectly Correlated: all genes exceed.05 threshold ~5% of the time Realistically correlated:.05 < f 1 < 1 of genes exceeds.05 threshold,.05 < f 2 < 1 of the cases Realistically correlated:.05 < f 1 < 1 of genes exceeds.05 threshold,.05 < f 2 < 1 of the cases New question: for a given f 1 and , how likely is it that a fraction f 1 of genes will exceed the  threshold? New question: for a given f 1 and , how likely is it that a fraction f 1 of genes will exceed the  threshold?

11. Step-Down p-Values Calculate single-step p-values for genes: p 1, …, p N Calculate single-step p-values for genes: p 1, …, p N Order the smallest k p-values: p (1), …, p (k) Order the smallest k p-values: p (1), …, p (k) For each k, ask: For each k, ask: How likely are we to get k p-values less than p (k) if no differences are real? How likely are we to get k p-values less than p (k) if no differences are real? Generate null distribution by permutations Generate null distribution by permutations More significant genes, at the same level of Type I error, compared with single-step procedures More significant genes, at the same level of Type I error, compared with single-step procedures See Ge, et al, Test, 2003 See Ge, et al, Test, 2003 Bioconductor package multtest Bioconductor package multtest

12. False Discovery Rate At threshold t* what fraction of genes are likely to be true positives? At threshold t* what fraction of genes are likely to be true positives? Illustration: 10,000 independent genes Illustration: 10,000 independent genes tp#sigE(FP)FDR* 1.96.0560050087% 2.57.0120010050% 3.29.001401020% In practice use permutation algorithm to compute FDR

13. pFDR How to estimate the FDR? How to estimate the FDR? ‘positive’ False Discovery Rate: ‘positive’ False Discovery Rate: E(#false positives/#positives) * P(#positives >0) E(#false positives/#positives) * P(#positives >0) Simes’ inequality allows this to be computed from p-values Simes’ inequality allows this to be computed from p-values