1. Multiple Testing Methods for the Analysis of Microarray Data
2/21/2008
Copyright © 2008 Dan Nettleton

2. Wild-type vs. Myostatin Knockout Mice
Belgian Blue
cattle have a
mutation in the
myostatin gene.

3. Affymetrix GeneChips on 5 Mice per GenotypeWT M WT WT M WT M WT M

4. The Dataset . . . . . . . . . . . Gene ID Wild Type Mutant 1 4835.8
4578.2
4856.3
4483.7
4275.3
4170.7
3836.9
3901.8
4218.4
4094.0 2
153.9
161.0
139.7
173.0
160.1
180.1
265.1
201.2
130.8
130.7 3
3546.5
3622.7
3364.3
3433.6
2757.2
3346.9
2723.8
2892.0
3021.3
2452.7 4
711.3
717.3
776.6
787.5
750.3
910.2
813.3
687.9
811.1
695.6 5
126.3
178.2
114.5
158.7
157.3
231.7
147.0
102.8
157.6
146.8 6
4161.8
4622.9
3795.7
4501.2
4265.8
3931.3
3327.6
3726.7
4003.0
3906.8 7
419.3
555.3
509.6
515.5
488.9
426.6
425.8
500.8
347.8
580.3 8
2420.7
2616.1
2768.7
2663.7
2264.6
2379.7
2196.2
2491.3
2710.0
2759.1 9
321.5
540.6
471.9
348.2
356.6
382.5
375.9
481.5
260.6
515.7 10
1061.4
949.4
1236.8
1034.7
976.8
1059.8
903.6
1060.3
960.1
1134.5 11
1293.3
1147.7
1173.8
1173.9
1274.2
1062.8
1172.1
1113.0
1432.1
1012.4 12
336.1
413.5
425.2
462.8
412.2
391.7
388.1
363.7
310.8
404.6 13
5718.1
4105.5
5620.9
6786.8
7823.0
1297.8
1303.8
1318.8
1189.2
1171.5
22690
249.6
283.6
271.0
246.9
252.7
214.2
217.9
266.6
193.7
413.2 . . . . . . . . . . .

5. A Standard Analysis Two-sample t-test for each gene.
Test the null hypothesis for the ith gene (wild type mean = mutant mean)
Compute p-values by comparing t-statistics to a t-distribution with 8 d.f.
Use an adjustment for multiple testing to create a list of genes declared to be differentially expressed.

6. The Dataset . . . . . . . . . . . . . . Gene ID Wild Type Mutant
p-value p1 p2 p3 p4 p5 p6 p7 p8 p9
p10
p11
p12
p13 1
4835.8
4578.2
4856.3
4483.7
4275.3
4170.7
3836.9
3901.8
4218.4
4094.0 2
153.9
161.0
139.7
173.0
160.1
180.1
265.1
201.2
130.8
130.7 3
3546.5
3622.7
3364.3
3433.6
2757.2
3346.9
2723.8
2892.0
3021.3
2452.7 4
711.3
717.3
776.6
787.5
750.3
910.2
813.3
687.9
811.1
695.6 5
126.3
178.2
114.5
158.7
157.3
231.7
147.0
102.8
157.6
146.8 6
4161.8
4622.9
3795.7
4501.2
4265.8
3931.3
3327.6
3726.7
4003.0
3906.8 7
419.3
555.3
509.6
515.5
488.9
426.6
425.8
500.8
347.8
580.3 8
2420.7
2616.1
2768.7
2663.7
2264.6
2379.7
2196.2
2491.3
2710.0
2759.1 9
321.5
540.6
471.9
348.2
356.6
382.5
375.9
481.5
260.6
515.7 10
1061.4
949.4
1236.8
1034.7
976.8
1059.8
903.6
1060.3
960.1
1134.5 11
1293.3
1147.7
1173.8
1173.9
1274.2
1062.8
1172.1
1113.0
1432.1
1012.4 12
336.1
413.5
425.2
462.8
412.2
391.7
388.1
363.7
310.8
404.6 13
5718.1
4105.5
5620.9
6786.8
7823.0
1297.8
1303.8
1318.8
1189.2
1171.5
22690
249.6
283.6
271.0
246.9
252.7
214.2
217.9
266.6
193.7
413.2 . . . . . . . . . . . . . .
p22690

7. Histogram of p-values from the Two-Sample t-Tests
Number of Genes
p-value

8. Example p-value Distributions
Two-Sample t-test of H0:μ1=μ2
n1=n2=5, variance=1
μ1-μ2=1
μ1-μ2=0.5
μ1-μ2=0

9. Histogram of p-values from the Two-Sample t-Tests
Number of Genes
p-value

10. The Multiple Testing Problem
Suppose one test of interest has been conducted for each of m genes in a microarray experiment.
Let p1, p2, ... , pm denote the p-values corresponding to the m tests.
Let H01, H02, ... , H0m denote the null hypotheses corresponding to the m tests.

11. The Multiple Testing Problem (continued)
Suppose m0 of the null hypotheses are true and m1 of the null hypotheses are false.
Let c denote a value between 0 and 1 that will serve as a cutoff for significance:
- Reject H0i if pi ≤ c (declare significant)
- Fail to reject (or accept) H0i if pi > c
(declare non-significant)

12. Table of Outcomes Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls U V m0
False Nulls T S m1
Total W R m

13. Table of Outcomes U=number of true negatives Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls U V m0
False Nulls T S m1
Total W R m
U=number of true negatives

14. Table of Outcomes V=number of false positives
Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls U V m0
False Nulls T S m1
Total W R m
V=number of false positives
=number of false discoveries
=number of type 1 errors

15. Table of Outcomes T=number of type 2 errors Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls U V m0
False Nulls T S m1
Total W R m
T=number of type 2 errors

16. Table of Outcomes S=number of true positives
Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls U V m0
False Nulls T S m1
Total W R m
S=number of true positives
=number of true discoveries

17. Table of Outcomes W=number of non-rejections
Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls U V m0
False Nulls T S m1
Total W R m
W=number of non-rejections
(number of null hypotheses accepted)

18. Table of Outcomes R=number of rejections (of null hypotheses)
Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls U V m0
False Nulls T S m1
Total W R m
R=number of rejections
(of null hypotheses)

19. Table of Outcomes Random Variables Constants Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls U V m0
False Nulls T S m1
Total W R m
Random Variables
Constants

20. Table of Outcomes Unobservable Observable Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls U V m0
False Nulls T S m1
Total W R m
Unobservable
Observable

21. Familywise Error Rate (FWER)
Traditionally statisticians have focused on controlling FWER when conducting multiple tests.
FWER is defined as the probability of one or more false positive results:
FWER=P(V>0).
Controlling FWER amounts to choosing the significance cutoff c so that FWER is less than or equal to some desired level α.

22. The Bonferroni Method
The Bonferroni Method is the simplest way to achieve control of the FWER at any desired level α.
Simply choose c = α / m.
With this value of c, the FWER will be no larger than α for any family of m tests.

23. A Conceptual Description of FWER
Suppose a scientist conducts 100 independent microarray experiments.
For each experiment, the scientist produces a list of genes declared to be differentially expressed by testing a null hypothesis for each gene.
Each list that contains one or more false positive results is considered to be in error.
The FWER is approximated by the proportion of 100 lists that contain one or more false positives.

24. FWER Too Conservative for Microarrays?
Suppose that one of the 100 gene lists consists of 500 genes declared to be differentially expressed.
Suppose that 1 of those 500 genes is not truly differentially expressed but that the other 499 are.
This list is considered to be in error and such lists are allowed to make up only a small proportion of the total number of lists if FWER is to be controlled.
However such a list seems quite useful from the scientific viewpoint. Perhaps it is not so important to control FWER for most microarray experiments.

25. False Discovery Rate (FDR)
FDR is an error measure that can be useful for multiple testing problems encountered in microarray experiments.
FDR was introduced by Benjamini and Hochberg (1995) and is formally defined as follows:
R = # rejected null hypotheses when conducting m tests
V = # of type I errors (false discoveries)
FDR=E(Q) where Q=V/R if R>0 and Q=0 otherwise.
Controlling FDR amounts to choosing the significance cutoff c so that FDR is less than or equal to some desired level α.

26. A Conceptual Description of FDR
Suppose a scientist conducts 100 independent microarray experiments.
For each experiment, the scientist produces a list of genes declared to be differentially expressed by testing a null hypothesis for each gene.
For each list consider the ratio of the number of false positive results to the total number of genes on the list (set this ratio to 0 if the list contains no genes).
The FDR is approximated by the average of the ratios described above.

27. FDR: The Appropriate Error Rate for Microarrays?
The hypothetical gene list discussed previously with 1 false positive and 499 true positives would be a good list that would help to keep the FDR down.
Some of the gene lists may contain a high proportion of false positive results and yet the method we are using may still control FDR at a given level because it is the average performance across repeated experiments that matters.
The comment above applies to FWER control as well.

28. The Benjamini and Hochberg Procedure for Strongly Controlling FDR at Level α
Let p(1), p(2), ... , p(m) denote the m p-values ordered from smallest to largest.
Find the largest integer k so that p(k) ≤ k α / m.
If no such k exists, set c = 0 (declare nothing significant).
Otherwise set c = p(k) (reject the nulls corresponding to the smallest k p-values).

29. An Example
Suppose 10,000 genes are tested for differential expression between two treatments.
Suppose 200th smallest p-value is
If no genes were truly differentially expressed, how many of the 10,000 p-values would be expected to be less than or equal to 0.001?
Use the calculations above to provide an estimate of the proportion of false positive results among the list of 200 genes with p-values no larger than

30. Solution
If all 10,000 null hypotheses were true, we would expect *10,000 = 10 tests to yield p-values less than 0.001
A simple estimate of the proportion of false positive results among the list of 200 genes with p-values less than is * 10,000 / 200 = 0.05.
Recall that the B&H FDR procedure involves finding the largest integer k so that p(k) ≤ k α / m.
This is equivalent to finding the largest integer k such that p(k) m / k ≤ α.

31. Other Methods for Estimating or Controlling FDR
Rather than finding the largest integer k such that
p(k) m / k ≤ α,
consider finding the largest integer k such that
p(k) m0 / k ≤ α,
where m0 is an estimate of the number of true null hypotheses among the m tests. ^ ^

32. Histogram of p-values for a Test of Interest
Number of Genes
p-value

33. Distribution stochastically smaller than uniform for tests
Mixture of a Uniform Distribution and a Distribution Stochastically Smaller than Uniform
Distribution stochastically
smaller than uniform for tests
with false nulls
Number of Genes
Uniform
distribution
for tests with
true nulls
p-value

34. Estimating FDR Using Estimates of m0
Benjamini Y. and Hochberg Y. (2000). On the adaptive control of the false discovery rate in multiple testing with independent statistics. Journal of Educational and Behavioral Statistics 25,
Mosig, M. O., Lipkin, E., Galina, K. Tchourzyna, E., Soller, M., and Friedmann, A. (2001). A whole genome scan for quantitative trait loci affecting milk protein percentage in Israeli-Holstein cattle, by means of selective milk DNA pooling in a daughter design, using an adjusted false discovery rate criterion. Genetics, 157,
Storey, J. D., and Tibshirani, R. (2001). Estimating false discovery rates under dependence, with applications to DNA microarrays. Technical Report , Department of Statistics, Stanford University.
Storey J. D. (2002). A direct approach to false discovery rates. Journal of the Royal Statistical Society, Series B, 64,

35. Estimating FDR Using Estimates of m0
Genovese, C. and Wasserman, L. (2002). Operating characteristics and extensions of the false discovery rate procedure, Journal of the Royal Statistical Society, Series B, 64,
Storey J. D. (2003). The positive false discovery rate: A Bayesian interpretation and the q-value. Annals of Statistics, 31,
Storey, J. D., and Tibshirani, R. (2003). Statistical significance for genomewide studies. Proceedings of the National Academy of Sciences 100,
Storey J. D., Taylor JE, and Siegmund D. (2004). Strong control, conservative point estimation, and simultaneous conservative consistency of false discovery rates: A unified approach. Journal of the Royal Statistical Society, Series B, 66,

36. Estimating FDR Using Estimates of m0
Fernando, R. L., Nettleton, D., Southey, B. R., Dekkers, J. C. M., Rothschild, M. F., and Soller, M. (2004). Controlling the proportion of false positives (PFP) in multiple dependent tests. Genetics. 166,
Genovese, C. and Wasserman, L. (2004). A stochastic process approach to false discovery control. The Annals of Statistics, 32,
Nettleton, D., Hwang, J.T.G., Caldo, R.A., Wise, R.P. (2006). Estimating the number of true null hypotheses from a histogram of p-values. Journal of Agricultural, Biological, and Environmental Statistics

37. Estimating FDR Using Estimates of m0
Ruppert, D., Nettleton, D., Hwang, J.T.G. (2007). Exploring the information in p-values for the analysis and planning of multiple-test experiments. Biometrics
Plus many more....

38. A method for obtaining a list of genes that has an estimated FDR ≤ α
Find the largest integer k such that
p(k) m0 / k ≤ α,
where m0 is an estimate of the number of true null hypotheses among the m tests.
2. If no such k exists, declare nothing significant. Otherwise, reject the null hypotheses corresponding to the smallest k p-values. ^ ^

39. q-values
Recall that a p-value for an individual test can be defined as the smallest significance level (tolerable type 1 error rate) for which we can reject the null the hypothesis.
The q-value for one test in a family of tests is the smallest FDR for which we can reject the null hypothesis for that one test and all others with smaller p-values.

40. The q-value for a given test fills the blanks in the following sentences:
“If I set my cutoff for significance c equal to this p-value, I must be willing to accept a false discovery rate of ______.”
“To reject the null hypothesis for this test and all others with smaller p-values, I must be willing to accept a false discovery rate of _______.”
“To include this gene on my list of differentially expressed genes, I must be willing to accept a false discovery rate of _____.”

41. Computation and Use of q-values
Let q(i) denote the q-value that corresponds to the ith smallest p-value p(i).
q(i) = min { p(k) m0 / k : k = i,...,m }.
To produce a list of genes with estimated FDR ≤ α, include all genes with q-values ≤ α. ^

42. We will convert these p-values to q-values
using the method of Storey and Tibshirani.
Number of Genes
p-value

43. p-values q-values If we want FDR to be 1%, we can declare only one
gene differentially expressed.

44. p-values q-values If we want FDR to be 5%, we can declare only one
gene differentially expressed.

45. p-values q-values If we want FDR to be 6%, we can declare 17 genes
If we want FDR to be 6%,
we can declare 17 genes
differentially expressed.

46. Histogram of Simulated p-values
Number of Genes
p-value

47. Control of FWER at 0.05 Using the Bonferroni Method
Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls
False Nulls
Total

48. Control of FDR at 0.05 Using the B&H Method
Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls
False Nulls
Total

49. Control of FDR at 0.05 Using the Storey and Tibshirani Method
Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls
False Nulls
Total

50. Control of FWER at 0.10 Using the Bonferroni Method
Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls
False Nulls
Total

51. Control of FDR at 0.10 Using the B&H Method
Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls
False Nulls
Total

52. Control of FDR at 0.10 Using the Storey and Tibshirani Method
Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls
False Nulls
Total

53. Control of FWER at 0.20 Using the Bonferroni Method
Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls
False Nulls
Total

54. Control of FWER at 0.20 Using Holm’s Method
Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls
False Nulls
Total

55. Control of FDR at 0.20 Using the B&H Method
Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls
False Nulls
Total

56. Control of FDR at 0.20 Using the Storey and Tibshirani Method
Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls
False Nulls
Total

57. Note that all our methods missed many differentially expressed genes
Methods of estimating m0 suggest that around or 1500 genes are differentially expressed.
In this case 1500 genes are truly differentially expressed.
What if we declare the 1500 genes with the smallest p-values to be differentially expressed?

58. Rejecting the Null Hypotheses for the Tests with the 1500 Smallest p-values
Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls
False Nulls
Total

59. Rejecting the null hypothesis for the 1500 tests with the smallest p-values will yield a high FDR
True
Positives V R
= = 35.7%
Number of Genes
False
Positives c
p-value

60. Can we get a list of genes that contains all 1500 of the truly differentially expressed genes?
It turns out that we would have to include genes on our list to get all 1500 truly differentially expressed genes in this case.
The list would have all the truly differentially expressed genes, but about 85% of the genes on the list would be false positives.

61. Performance of the Shortest List that Contains All Truly Differentially Expressed Genes
Accept Null Reject Null
Declare Non-Sig. Declare Sig.
No Discovery Declare Discovery
Negative Result Positive Result
True Nulls
False Nulls
Total

62. Concluding Remarks
In many cases, it will be difficult to separate the many of the differentially expressed genes from the non- differentially expressed genes.
Genes with a small expression change relative to their variation will have a p-value distribution that is not far from uniform if the number of experimental units per treatment is low.
To do a better job of separating the differentially expressed genes from the non-differentially expressed genes we need to use good experimental designs with more replications per treatment.

63. Concluding Remarks (continued)
We have looked at only one simulated example.
The behavior of the methods will vary from data set to data set.
Simulations suggest that the methods control their error rates at nominal levels for a variety of practical situations.

64. Using Information about Genes to Interpret the Results of Microarray Experiments
Based on a large body of past research, some information is known about many of the genes represented on a microarray.
The information might include tissues in which a gene is known to be expressed, the biological process in which a gene’s protein is known to act, or other general or quite specific details about the function of the protein produced by a gene.
By examining this information in concert with the results of a microarray experiment, biologists can often gain a greater understanding of their microarray experiments.

65. Gene Ontology (GO) Terms
GO terms provide one example of information that is available about genes.
The GO project provides three ontologies (structured controlled vocabularies) that describe a gene’s
1. Biological Processes,
2. Cellular Components, and
3. Molecular Functions.

66. Gene Ontology (GO) Terms
Each gene may be associated with 0 or more GO terms in a given ontology.
The GO terms in each ontology have varying levels of specificity.
The GO terms in each ontology can be organized in a directed acyclic graph (DAG) where each node represents a term and arrows point from specific terms to more general terms.

67. Portion of the Biological Processes Ontology Shown in a DAG
Alcohol Metabolic
Process
Energy Derivation by Oxidation
of Organic Compounds
Generation of Precursor
Metabolites and Energy
Carbohydrate Metabolic
Process
Cellular Metabolic
Process
Macromolecule
Metabolic Process
Primary Metabolic
Process
Cellular
Process
Metabolic
Process
Biological
Process

68. Constructing Gene Categories from GO Terms
The set of genes associated with any particular GO term could be considered as a category or gene set of interest for subsequent testing.
For example, we might ask if genes that are associated with the Molecular Function term muscle alpha-actinin binding are affected by a treatment of interest.
We could simultaneously query many groups, general and specific, to better understand the impact of treatment on expression.

69. Simultaneous Testing of Multiple Categories with Various Levels of Specificity
muscle alpha-actinin
binding
alpha-actinin
binding
beta-actinin
binding
actinin
binding
myosin
binding
ATPase
binding
RNA polymerase
core enzyme binding
cytoskeletal protein
binding
enzyme
binding
protein
binding
binding
molecular
function

70. Some Formal Methods for Testing Gene Categories with Microarray Data
Fisher’s exact test on lists of gene declared to be differentially expressed (DDE)
Gene Set Enrichment Analysis (GSEA)
Significance Analysis of Function and Expression (SAFE)
Pathway Level Analysis of Gene Expression (PLAGE)
Domain Enhanced Analysis (DEA)
Many others appearing and soon to appear

71. Number of Genes Declared to be Differentially Expressed for Various Estimated FDR Levels
FDR Number of Genes P-Value Threshold
FDR estimated using the method
of Storey and Tibshirani (2003).

72. Are genes of category X overrepresented among the genes declared to be differentially expressed?
Gene of
Category X?
yes no
yes
Declared to be
Differentially
Expressed? no
Highly significant overrepresentation according
to a chi-square test or Fisher’s exact test.

73. Problems with Chi-Square or Fisher’s Exact Test for Detecting Overrepresentation
The outcome of the overrepresentation test depends on the significance threshold used to declare genes differentially expressed.
Functional categories in which many genes exhibit small changes may go undetected.
Genes are not independent, so a key assumption of the chi-square and Fisher’s exact tests is violated.
Information in the multivariate distribution of genes in a category is not utilized.

74. Advantage of a Multivariate Approach
Gene 2 Expression
Gene 2 Expression
Gene 1 Expression
Gene 1 Expression

75. A multivariate approach to the gene category testing problem is described in
Nettleton, D., Recknor, J., Reecy, J.M. (2007).
Identification of differentially expressed gene
categories in microarray studies using
nonparametric multivariate analysis.
Bioinformatics