1 A Discussion of False Discovery Rate and the Identification of Differentially Expressed Gene Categories in Microarray Studies Ames, Iowa August 8, 2007 Dan Nettleton Iowa State University

2 Myostatin Knockout Mice vs. Wild Type Belgian Blue cattle have a mutation in the myostatin gene.

3 Affymetrix GeneChips on 5 Mice per Genotype WT M M M M M

4 The Dataset Gene ID Wild Type Mutant p 1 p 2 p 3 p 4 p 5 p 6 p 7 p 8 p 9 p 10 p 11 p 12 p p p-value

5 A Standard Analysis Two-sample t-tests for each gene. Compute p-values by comparing t-statistics to a t-distribution with 8 d.f. Use an adjustment for multiple testing to create a list of genes declared to be differentially expressed.

6 Histogram of p-values from the Two-Sample t-Tests p-value Number of Genes

7 Example p-value Distributions Two-Sample t-test of H 0 :μ 1 =μ 2 n 1 =n 2 =5, variance=1 μ 1 -μ 2 =1 μ 1 -μ 2 =0.5 μ 1 -μ 2 =0

8 Histogram of p-values from the Two-Sample t-Tests p-value Number of Genes

9 False Discovery Rate (FDR) FDR is an error measure that can be useful for multiple testing problems encountered in microarray experiments. FDR was introduced by Benjamini and Hochberg (1995) and is formally defined as follows: R = # rejected null hypotheses when conducting m tests V = # of type I errors (false discoveries) FDR=E(Q) where Q=V/R if R>0 and Q=0 otherwise.

10 A Conceptual Description of FDR Suppose a scientist conducts many independent microarray experiments. For each experiment, the scientist uses a method for producing a list of genes declared to be differentially expressed. For each list consider the ratio of the number of false positive results to the total number of genes on the list (set this ratio to 0 if the list contains no genes). The FDR is approximated by the average of the ratios described above.

11 Some of the gene lists may contain a high proportion of false positive results and yet the method used may still control FDR at a given level. It is the average performance across repeated experiments that matters. A Conceptual Description of FDR (continued)

12 Number of Genes Declared to be Differentially Expressed for Various Estimated FDR Levels FDRNumber of Genes P-Value Threshold FDR estimated using the method of Storey and Tibshirani (2003).

13 Using Information about Genes to Interpret the Results of Microarray Experiments Based on a large body of past research, some information is known about many of the genes represented on a microarray. The information might include tissues in which a gene is known to be expressed, the biological process in which a gene’s protein is known to act, or other general or quite specific details about the function of the protein produced by a gene. By examining this information in concert with the results of a microarray experiment, biologists can often gain a greater understanding of their microarray experiments.

14 Gene Ontology (GO) Terms GO terms provide one example of information that is available about genes. The GO project provides three ontologies (structured controlled vocabularies) that describe a gene’s 1. Biological Processes, 2. Cellular Components, and 3. Molecular Functions.

15 Gene Ontology (GO) Terms Each gene may be associated with 0 or more GO terms in a given ontology. The GO terms in each ontology have varying levels of specificity. The GO terms in each ontology can be organized in a directed acyclic graph (DAG) where each node represents a term and arrows point from specific terms to more general terms.

16 Portion of the Biological Processes Ontology Shown in a DAG Biological Process Cellular Process Metabolic Process Cellular Metabolic Process Primary Metabolic Process Macromolecule Metabolic Process Carbohydrate Metabolic Process Generation of Precursor Metabolites and Energy Energy Derivation by Oxidation of Organic Compounds Alcohol Metabolic Process

17 Constructing Gene Categories from GO Terms The set of genes associated with any particular GO term could be considered as a category or gene set of interest for subsequent testing. For example, we might ask if genes that are associated with the Molecular Function term muscle alpha-actinin binding are affected by a treatment of interest. We could simultaneously query many groups, general and specific, to better understand the impact of treatment on expression.

18 Simultaneous Testing of Multiple Categories with Various Levels of Specificity molecular function binding protein binding actinin binding cytoskeletal protein binding alpha-actinin binding muscle alpha-actinin binding enzyme binding ATPase binding RNA polymerase core enzyme binding myosin binding beta-actinin binding

19 Some Formal Methods for Testing Gene Categories with Microarray Data Fisher’s exact test on lists of gene declared to be differentially expressed (DDE) Gene Set Enrichment Analysis (GSEA) Significance Analysis of Function and Expression (SAFE) Pathway Level Analysis of Gene Expression (PLAGE) Domain Enhanced Analysis (DEA) Many others appearing and soon to appear

20 Number of Genes Declared to be Differentially Expressed for Various Estimated FDR Levels FDRNumber of Genes P-Value Threshold FDR estimated using the method of Storey and Tibshirani (2003).

21 Are genes of category X overrepresented among the genes declared to be differentially expressed? Declared to be Differentially Expressed? yes no Gene of Category X? yes no Highly significant overrepresentation according to a chi-square test or Fisher’s exact test.

22 Problems with Chi-Square or Fisher’s Exact Test for Detecting Overrepresentation The outcome of the overrepresentation test depends on the significance threshold used to declare genes differentially expressed. Functional categories in which many genes exhibit small changes may go undetected. Genes are not independent, so a key assumption of the chi-square and Fisher’s exact tests is violated. Information in the multivariate distribution of genes in a category is not utilized.

23 Advantage of a Multivariate Approach Gene 1 Expression Gene 2 Expression

24 Multiresponse Permutation Procedure (MRPP) Mielke and Berry, (2001). Permutation Methods: A Distance Function Approach. Springer, N.Y. Nonparametric test for a difference among multivariate distributions Test statistic based on within-group inter-point distances P-value obtained by data permutation

25 The MRPP Test: An Illustrative Example Expression Measure for Gene 1 Expression Measure for Gene 2 For balanced data, test statistic is sum of within-group inter-point distances.

26 The test statistic is computed for all permutations of the data. Expression Measure for Gene 1 Expression Measure for Gene 2

27 An Example Permutation Expression Measure for Gene 1 Expression Measure for Gene 2

28 Test statistic will be larger for permutations than for the original data Expression Measure for Gene 1 Expression Measure for Gene 2 Permutation p-value = # ( T original ≥ T permutation ) = 2 # permutations 252

29 Portion of Directed Acyclic Graph of GO Molecular Function Terms