1. 1 השוואות מרובות מדדי טעות, עוצמה, רווחי סמך סימולטניים ד"ר מרינה בוגומולוב מבוסס על ההרצאות של פרופ' יואב בנימיני ופרופ' מלכה גורפיין

2. 2 Family: Any collection of inferences for which it is meaningful to take into account some combined measure of errors (Hochberg and Tamhane, 1987). Family: The smallest set of items of inference from which selection of results for presentation or highlighting was made (Prof. Yoav Benjamini). It follows from the definition that a family must consist of hypotheses related in terms of their intended use by the researcher, otherwise it would make no sense to select the hypotheses within that family. It is upon the researcher to decide what is (are) the set (sets) of related hypotheses from which the interesting finding/s may/will be selected. Once the decision is made, the interpretation of the results must be made accordingly. The definition of families should depend on the desired interpretation of the results. For a single data set, the families may be defined in various ways leading to different interpretation of the results, each of which is important form some point of view. Family of hypotheses

3. תזכורת ממצגת 1 … 448293 … … 31622 … … … … … Y. Benjamini and M.Bogomolov “Selective inference on multiple families hypotheses” JRSS-B

4. For a given family of hypotheses, let m be the number of hypotheses within the family. 4 Multiple comparison procedure (MCP): statistical test procedure designed to account for and properly control the multiplicity effect through some combined measure of erroneous inferences. HypothesesNot rejectedRejectedTotal TrueUV FalseTS TotalWR S, T, U, V-unobservable random variables; R,W-observable r.v. unknown. - - fixed numbers, Notations

5. Control of Familywise Error Rate The Familywise Error-Rate (in the weak sense): The Familywise Error-Rate (in the strong sense): For any configuration of true and false null hypotheses 5 Thus by assuring FWER ≤ , the probability of making even one type I error in the family, is controlled at level  Example: Fisher’s LSD procedure controls FWER in the weak sense, but does not control the FWER in the strong sense.

6. Error rates The Familywise Error-Rate For any configuration of true and false null hypotheses The Per Family Error-Rate For any configuration of hypotheses Expected number of type I errors The Per Comparison Error-Rate For any configuration of hypotheses Note: testing at (nominal) level  assures per comparison level is  ; amounts to “don’t worry – be happy” approach. *Other error-rates will be introduced later. 6

7. Comparison of error-rates The Familywise Error-Rate The Per Family Error-Rate The Per Comparison Error-Rate A multiple comparison procedure which controls the PFER also controls FWER and PCER, but not vice versa. 7

8. Power 8 Analogous to extending the type I error rate, power can be generalized in various ways when moving from single to multiple hypotheses test problems. The individual power: the rejection probability for a false hypothesis Average power: the expected number of correct rejections among all false null hypotheses The disjunctive power: the probability of rejecting at least one false null hypothesis. The conjunctive power: the probability of rejecting all false null hypotheses.

9. Bonferroni procedure If we test each hypothesis separately at level So to assure we may use Bonferroni multiple testing procedure: test each hypothesis separately at level Bonferroni procedure assures  control of PFER and FWER). –Is any condition needed? Example: GWAS, common threshold for significance: 9 Bonferroni inequality

10. Sidak’s procedure If the test statistics are independent, and we test each hypothesis separately at level To assure we may use Sidak’s multiple testing procedure: test each hypothesis separately at level Sidak’s procedure controls FWER at level when the test statistics are independent. Note: if Sidak’s procedure assures 10

11. Comparison between Bonferroni and Sidak’s procedures Idea: use dependency structure to get a better test. How much better? Very little gain even for small m: 11 m 0.025320570.0252 0.010206220.015 0.0051161970.00510

12. Adjusted p-value 12 One hypothesis: p-value is the smallest significance level for which the hypothesis can be rejected. Family of hypotheses: an adjusted p-value is defined as the smallest error rate level for which one rejects the hypothesis given a particular multiple comparison procedure. For FWER, if such an exists, and otherwise. If the null hypothesis can be rejected while controlling the FWER at level The marginal p-value for : denoted as the unadjusted p-value.

13. Bonferroni procedure and Sidak’s procedure: Adjusted p-value 13 Bonferroni procedure: test each hypothesis separately at level i.e. reject if Adjusted p-value for : Sidak’s procedure: test each hypothesis separately at level at level i.e. reject if Adjusted p-values: Homework. Computation of Bonferroni adjusted p-values using R: >p=c(0.01,0.015,0.005) >p.adjust(p, method="bonferroni") [1] 0.030 0.045 0.015

14. Example: Behavioral genetics 14 Study the genetics of behavioral traits: Hearing, sight, smell, alcoholism, locomotion, fear, exploratory behavior Compare inbred strains, crosses, knockouts…

15. 15 Significance of strain differences Behavioral Endpoint P-value Prop. Lingering Time 0.0029 # Progression segments 0.0068 Median Turn Radius (scaled) 0.0092 Time away from wall 0.0108 Distance traveled 0.0144 Acceleration0.0146 # Excursions 0.0178 Time to half max speed 0.0204 Max speed wall segments 0.0257 Median Turn rate 0.0320 Spatial spread 0.0388 Lingering mean speed 0.0588 Homebase occupancy 0.0712 # stops per excursion 0.1202 Stop diversity 0.1489 Length of progression segments 0.5150 Activity decrease 0.8875 Bonferroni.05/17=.0029 Unadjusted

16. Multiplicity problem for confidence intervals 16

17. Multiplicity problem for confidence intervals 17 Parameters: The observations are X (possibly X=(X 1,…,X m )). is a 1-  confidence interval (nominally/marginally) for parameter if Recall: Testing vs by rejecting H 0i iff the 1-  CI does not include, is an  -level test Therefore, same multiplicity problem for CIs as for testing

18. Multiplicity problem for confidence intervals 18 A set of confidence intervals are simultaneous confidence intervals at level for if Or Testing vs by rejecting iff where are simultaneous confidence intervals for at level is an FWER- controlling procedure at level

19. Simultaneous confidence intervals: examples 19 Bonferroni confidence intervals: marginal confidence intervals at nominal level. Show that Bonferroni confidence intervals are simultaneous confidence intervals at confidence level of at least 1-  Construct simultaneous confidence intervals that are equivalent to Sidak’s procedure (in the sense given in slide18). For each type of adjustments: Are there assumptions needed? If yes, what are they?