Download presentation

Presentation is loading. Please wait.

Published byRonan Wyler Modified over 2 years ago

1
Comparing Three or More Groups: Multiple Comparisons vs Planned Comparisons Robert Boudreau, PhD Co-Director of Methodology Core PITT-Multidisciplinary Clinical Research Center for Rheumatic and Musculoskeletal Diseases

2
First a simple thought experiment Flip a fair coin 100 times: Let H=# heads H = 0,1,2, …, 100 are the possible outcomes H = 0,1,2, …, 100 are the possible outcomes H has a binomial distribution with known probs H has a binomial distribution with known probs Prob[ 40 < H < 60 ] very close to 0.95 Prob[ 40 < H < 60 ] very close to 0.95 Prob [ H ≤ 40 ] + P[ H ≥ 60] = 0.05 -------------------------------------------------------------------------------------------------------

3
First a simple thought experiment Flip a fair coin 100 times: Let H=# heads H = 0,1,2, …, 100 are the possible outcomes H = 0,1,2, …, 100 are the possible outcomes H has a binomial distribution with known probs H has a binomial distribution with known probs Prob[ 40 < H < 60 ] very close to 0.95 Prob[ 40 < H < 60 ] very close to 0.95 Prob [ H ≤ 40 ] + P[ H ≥ 60] = 0.05 ------------------------------------------------------------------------------------------------------- Experiment: 20 people flip their own coin 100 times Q: Approx how many will get 40 or fewer heads or 60+ heads? or 60+ heads?

4
First a simple thought experiment Flip a fair coin 100 times: Let H=# heads H = 0,1,2, …, 100 are the possible outcomes H = 0,1,2, …, 100 are the possible outcomes H has a binomial distribution with known probs H has a binomial distribution with known probs Prob[ 40 < H < 60 ] very close to 0.95 Prob[ 40 < H < 60 ] very close to 0.95 Prob [ H ≤ 40 ] + P[ H ≥ 60] = 0.05 ------------------------------------------------------------------------------------------------------- Experiment: 20 people flip their own coin 100 times Q: Approx how many will get less than 40 heads or 60+ heads? Answer: One or 60+ heads? Answer: One

5
First a simple thought experiment Flip a fair coin 100 times: Let H=# heads H = 0,1,2, …, 100 are the possible outcomes H = 0,1,2, …, 100 are the possible outcomes H has a binomial distribution with known probs H has a binomial distribution with known probs Prob[ 40 < H < 60 ] very close to 0.95 Prob[ 40 < H < 60 ] very close to 0.95 Prob [ H ≤ 40 ] + P[ H ≥ 60] = 0.05 ------------------------------------------------------------------------------------------------------- Experiment: 20 people flip their own coin 100 times Q: Approx how many will get less than 40 heads or 60+ heads? Answer: One (1/20 = 5%) or 60+ heads? Answer: One (1/20 = 5%)

6
First a simple thought experiment Experiment: 20 people flip their own coin 100 times One (1/20=0.05) will flip an unusually small or unusually large # heads (on average) One (1/20=0.05) will flip an unusually small or unusually large # heads (on average) Q: Can we conclude that this person “X” flips an “unfair” coin, or was this explainable by “chance”?

7
Controlling Experiment-wise Error Experiment: 20 people flip their own coin 100 times Person X’s confidence interval didn’t cover 0.5 Q: What alpha level should be used so that 95% of the time all 20 confidence intervals each cover 0.5? (i.e. so that the correct conclusion is drawn about (i.e. so that the correct conclusion is drawn about every single coin) every single coin)

8
Controlling Experiment-wise Error Experiment: 20 people flip their own coin 100 times Person X’s confidence interval didn’t cover 0.5 Q: What alpha level should be used so that 95% of the time all 20 confidence intervals each cover 0.5? (i.e. so that the correct conclusion is drawn about (i.e. so that the correct conclusion is drawn about every single coin) every single coin) Equivalent to drawing a “wrong” conclusion about at least one of the coins only 5% of the time (Experiment-wise Type I error)

9
Controlling Experiment-wise Error Q: What alpha level should be used so that there’s a 95% probability that all 20 confidence intervals each cover 0.5? (aka Experiment-wise correct conclusion) Experiment-wise α=0.05, solve for comparison-wise α*: α = Prob[ At least one C.I. misses 0 ] α = Prob[ At least one C.I. misses 0 ] = 1 – Prob[ All C.I.’s cover 0 ] = 1 – Prob[ All C.I.’s cover 0 ] = 1 – (1 – α* ) 20 = 1 – (1 – α* ) 20 Sidak: Comparison-wise α* = 1 – (1 – α) 1/n n=20 “comparisons”: α* = 1 – (1-.05) 1/20 = 0.00256 n=20 “comparisons”: α* = 1 – (1-.05) 1/20 = 0.00256

10
Controlling Experiment-wise Error Q: What alpha level should be used so that there’s a 95% probability that all 20 confidence intervals each cover 0.5? Sidak: Comparison-wise α* = 1 – (1 – α) 1/n n=20 “comparisons”: α* = 1 – (1-.05) 1/20 = 0.00256 n=20 “comparisons”: α* = 1 – (1-.05) 1/20 = 0.00256 Bonferroni: α* = α/n ( 0.05/20=0.0025)

11
Controlling Experiment-wise Error Mathematically: α/n < 1 – (1 – α) 1/n Mathematically: α/n < 1 – (1 – α) 1/n Bonferroni < Sidak (i.e. higher α-level) But usually very close Sidak slightly more powerful Bonferroni works in all situations to guarantee control of experimentwise error (but may be conservative) Bonferroni works in all situations to guarantee control of experimentwise error (but may be conservative) Sidak (derived assuming independence) can under- control in presence of high correlations Sidak (derived assuming independence) can under- control in presence of high correlations

12
Comparison of Adverse Effect of 4 Drugs on Systolic BP

15
Unadjusted pairwise t-tests (α = 0.05 each comparison) critical value of t=2.13145

16
Pairwise t-tests (Bonferroni) critical value of t=3.03628 Pairwise t-tests (Bonferroni) critical value of t=3.03628

17
Pairwise t-tests (Sidak) critical value of t=3.02585 Pairwise t-tests (Sidak) critical value of t=3.02585

18
Comparison of critical values Scheffe: * Designed for arbitrary post-hoc testing * Controls experimentwise error for all possible simultaneous comparisons and contrasts

19
Comparison of Adverse Effect of 4 Drugs on Systolic BP (v2) Note: For Drug 4, I’ve subtracted 6 from the previous values s s

20
Comparison of Adverse Effect of 4 Drugs on Systolic BP (v2) ANOVA F-test

21
Unadjusted pairwise t-tests (v2) (α = 0.05 each comparison) critical value of t=2.13145

22
Pairwise t-tests (Bonferroni) (v2) critical value of t=3.03628 Pairwise t-tests (Bonferroni) (v2) critical value of t=3.03628

23
Pairwise t-tests (Sidak) (v2) critical value of t=3.02585 Pairwise t-tests (Sidak) (v2) critical value of t=3.02585

24
Tukey’s Studentized Range Test Related in concept to Scheffe’s Method Related in concept to Scheffe’s Method Designed for all pairwise comparisons exclusively Designed for all pairwise comparisons exclusively (recall: Scheffe applies to all possible simultaneous pairwise comparisons and contrasts) pairwise comparisons and contrasts) Exact experimentwise error coverage if sample sizes equal Exact experimentwise error coverage if sample sizes equal Critical values smaller than Bonferroni or Sidak Critical values smaller than Bonferroni or Sidak More powerful in finding differences

25
Pairwise t-tests (Tukey) (v2) critical value of t=2.88215 Pairwise t-tests (Tukey) (v2) critical value of t=2.88215

26
Comparison of Adverse Effect of 4 Drugs on Systolic BP

27
Dunnett’s Method (Comparison vs a Control) Related in concept to Scheffe and Tukey Methods Related in concept to Scheffe and Tukey Methods Designed for pairwise comparisons vs a single control exclusively Designed for pairwise comparisons vs a single control exclusively Exact experimentwise error coverage of those comparisons if sample sizes equal Exact experimentwise error coverage of those comparisons if sample sizes equal Critical values smaller than Bonferroni, Sidak or Tukey Critical values smaller than Bonferroni, Sidak or Tukey More powerful in finding differences vs control

28
Comparison vs Control (Dunnett) (v2) critical value of t=2.61702 Comparison vs Control (Dunnett) (v2) critical value of t=2.61702

29
Controlling for Multiple Comparisons in Exploratory Analyses Caterina Rosano, Howard J. Aizenstein, Stephanie Studenski, Anne B. Newman. Caterina Rosano, Howard J. Aizenstein, Stephanie Studenski, Anne B. Newman. A Regions-of-Interest Volumetric Analysis of Mobility Limitations in Community-Dwelling Older Adults. Journal of Gerontology: Medical Sciences 2007 A Regions-of-Interest Volumetric Analysis of Mobility Limitations in Community-Dwelling Older Adults. Journal of Gerontology: Medical Sciences 2007

30
Controlling for Multiple Comparisons in Exploratory Analyses A Regions-of-Interest Volumetric Analysis of Mobility Limitations in Community-Dwelling Older Adults. Journal of Gerontology: Medical Sciences 2007 A Regions-of-Interest Volumetric Analysis of Mobility Limitations in Community-Dwelling Older Adults. Journal of Gerontology: Medical Sciences 2007

31
Controlling for Multiple Comparisons in Exploratory Analyses

32
c

33
Thank you ! Any Questions? Robert Boudreau, PhD Co-Director of Methodology Core PITT-Multidisciplinary Clinical Research Center for Rheumatic and Musculoskeletal Diseases

Similar presentations

OK

Evaluating Hypotheses. Outline Empirically evaluating the accuracy of hypotheses is fundamental to machine learning – How well does this estimate accuracy.

Evaluating Hypotheses. Outline Empirically evaluating the accuracy of hypotheses is fundamental to machine learning – How well does this estimate accuracy.

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google