1. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 1 Multiple comparisons What are multiple comparisons? The problem of experiment-wise  error When do we do multiple comparisons? Statistical tests which control  e Estimating treatment effects

2. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 2 What are multiple comparisons? Pair-wise comparisons of different treatments These comparisons may involve group means, medians, variances, etc. for means, done after ANOVA In all cases, H 0 is that the groups in question do not differ. Yield CC NN  N+P Control Experimental (N) Experimental (N+P) c:Nc:N  N :  N+P  C :  N+P Frequency

3. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 3 Types of comparisons planned (a priori): independent of ANOVA results; theory predicts which treatments should be different. unplanned (a posteriori): depend on ANOVA results; unclear which treatments should be different. Test of significance are very different between the two! Y Y X1X1 X2X2 X3X3 X4X4 X5X5 X1X1 X2X2 X3X3 X4X4 X5X5 Planned unplanned

4. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 4 Planned comparisons (a priori contrasts): catecholamine levels in stressed fish Comparisons of interest are determined by experimenter beforehand based on theory and do not depend on ANOVA results. Prediction from theory: catecholamine levels increase above basal levels only after threshold PA O2 = 30 torr is reached. So, compare only treatments above and below 30 torr (N T = 12). Predicted threshold PA O 2 (torr) 10 20 30 4050 [Catecholamine] 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

5. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 5 Unplanned comparisons (a posteriori contrasts): catecholamine levels in stressed fish Comparisons are determined by ANOVA results. Prediction from theory: catecholamine levels increase with increasing PA O2. So, comparisons between any pairs of treatments may be warranted (N T = 21). Predicted relationship PA O 2 (torr) 10 20 3040 50 [Catecholamine] 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

6. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 6 The problem: controlling experiment- wise  error For k comparisons, the probability of accepting H 0 (no difference) is (1 -  ) k. For 4 treatments, (1 -  ) k = (0.95) 6 =.735, so experiment-wise  (  e ) = 0.265. Thus we would expect to reject H 0 for at least one paired comparison about 27% of the time, even if all four treatments are identical. Nominal  =.05 Number of treatments 02 46 8 10 Experiment-wise  (  e ) 0.0 0.2 0.4 0.6 0.8 1.0

7. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 7 Controlling experiment-wise  error at nominal   by  adjusting by total number of comparisons To maintain  e at nominal , we need to adjust  for each comparison by the total number of comparisons. In this manner,  e becomes independent of the number of treatments and/or comparisons. Number of treatments 0 24 68 10 Experiment-wise  (  e ) 0.0 0.2 0.4 0.6 0.8 1.0 Nominal  =.05

8. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 8 Controlling experiment-wise  error at nominal  by using modified test statistics Use modified test- statistic S for pair-wise comparisons whose distribution depends on the total number of comparisons N T such that p(S) increases with N T. 05101520 Value of test statistic (S) 0 0.2 0.3 Probability (p) S, N T = 1 S  N T = 2 S, N T = 3

9. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 9 Using multiple comparisons Use only after H 0 is rejected on the basis of an ANOVA because...... ANOVA is more robust and reliable than multiple comparisons. So, if H 0 is accepted in original ANOVA, do not proceed to do multiple comparisons. Note, however, that there is no universally agreed-upon method for doing multiple comparisons, and... …results may differ depending on which method you use. So, proceed with caution!

10. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 10 Controlling  e by adjusting individual  ’s

11. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 11 Controlling  e by adjusting individual  ’s p is probability associated with t-test of difference between 1 pair of means; k is total number of comparisons.

12. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 12 Example: Temporal variation in size of sturgeon (Model II ANOVA) Prediction: dam construction resulted in loss of large sturgeon Test: compare sturgeon size before and after dam construction H 0 : mean size is the same for all years Construction du barrage

13. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 13 Example: Temporal variation in size of sturgeon (ANOVA results) Conclusion: reject H 0 Type III Sum of Squares Df Sum of Sq Mean Sq F Value Pr(F) YEAR 3 485.264 161.7547 5.95744 0.0008246026 Residuals 114 3095.295 27.1517

14. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 14 Multiple comparison results 95 % simultaneous confidence intervals for specified linear combinations, by the Bonferroni method critical point: 2.6851 response variable: FKLNGTH intervals excluding 0 are flagged by '****' Estimate Std.Error Lower Bound Upper Bound 1954-1958 -0.187 1.33 -3.7700 3.39 1954-1965 5.510 1.73 0.8600 10.20 **** 1954-1966 3.310 1.17 0.1750 6.45 **** 1958-1965 5.690 1.82 0.8120 10.60 **** 1958-1966 3.500 1.29 0.0241 6.98 **** 1965-1966 -2.190 1.70 -6.7600 2.37

15. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 15 Multiple comparison results (con’t) Conclusion: Fish size was smaller after the construction of the dam.

16. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 16 Controlling  e by using modified test statistics

17. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 17 Making a choice Tukey’s, GT2 Try several methods, and if you get similar results, you’re on safe ground. If you get differences, they are due to: –how conservative/liberal the test is –how powerful it is If comparisons using Bonferroni are still significant, you’re O.K. If comparisons using Sidak are still non-significant, you’re also O.K.

18. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 18 Estimating treatment effects (Model I ANOVA only!) Concern is not just with whether treatments differ, but with how much they differ. For example, what effect does a change in water temperature of 4°C have on trout growth rate? Since each treatment mean has a certain precision, so too will the estimate of the effect. Water temperature (°C) 16202428 0.00 0.04 0.08 0.12 0.16 0.20 Growth rate (cm/day) Estimated effect Range of possible effects

19. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 19 Estimating treatment effects: confidence limits for group means using MS error Use MS error from ANOVA table as an estimate of the variance of each treatment to calculate confidence intervals for treatment means.

20. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 20 Estimating treatment effects: confidence limits for group means using within- group variances Use estimated standard deviations s i for each treatment (group):

21. University of Ottawa - Bio 4158 – Applied Biostatistics © Antoine Morin and Scott Findlay 20/02/2016 11:23 PM 21 Computing multiple CI’s for treatment means For any group,  =.05, i.e. 5% of the time the true mean will lie outside the estimated 95% CI’s. So, if you calculate multiple CI’s, you should control for  e. For example, using Bonferonni  ’ =  /k, we would have: