1. Locating Variance: Post-Hoc Tests
Developing Study Skills and Research Methods (HL20107)
Dr James Betts

2. Lecture Outline: Influence of multiple comparisons on P
Tukey’s HSD test
Bonferroni Corrections
Ryan-Holm-Bonferroni Adjustments.

3. Placebo
Placebo
Lucozade
Lucozade
Gatorade
Gatorade
Powerade
Powerade

4. Placebo
Placebo
Lucozade
Lucozade
Gatorade
Gatorade
Powerade
Powerade

5. Why not multiple t-tests?
i.e.
Placebo vs Lucozade
Placebo vs Gatorade
Placebo vs Powerade
Lucozade vs Gatorade
Lucozade vs Powerade
Gatorade vs Powerade
We accept ‘significance’ and reject the null hypothesis at P0.05 (i.e. a 5% chance that we are wrong)
Performing multiple tests therefore means that our overall chance of committing a type I error is >5%.

6. Post-hoc Tests
A popular solution is the Tukey HSD (Honestly Significant Difference) test
This uses the omnibus error term from the ANOVA to determine which means are significantly different
T = (q)
Error Variance √ n

9. q table for Tukey’s HSD

12. Tukey Test Critique
As you learnt last week, the omnibus error term is not reflective of all contrasts if sphericity is violated
Placebo
Lucozade
So Tukey tests commit many type I errors with even a slight degree of asphericity.
Gatorade
Powerade

13. Solution for Aspherical Data
There are alternatives to the Tukey HSD test which use specific error terms for each contrast
Fisher’s LSD (Least Significant Difference)
Sidak
Bonferroni
Many others…
e.g. Newman-Kewls, Scheffe, Duncan, Dunnett, Gabriel, R-E-G-W, etc.

16. Trial 2
Trial 1
Trial 4
Fisher’s LSD
Bonferroni
Trial 3

17. Bonferroni Correction Critique
Correction of LSD values successfully controls for type I errors following a 1-way ANOVA
However, factorial designs often involve a larger number of contrasts, many of which may not be relevant.
Recovery Supp. 1
Recovery Supp. 2
See also Perneger (1998) BMJ 316: 1236

18. Solution for Factorial Designs
An adjustment to the standard Bonferroni correction can be applied for factorial designs
This ‘Ryan-Holm-Bonferroni’ or ‘stepwise’ method involves returning to the P values of interest from our LSD test
These P values are placed in numerical order and the most significant is Bonferroni corrected (i.e. P x m)
However, all subsequent P values are multplied by m minus the number of contrasts already corrected.

19. Summary Post-Hoc Tests
A Tukey test may be appropriate when sphericity can be assumed
Multiple t-tests with a Bonferroni correction are more appropriate for aspherical data
Stepwise correction of standard Bonferroni procedures maintain power with factorial designs
Best option is to keep your study simple:
Pre-planned contrast at a specific time point
Summary statistics (e.g. rate of change, area under curve)
Just make an informed based on the data available.

20. Further reading from this lecture…
Atkinson, G. (2001) Analysis of repeated measurements in physical therapy research Physical Therapy in Sport 2: p
Atkinson, G. (2002) Analysis of repeated measurements in physical therapy research: multiple comparisons amongst level means and multi-factorial designs Physical Therapy in Sport 3: p

21. Compulsory reading for next week’s lecture…
Batterham A. M. & Atkinson, G. (2005) How Big Does My Sample Need to Be? A primer on the Murky World of Sample Size Estimation Physical Therapy in Sport 6: p