STA MCP1 Multiple Comparisons: Example Study Objective: Test the effect of six varieties of wheat to a particular race of stem rust. Treatment: Wheat Variety Levels: A(i=1), B (i=2), C (i=3), D (i=4), E (i=5), F (i=6) Experimental Unit: Pot of well mixed potting soil. Replication: Four (4) pots per treatment, four(4) plants per pot. Randomization: Varieties randomized to 24 pots (CRD) Response: Yield (Y ij ) (in grams) of wheat variety(i) at maturity in pot (j). Implementation Notes: Six seeds of a variety are planted in a pot. Once plants emerge, the four most vigorous are retained and inoculated with stem rust.

STA MCP2 Statistics and AOV Table RankVarietyMean Yield 5A50.3 4B69.0 6C24.0 2D94.0 3E75.0 1F95.3 n 1 =n 2 =n 3 =n 4 =n 5 =n=4 ANOVA Table SourcedfMeanSquareF Variety ** Error

STA MCP3 Overall F-test indicates that we reject H 0 and assume H A Which mean is not equal to which other means. Consider all possible comparisons between varieties: First sort the treatment levels such that the level with the smallest sample mean is first down to the level with the largest sample mean. Then in a table (matrix) format, compute the differences for all of the t(t-1)/2 possible pairs of level means.

STA MCP4 Question: How big does the difference have to be before we consider it “significantly big”? Largest Difference Smallest difference Differences for all of the t(t-1)/2=15 possible pairs of level means

STA MCP5 Fisher’s Protected LSD F=24.8 > F 5,18,.05 = > F is significant ‡‡‡ ‡ ‡ ‡‡ ‡ ‡ ‡ ‡ ‡‡ ‡ Implies that the two treatment level means are statistically different at the  = 0.05 level. acbdd c Alternate ways to indicate grouping of means.

STA MCP6 Tukey’s W (Honestly Significant Difference) Not protected hence no preliminary F test required. ‡‡‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ Implies that the two treatment level means are statistically different at the  = 0.05 level. abcbddc d Table 10

STA MCP7 Student-Newman-Keul Procedure (SNK) Not protected hence no preliminary F test required. Table 10 row Error df=18  = 0.05 col = r neighbors One between Two between

STA MCP8 ‡‡‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ Implies that the two treatment level means are statistically different at the  = 0.05 level. acbddc ‡ ‡ ‡ SNK

STA MCP9 Duncan’s New Multiple Range Test (Passe) Not protected hence no preliminary F test required. Table 11 (next pages) row error df = 18  = 0.05 col = r neighbors One between Two between

STA MCP10 Duncan’s Test Critical values

STA MCP11

STA MCP12 ‡‡‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ Implies that the two treatment level means are statistically different at the  = 0.05 level. acbddc ‡ ‡ ‡ Duncan’s MRT

STA MCP13 Scheffé’s S Method F=24.8 > F 5,18,.05 =2.77 => F is significant For comparing Reject H o : l=0 at  =0.05 if Since each treatment is replicated the same number of time, S will be the same for comparing any pair of treatment means.

STA MCP14 ‡‡‡ ‡‡ ‡ ‡ Implies that the two treatment level means are statistically different at the  = 0.05 level. ab ca bccb c Any difference larger than S=28.82 is significant. Very conservative => Experimentwise error driven. Scheffe’s S Method

STA MCP15 Grouping of Ranked Means LSD SNK Duncan’s Tukey’s HSD Scheffe’s S Which grouping will you use? 1) What is your risk level? 2) Comparisonwise versus Experimentwise error concerns.

STA MCP16 So, which MC method should you use…? There is famous story of a statistician and his two clients: Client 1 arrives daily with his hypothesis test and asks for assistance. The statistician helps him using α=0.05. After 1 year they have done 365 tests. If all nulls tested were indeed true, they would have made approx (365)(0.05) = 18 erroneous rejections, but they are satisfied with the progress of the research. Client 2 saves all his statistical analysis for end of the year, and approaches the statistician for help. The statistician responds: “My! You have a terrible multiple comparisons problem!” In cases where the researcher is just searching the data (does not have an interest in every comparison made), some form of error rate control beyond the simple Fisher’s LSD may be appropriate. On the other hand, if you definitely have an interest in every comparison, it may be better to use LSD (and accept the comparison-wise error rate).

STA MCP17 Which method to use? Some practical advice If comparisons were decided upon before examining the data (best): Just one comparison – use the standard (two-sample) t-test. (In this case use the pooled estimate of the common variance, MSE, and it’s corresponding error df. This is just Fisher’s LSD.) Few comparisons – use Bonferroni adjustment to the t-test. With m comparisons, use  /m for the critical value. Many comparisons – Bonferroni becomes increasingly conservative as m increases. At some point it is better to use Tukey (for pairwise comparisons) or Scheffe (for contrasts). If comparisons were decided upon after examining the data: Just want pairwise comparisons – use Tukey. All contrasts (linear combinations of treatment means) – use Scheffe.