1. Everyday is a new beginning in life.
Every moment is a time for self vigilance.

2. Multiple Comparisons Error rate of control Pairwise comparisons
Comparisons to a control
Linear contrasts

3. Multiple Comparison Procedures
Once we reject H0: ==...c in favor of H1: NOT all ’s are equal, we don’t yet know the way in which they’re not all equal, but simply that they’re not all the same. If there are 4 columns, are all 4 ’s different? Are 3 the same and one different? If so, which one? etc.

4. These “more detailed” inquiries into the process are called MULTIPLE COMPARISON PROCEDURES.
Errors (Type I):
We set up “” as the significance level for a hypothesis test. Suppose we test 3 independent hypotheses, each at = .05; each test has type I error (rej H0 when it’s true) of However,
P(at least one type I error in the 3 tests)
= 1-P( accept all ) = 1 - (.95)3  .14
3, given true

5. In other words, Probability is
In other words, Probability is .14 that at least one type one error is made. For 5 tests, prob = .23.
Question - Should we choose = .05, and suffer (for 5 tests) a .23 Experimentwise Error rate (“a” or aE)? OR
Should we choose/control the overall error rate, “a”, to be .05, and find the individual test  by 1 - (1-)5 = .05, (which gives us  = .011)?

6. would be valid only if the tests are independent; often they’re not.
The formula
1 - (1-)5 = .05
would be valid only if the tests are independent; often they’re not.
[ e.g., 1=22= 3, 1= 3
IF accepted & rejected, isn’t it more likely that rejected? ] 1 2 3 1 2 3

7. Error Rates
When the tests are not independent, it’s usually very difficult to arrive at the correct for an individual test so that a specified value results for the experimentwise error rate (or called family error rate).

8. There are many multiple comparison procedures. We’ll cover only a few.
Pairwise Comparisons
Method 1: (Fisher Test) Do a series of pairwise t-tests, each with specified  value (for individual test).
This is called “Fisher’s LEAST SIGNIFICANT DIFFERENCE” (LSD).

9. Example: Broker Study
A financial firm would like to determine if brokers they use to execute trades differ with respect to their ability to provide a stock purchase for the firm at a low buying price per share. To measure cost, an index, Y, is used.
Y=1000(A-P)/A
where
P=per share price paid for the stock;
A=average of high price and low price per share, for the day.
“The higher Y is the better the trade is.”

10. } R=6 Five brokers were in the study and six trades
CoL: broker 1 12 3 5 -1 6 2 7 17 13 11 12 3 8 1 7 4 5 4 21 10 15 12 20 6 14 5 24 13 14 18 19 17 }
R=6
Five brokers were in the study and six trades
were randomly assigned to each broker.

11.  = .05, FTV = 2.76 (reject equal column MEANS)
“MSW”
 = .05, FTV = 2.76
(reject equal column MEANS)

12. For any comparison of 2 columns,
Yi -Yj
/2
/2 CL Cu
AR: 0+ ta/2 x MSW x 1 + 1 ni nj
dfw
(ni = nj = 6, here)
Pooled Variance, the estimate for the common variance
MSW :

13. In our example, with=.05 0  2.060 (21.2 x 1 + 1 ) 0 5.48
This value, 5.48 is called the Least Significant Difference (LSD).
When same number of data points, R, in each column, LSD = ta/2 x 2xMSW. R

14. Underline Diagram Col: 3 1 2 4 5
Summarize the comparison results. (p. 443)
Now, rank order and compare:
Col:

15. Step 2: identify difference > 5.48, and mark accordingly:
3: compare the pair of means within
each subset:
Comparison difference vs. LSD
3 vs. 1
2 vs. 4
2 vs. 5
4 vs. 5
< * 5
* Contiguous; no need to detail

16. > Conclusion : 3, 1 2, 4, 5 Can get “inconsistency”: Suppose col 5
were 18:
Now: Comparison |difference| vs. LSD
<
>
3 vs. 1
2 vs. 4
2 vs. 5
4 vs. 5 * 6
Conclusion : 3,
???

17. Conclusion : 3,
Broker 1 and 3 are not significantly different but they are significantly different to the other 3 brokers.
Broker 2 and 4 are not significantly different, and broker 4 and 5 are not significantly different, but broker 2 is different to (smaller than) broker 5 significantly.

19. Minitab: Stat>>ANOVA>>One-Way Anova then click “comparisons”.
Fisher's pairwise comparisons (Minitab)
Family error rate = 0.268
Individual error rate =
Critical value =  t_a/2
Intervals for (column level mean) - (row level mean)
-0.524
Col 1 < Col 2
Col 2 = Col 4

20. Pairwise comparisons
Method 2: (Tukey Test) A procedure which controls the experimentwise error rate is “TUKEY’S HONESTLY SIGNIFICANT DIFFERENCE TEST ”.

21. or, for equal number of data points/col
Tukey’s method works in a similar way to Fisher’s LSD, except that the “LSD” counterpart (“HSD”) is not
ta/2 x MSW x  1 + 1 ni nj (
or, for equal number of data points/col ) = ,
ta/2 x 2xMSW R
but tuk X 2xMSW , R
a/2
where tuk has been computed to take into account all the inter-dependencies of the different comparisons.

22. HSD = tuka/2x2MSW R _______________________________________
A more general approach is to write
HSD = qaxMSW R where qa = tuka/2 x2
--- q = (Ylargest - Ysmallest) / MSW R
---- probability distribution of q is called the
“Studentized Range Distribution”.
--- q = q(c, df), where c =number of columns,
and df = df of MSW

23. With c = 5 and df = 25, from table (or Minitab): q = 4. 15 tuk = 4
Then,
HSD = 4.15 21.2/6 = 7.80
also, 2.93 2x21.2/6 = 7.80

24. In our earlier example:
Rank order:
(No differences [contiguous] > 7.80)

25. Comparison |difference| >or< 7.80 3 vs. 1 3 vs. 2 3 vs. 4
(contiguous) * 7 9 12 * 8 11 5
3, 1, 2
4, 5
2 is “same as 1 and 3, but also same as 4 and 5.”

26. Minitab: Stat>>ANOVA>>One-Way Anova then click “comparisons”.
Tukey's pairwise comparisons (Minitab) Family error rate = Individual error rate = Critical value = 4.15  q_a Intervals for (column level mean) - (row level mean)

27. Method 3: Dunnett’s test
Special Multiple Comp.
Method 3: Dunnett’s test
Designed specifically for (and incorporating the interdependencies of) comparing several “treatments” to a “control.”
Col
Example: }
R=6
CONTROL
Analog of LSD
(=t/2 x 2 MSW )
D = Dut/2 x 2 MSW R R
From table or Minitab

28. Comparison |difference| >or< 6.94
D= Dut/2 x 2 MSW/R
= 2.61 (2(21.2) )
= 6.94
CONTROL 6
In our example:
Comparison |difference| >or< 6.94
1 vs. 2
1 vs. 3
1 vs. 4
1 vs. 5
<
> 6 1 8 11
- Cols 4 and 5 differ from the control [ 1 ].
- Cols 2 and 3 are not significantly different
from control.

29. Minitab: Stat>>ANOVA>>General Linear Model then click “comparisons”.
Dunnett's comparisons with a control (Minitab)
Family error rate =  controlled!!
Individual error rate =
Critical value = 2.61  Dut_a/2
Control = level (1) of broker
Intervals for treatment mean minus control mean
Level Lower Center Upper
( * )
( * )
( * )
( * )

30. What Method Should We Use?
Fisher procedure can be used only after the F-test in the Anova is significant at 5%.
Otherwise, use Tukey procedure. Note that to avoid being too conservative, the significance level of Tukey test can be set bigger (10%), especially when the number of levels is big.

31. Contrast Example 1 Suppose the questions of interest are2 3 4
Sulfa
Type S1
Sulfa
Type S2
Anti-biotic Type A
Placebo
Suppose the questions of interest are
(1) Placebo vs. Non-placebo
(2) S1 vs. S2
(3) (Average) S vs. A

32. In general, a question of interest can be expressed by a linear combination of column means such as
with restriction that Saj = 0.
Such linear combinations are called contrasts.

33. Test if a contrast has mean 0
The sum of squares for contrast Z is
where R is the number of rows (replicates).
The test statistic Fcalc = SSC/MSW is distributed as F with 1 and (df of error) degrees of freedom.
Reject E[C]= 0 if the observed Fcalc is too large
(say, > F0.05(1,df of error) at 5% significant level).

34. Example 1 (cont.): aj’s for the 3 contrasts
P vs. P: C1
S1 vs. S2:C2
S vs. A: C3

35. Calculating
top row
middle row
bottom row
 
 
 

36. Y.1 Y.2 Y.3 Y.4 Placebo vs. drugs S1 vs. S2 Average S vs. A 5 6 7 10
P S S A
Placebo
vs. drugs
S1 vs. S2
Average S
vs. A -3 1 1 1
5.33
0.50 -1 1
8.17 -1 -1 2
14.00

37. 5.33
42.64
.50
4.00
8.17
65.36

38. Tests for Contrasts F1-.05(1,28)=4.20 Source SSQ df MSQ F C1 C2 C3
42.64
4.00
65.36
42.64
4.00
65.36
8.53
.80
13.07 1
Error
F1-.05(1,28)=4.20

39. Example 1 (Cont.): Conclusions
The mean response for Placebo is significantly different to that for Non-placebo.
There is no significant difference between using Types S1 and S2.
Using Type A is significantly different to using Type S on average.