1 Multiple Comparison Procedures Once we reject H 0 :   =   =...  c in favor of H 1 : NOT all  ’s are equal, we don’t yet know the way in which they’re not all equal, but simply that they’re not all the same. If there are 4 columns, are all 4  ’s different? Are 3 the same and one different? If so, which one? etc.

2 These “more detailed” inquiries into the process are called MULTIPLE COMPARISON PROCEDURES. Errors (Type I): We set up “  ” as the significance level for a hypothesis test. Suppose we test 3 independent hypotheses, each at  =.05; each test has type I error (rej H 0 when it’s true) of.05. However, P(at least one type I error in the 3 tests) = 1-P( accept all ) = 1 - (.95) 3 .14 3, given true

3 In other words, Probability is.14 that at least one type one error is made. For 5 tests, prob =.23. Question - Should we choose  =.05, and suffer (for 5 tests) a.23 OVERALL Error rate (or “a” or  experimentwise )? OR Should we choose/control the overall error rate, “a”, to be.05, and find the individual test  by 1 - (1-  ) 5 =.05, (which gives us  =.011)?

4 The formula 1 - (1-  ) 5 =.05 would be valid only if the tests are independent; often they’re not. [ e.g.,  1 =  2  2 =  3,  1 =  3 IF accepted & rejected, isn’t it more likely that rejected? ]

5 When the tests are not independent, it’s usually very difficult to arrive at the correct  for an individual test so that a specified value results for the overall error rate.

6 Categories of multiple comparison tests - “Planned”/ “a priori” comparisons (stated in advance, usually a linear combination of the column means equal to zero.) - “Pairwise” comparisons (every column mean compared with each other column mean) - “Post hoc”/ “a posteriori” comparisons (decided after a look at the data - which comparisons “look interesting”)

7 (Pairwise comparisons are traditionally considered as “post hoc” and not “a priori”, if one needs to categorize all comparisons into one of the two groups) There are many multiple comparison procedures. We’ll cover only a few. Method 1: Do a series of pairwise t-tests, each with specified  value (for individual test). This is called “Fisher’s LEAST SIGNIFICANT DIFFERENCE” (LSD).

8 Example: Broker Study A financial firm would like to determine if brokers they use to execute trades differ with respect to their ability to provide a stock purchase for the firm at a low buying price per share. To measure cost, an index, Y, is used. Y=1000(A-P)/A where P=per share price paid for the stock; A=average of high price and low price per share, for the day. “The higher Y is the better the trade is.”

9 } R=6 CoL: broker Five brokers were in the study and six trades were randomly assigned to each broker.

10  =.05, F TV = 2.76 (reject equal column MEANS) “MSW”

11 0 For any comparison of 2 columns,  /2 CLCL CuCu Y i -Y j AR: 0 + t  /2 x  MSW x  nini njnj 25 df (n i = n j = 6, here) SQ Root of Pooled Variance, “s 2 ”, perhaps, in earlier class in basic statistics : p

12 In our example, with  =.05 0  (  21.2 x  ) 0  This value, 5.48 is called the Least Significant Difference (LSD). When same number of data points, R, in each column, LSD = t  /2 x  2xMSW. R

13 Col: Now, rank order and compare: Underline Diagram

14 Step 1: identify difference > 5.48, and mark accordingly: : compare the pair of means within each subset: Comparison difference vs. LSD 3 vs. 1 2 vs. 4 2 vs. 5 4 vs. 5 ****** <<<<<<<< * Contiguous; no need to detail 5

15 Conclusion : 3, 1 2, 4, 5 Can get “inconsistency”: Suppose col 5 were 18: Now: Comparison |difference| vs. LSD 3 vs. 1 2 vs. 4 2 vs. 5 4 vs. 5 * <<><<<>< Conclusion : 3, ??? 6

16 Broker 1 and 3 are not significantly different but they are significantly different to the other 3 brokers. Conclusion : 3, Broker 2 and 4 are not significantly different, and broker 4 and 5 are not significantly different, but broker 2 is different to (smaller than) broker 5 significantly.

17

18

19

20

21 Fisher's pairwise comparisons (Minitab) Family error rate = Individual error rate = Critical value =  t_(  /2) Intervals for (column level mean) - (row level mean) Minitab: Stat>>ANOVA>>one way anova then click “comparisons”.

22 In the previous procedure, each individual comparison has error rate  =.05. The overall error rate is, were the comparisons independent, 1- (.95) 10 =.401. However, they’re not independent. Method 2: A procedure which takes this into account and pre-sets the overall error rate is “TUKEY’S HONESTLY SIGNIFICANT DIFFERENCE TEST ”.

23 Tukey’s method works in a similar way to Fisher’s LSD, except that the “LSD” counterpart (“HSD”) is not t  /2 x  MSW x  nini njnj t  /2 x  2xMSW R = or, for equal number of data points/col ( ), but tuk X  2xMSW, R where t uk has been computed to take into account all the inter-dependencies of the different comparisons.  /2

24 HSD = t uk  /2 x  2MSW R ________________________________________ A more general approach is to write HSD = q  /2 x  MSW R where q  /2 = t uk  /2 x  q = (Y largest - Y smallest ) /  MSW R ---- probability distribution of q is called the “Studentized Range Distribution”. --- q = q(c, df), where c =number of columns, and df = df of MSW

25 q table

26 With c = 5 and df = 25, from table: q = 4.16 (between 4.10 and 4.17) t uk = 4.16/1.414 = 2.94 Then, HSD = 4.16   also    x 

27 In our earlier example: Rank order: (No differences [contiguous] > 7.82)

28 Comparison |difference| >or< vs. 1 3 vs. 2 3 vs. 4 3 vs. 5 1 vs. 2 1 vs. 4 1 vs. 5 2 vs. 4 2 vs. 5 4 vs. 5 * <<>><>><<<<<>><>><<< 9 12 * 8 11 * 5 * (contiguous) 7 3, 1, 24, 5 2 is “same as 1 and 3, but also same as 4 and 5.”    

29

30

31 Tukey's pairwise comparisons (Minitab) Family error rate = Individual error rate = Critical value = 4.15  q_(1-  /2) Intervals for (column level mean) - (row level mean)

32 Exercise: Drug Study A drug company are developing two new drug formulations for treating flu, denoted as drug A and drug B. Two groups of 10 volunteers were taken drug A and drug B, respectively, and after three days, their responses (Y) were recorded. A placebo group was added to check the effectiveness of drugs. The larger the Y value is, the more effective the drug is. Here is the data: (MSE=1) Drug ADrug BPlacebo Index i123 Column mean Sample size10

33 LSD = t 97.5%;27 df  2/10 =  2/10 = HSD = q 97.5%;27 df  1/10 = 3.51  1/10 = Placebo Drug B Drug A Placebo Drug B Drug A

34 Method 3: Dunnett’s test Designed specifically for (and incorporating the interdependencies of) comparing several “treatments” to a “control.” Example: Col } R=6 CONTROL Analog of LSD (=t 1-  /2 x  2 MSW ) R = Dut 1-  /2 x  2 MSW R

35 D table p. 107

36 Dut 1-  /2 x  2 MSW/R = 2.61 (  2(21.2) ) = Cols 4 and 5 differ from the control [ 1 ]. - Cols 2 and 3 are not significantly different from control. 6 In our example: CONTROL Comparison |difference| >or< vs. 2 1 vs. 3 1 vs. 4 1 vs < >

37

38 Dunnett's comparisons with a control (Minitab) Family error rate = Individual error rate = Critical value = 2.61  Dut_1-  /2 Control = level (1) of broker Intervals for treatment mean minus control mean Level Lower Center Upper ( * ) ( * ) ( * ) ( * )

39 This procedure provides a subset of treatments that cannot distinguished from the best. The probability of that the “best” treatment is included in this subset is controlled at 1- . Method 4: MCB Procedure (Compare to the best) *Assume that larger is better.

40 STEP 1: Calculate the following for all index i where l (not i) is the group of which mean reaches

41 STEP 2: Conduct tests The treatment i is included in the best subset if

42 Drug ADrug BPlacebo Index i123 Column mean DiDi What drugs are in the best subset? (Given MSE = 1.)

43 Identify the subset of the best brokers Hsu's MCB (Multiple Comparisons with the Best) Family error rate = Critical value = 2.27 Intervals for level mean minus largest of other level means Level Lower Center Upper (------* ) ( *------) ( * ) (------* ) ( *------) Brokers 2, 4, 5

Post Hoc comparisons *F test for contrast (in “Orthogonality”) *Scheffe test (p.108; skipped) To test all linear combinations at once. Very conservative; not to be used for pairwise comparisons. ----A Priori comparisons * covered later in chapter on “Orthogonality”