1 Psych 5510/6510 Chapter 12 Factorial ANOVA: Models with Multiple Categorical Predictors and Product Terms. Spring, 2009

2 Factorial Designs Designs with ‘q’ number of independent variables, each having two or more levels (i.e. each IV has two or more groups), combined in such a way that every level of one independent variable is combined with every level of the other independent variables.

3 Our example Independent Variable A: Type of Drug A1: Drug A A2: Drug B A3: Placebo ‘a’ = number of levels of A = 3 Independent Variable B: Enzyme B1: With Enzyme B2: Without Enzyme ‘b’ = number of levels of B = 2 Dependent Variable = Mood

4 Layout ‘S i ’ represents some subject. 3 scores per cell (i.e. n=3) 18 scores total (i.e. N=18) This is a completely independent group design (both variables are ‘between subject’ variables).

5 Data

6 Means Independent Variable A H 0 : μ Drug_A = μ Drug_B = μ Placebo H A : at least one μ is different. Independent Variable B H 0 : μ With_E = μ Without_E H A : at least one μ is different. Interaction of AB H 0 : A and B do not interact H A : A and B interact

7 Interaction A and B interact if the effect of one depends upon the level of the other. For example: the overall effect of moving from Drug A to Drug B is to lower mood by 2 (i.e =2), and the overall effect of moving from Drug B to the Placebo is to lower mood by 9.5. If that pattern occurs across all levels of B (i.e. if that effect is found in both the ‘With Enzyme’ as well as in the ‘Without Enzyme’ groups) then the two variables DO NOT interact.

8 Interaction If A interacts with B, then B interacts with A (it’s a relationship). The overall effect of moving from With Enzyme group to Without Enzyeme group is to lower mood by 10 (i.e =10). If that pattern occurs across all levels of A then the two variables DO NOT interact.

9 Cell Means

10 Cell Means

11 Traditional ANOVA Summary Table

12 Drawbacks to Traditional Approach 1.Overall F tests often of little interest. 2.Less power than 1 parameter contrasts.

13 Book’s Approach: Setting up Contrast Codes Layout all six cells (treatment combinations) in a row. To completely code this experiment will require 5 contrast codes: (a)(b)-1 = (3)(2)-1=5. Or in other words, (the number of cells –1).

14 Generating codes Begin by creating a complete set of contrast codes for each independent variable.

15 IV: Type of Drug Contrast One: The two drug groups vs. the placebo group Contrast Two: Drug A group vs. Drug B group. Note that we have to select comparisons that are orthogonal to each other (otherwise they wouldn’t fit our definition of being contrasts).

16 Apply Contrast One Apply the lambdas for Contrast One to all of the cells, including both levels of Independent Variable B (level of enzyme).

17 Apply Contrast Two Apply the lambdas for Contrast Two to all the cells).

18 Contrasts So Far Note these two contrasts are orthogonal to each other.

19 IV: Level of Enzyme Contrast Three: Now completely code the second independent variable ‘With Enzyme’ vs ‘Without Enzyme’. There are only two levels of this IV and thus we only need one contrast, if we needed more than one comparison to code IV B then we would need to make sure they were orthogonal to each other.

20 Apply Contrast Three Apply the lambda for Contrast Three to all the cells, including levels of Independent Variable A (type of drug).

21 Contrasts So Far Note: the contrasts for IV A will automatically be orthogonal to the contrasts for IV B (this is due to the way we designed the experiment...as a factorial design).

22 Interaction of Type of Drug and Level of Enzyme Remember we should end up with 5 contrasts, so far we have come up with 3. The final two contrasts will cover the interaction of our two independent variables. To generate interaction contrast codes multiply each contrast in one IV by each contrast in the other IV.

23 The Interaction Contrasts Independent Variable A: Contrast 1 Contrast 2 Independent Variable B: Contrast 3 Interaction of A and B Contrast 4 = Contrast 1 x Contrast 3 Contrast 5 = Contrast 2 x Contrast 3

24 Interaction Contrasts Contrast 4 will be Contrast 1 x Contrast 3

25 What Does Contrast 4 test? Contrast 1 looks for the difference between the Drug Groups and the Placebo Group. Contrast 3 looks for the difference between the enzyme and no enzyme group. Contrast 4--which is Contrast 1 x Contrast 3- -looks to see if the difference between the Drug groups and the Placebo group is the same in the enzyme condition as it is in the no enzyme condition.

26 Contrast 1: drug groups vs. placebo Contrast 3: with enzyme vs. without enzyme Contrast 4: (drug vs. placebo with enzyme) vs. (drug vs. placebo without enzyme). Note strategy for writing interaction, state first contrast in its ‘=0’ form, then compare the size of that in the two conditions of the second contrast to see if there is an interaction.

27 Interaction Contrasts Contrast 5 will be Contrast 2 x Contrast 3

28 What Does Contrast 5 test? Contrast 2 looks for the difference between the Drug A group and the Drug B group. Contrast 3 looks for the difference between the enzyme and no enzyme group. Contrast 5--which is Contrast 2 x Contrast 3- -looks to see if the difference between the two drug groups is the same in the enzyme condition as it is in the no enzyme condition.

29 Or.... Contrast 2: Contrast 3: Contrast 5:

30 Complete set of Contrasts The interaction contrasts will automatically be orthogonal to each other and to the other contrasts in the set, this is because multiplying orthogonal contrasts leads to an orthogonal contrast.

31 SPSS Regression Analysis For the analysis of the overall model: MODEL C: Ŷ i = β 0 MODEL A: Ŷ i = β 0 + β 1 X i1 + β 2 X i2 + β 3 X i3 + β 4 X i4 + β 5 X i5 Ŷi = Xi1 +1.0Xi2+ 5.0Xi Xi Xi5 With this formula we end up predicting that each person’s score equals the mean of his or her cell (treatment combination)

32 SPSS Regression Analysis For the analysis of the overall model: MODEL C: Ŷ i = β 0 MODEL A: Ŷ i = β 0 + β 1 X i1 + β 2 X i2 + β 3 X i3 + β 4 X i4 + β 5 X i5

33 Summary Table from Regression ANOVA Values plugged in from overall regression in previous slide.

34 SPSS Analysis (cont) Remember that for each contrast: df=1 F=t² PRE=Partial Correlation² SSE(Contrast)=(MS residual )(F contrast ) For the analysis of each contrast....

35 Summary Table from Regression ANOVA

36 Summary Table In SPSS the overall regression of Y on X 1 through X 5 will give you the SS, df, MS, F, PRE, and p for: Between Groups, Within Groups, and Total. The analysis of the partial regression coefficients and partial correlations will give you the information you need on each individual contrast. What is not provided is the information you need for each independent variable (the main effects of Drug and Enzyme) and the main interaction of the independent variables (DrugxEnzyme interaction)

37 What is Not Provided Directly by SPSS Regression Analysis

38 Solution 1 If you want those missing values, then one option is to have SPSS do a 2 Factor ANOVA on your data (we covered this last semester), copy over those values to the more complete table. This will give you everything but the PRE value.

39 Solution 1 (cont.) PRE = (SS for the variable) / (SS for the variable + SS Within Groups)

40 Solution 2 A second option is to simply use the model comparison approach to get your values. For example, independent variable ‘type of drug’ was coded with contrasts 1 and 2 (using variables X1 and X2). To determine the overall effect of ‘type of drug’ do this: MODEL C: Ŷ i = β 0 + β 3 X i3 + β 4 X i4 + β 5 X i5 MODEL A: Ŷ i = β 0 +β 3 X i3 + β 4 X i4 + β 5 X i5 +β 1 X i1 + β 2 X i2 SPSS doesn’t do this directly, you have to use the procedures described in Chapter 8. This will get you everything except the p value, which you can then obtain from the PRE tool.

41 The Full Regression Analysis Note that the SS Drug =SS X1 +SS X2, and SS DxE =SS X4 +SS X5, this will be the case when each cell (treatment combination) has equal N.

42 Note what we gain from having the comparisons in the analysis. While the overall effect of I.V. drug is significant, we can see it is due to the difference between the drugs and the placebo group (X1) and not due to a difference between the two drugs (X2).

43 And, rather interestingly, while there is no overall difference in the effect of the two drugs (X2), there is an interaction between the effect of the two drugs and the presence or absence of the enzyme (X5).

44 There is reason to question whether the overall effect of Drug, Enzyme and Interaction are even of any interest, why not just stick with evaluating the contrasts?

45 Higher-Order ANOVA: Generating Contrast Codes 1. Generate a complete set of contrast codes for each categorical variable. 2. For each pair of categorical variables, construct interaction codes by multiplying all possible pairs of contrast codes, one from each of the two categorical variables. 3. For each triple of categorical variables, construct interaction codes by multiplying all possible triads of contrast codes, one from each of the three categorical variables. 4. And so on...remember the total number of contrast codes needed to completely code the experiment will equal the number of treatment combinations - 1. For example, (ab-1) for 2 IVs, or (abc-1) for 3, etc.

46 Example IV A has 3 levels, IV B has 2, IV C has 2, that is 12 cells, so the number of contrasts = 11. IV A Contrast 1 Contrast 2 IV B Contrast 3 IV C Contrast 4 A x B Interaction Contrast 5 = Contrast 1 x Contrast 3 Contrast 6 = Contrast 2 x Contrast 3 (continued next page)

47 Example (cont.) A x C Interaction Contrast 7 = Contrast 1 x Contrast 4 Contrast 8 = Contrast 2 x Contrast 4 B x C Interaction Contrast 9 = Contrast 3 x Contrast 4 A x B x C Interaction Contrast 10 = Contrast 1 x Contrast 3 x Contrast 4 Contrast 11 = Contrast 2 x Contrast 3 x Contrast 4

48 Yikes! The process of generating contrast codes can lead to a large number of codes in a factorial design, the 4x3x2 design in the example in the book leads to 23 contrasts (4x3x2-1), 6 of which involve triple interactions. These are rarely all of a priori interest, let alone understandable.

49 Galloping (Type 1) Error Rates Remember from last semester, if we make many null hypothesis significance tests, and if H0 is true for all of them, then the probability of making at least one Type 1 error (rejecting H0 when H0 is false) becomes unacceptably high. Also, remember that the probability of making a type 1 error when H0 is true in any particular test equals our significance level, usually.05.

50 Again, from last semester, the following formula can be used to determine the error rate per comparison set for a specific number of independent comparisons: α PC = error rate per comparison, and α PCS = error rate per comparison set, and k=the number of independent comparisons that will be made, Then: α PCS =1 – (1- α PC ) K With 23 contrasts α PCS =1 – (1-.05) 23 =0.70

51 All 23 contrasts are independent of each other as they are orthogonal, if each comparison has a.05 chance of making a type 1 error then the error rate for the whole analysis equals.70. There is a good chance that some of these comparison will be significant just due to chance, and that in turn could cause you to spend a great deal of time and energy tying your theory (and your brain) into knots trying to explain them.

52 Solution Probably the best solution is to attend carefully to which of these contrasts have a priori interest, either because they replicate previous studies or because they test specific theories you are investigating. Before you gather your data make a list of which contrasts have a priori interest and make those the focus of your analysis. For those you can either simply use either.05 or (.05)/(# of a priori contrasts) as your significance level (I would choose the latter if there were more than just a few a priori comparisons), then use Scheffe’s for the other (a posteriori) comparisons. In determining how to handle error rate remember, you are trying to understand your data, not get brownie points for the number of significant results.

53 Solution (cont.) Any contrasts that look interesting a posteriori might play a role in what study you do next. There might be something really interesting that crops up, but be aware that it might well just be just due to chance, you might not want to invest a lot of thought and energy into pursuing it until you find if it can be replicated. You could also choose to just note any interesting contrasts and then see if they appear again in later studies before you investigate further.

54 Three Thoughts (1) While it may be a bother to work out all of the contrasts to do the equivalent of a factorial ANOVA, remember that you gain several things by doing so: 1.Deciding upon the comparisons helps insure that you are thinking quite specifically about your theory and what you are trying to test in the analysis.

55 Three Thoughts (2) 2.Comparisons give you much more specific information about what is going on among the groups, going beyond simply asking did anything happen? This seems particularly true to me as we work our way up into the interactions. Go back to our example with the complete summary table to see what we gained in understanding by having the contrasts in our analysis.

56 Three Thoughts (3) 3.Comparisons have more power than the overall (main) effects. If an independent variable is coded with three contrasts, then the test for the overall effect of that independent variable is based upon the PRE per parameter added (in this case three parameters, one for each contrast). If one of those contrasts is strong and the other two weak, then the PRE per parameter added for the independent variable considered as a whole may not be statistically significant.

57 More on Power So, if an independent variable is coded with more than one contrast, the tests for the statistical significance of each contrast will be more powerful than the test for the overall effect of the independent variable. There is, in addition, another statement to be made about power, and that is that the power of the test for any part of the analysis (individual contrasts or the overall effect of an independent variable) is likely to be greater in a factorial design than in a design that looks at each contrast, or each independent variable, separately. The following slides will (I hope) make this clear.

58 Evaluating the ‘Worthwhileness’ of X1 Model C: Ŷ i =β 0 Model A: Ŷ i =β 0 + β 1 X i1

59 Evaluating the ‘Worthwhileness’ of X1 in Context of X2 Model C: Ŷ i =β 0 + β 2 X i2 Model A: Ŷ i =β 0 + β 2 X i2 + β 1 X i1 Note the PRE of adding X1 is greater when X2 is already in the model

60 Redundancy Note that the PRE of adding X1 is reduced if it is redundant with X2 But, contrast codes are orthogonal, which means they will not be redundant! (as long as the N’s in each group are equal...more on that later).

61 Evaluating the ‘Worthwhileness’ of X1 in Context of X2, X3, X4, X5 Model C: Ŷ i =β 0 + β 2 X i2 + β 3 X i3 + β 4 X i4 + β 5 X i5 Model A: Ŷ i =β 0 + β 2 X i2 + β 3 X i3 + β 4 X i4 + β 5 X i5 + β 1 X i1 The other four contrasts reduced SSE(C), so while SSR from X1 didn’t get greater, it is a larger proportion of SSE(C).

62 Unequal N’s As in the previous chapter, there is no particular reason why the n’s in each treatment cell have to be equal, if they are not then: 1.SS for an independent variable will probably not equal the sum of the SS for its contrasts. 2.There could be a loss of power due to redundancies that appear between the contrasts due to the unequal n’s. It is important to look at the tolerances of the contrasts so that you can appropriate interpret a lack of significance (i.e. it could be due to low tolerance rather than to low effect).

63...and one final note. An advantage of the way the text lays out a two dimensional table into a row to help construct contrasts arose when I was helping out a student. His design looked like this... a1a2a3 b1 b2 It was a 3 x 2 design but the treatment group of A3B2 was meaningless, and he didn’t want to have it be part of the analysis. If you think about this in terms of a 2-factor ANOVA it is hard to figure out how to analyze the data (I couldn’t figure out how to exclude a cell when performing a factorial ANOVA).

64 However, if you lay the treatment combinations out in a line, like this, then there is nothing to it. There are five cells, just select four orthogonal contrasts that are of the greatest interest and analyze the data using linear regression. a1b1a2b1a3b1a1b2a2b2