1. Chapter 11 One-way ANOVA: Models with a Single Categorical Predictor
Psych 5510/6510
Chapter 11
One-way ANOVA: Models with a Single Categorical Predictor
Spring, 2009

2. Categorical Predictors
Using predictor variables that represent ‘nominal’ measures (scores that reflect qualitative or categorical differences).
This will allow us to extend our approach to designs that are traditionally analyzed with t tests (2 groups) or ANOVAs (2 or more groups)

3. Example
We want a model that predicts a team’s batting average (Yi) based upon whether it is in the National or American league (Xi: our categorical predictor variable).
If American League: Xi = -1
If National League: Xi = +1
As ‘league’ is a nominal variable, any two numbers could be used (e.g. Xi = 23 and 512).

4. Symbols
‘k’ will stand for the level of the predictor variable (i.e. which group of scores we are talking about, group 1 or group 2)
λk (lambda) will be the value assigned to that variable for that group. In this case:
λ1 = (i.e. Xi =-1 for the scores in group 1)
λ2 = +1 (i.e. Xi =+1 for the scores in group 2)
‘m’ will stand for how many groups there are (2 in this case)

5. Categorical variable ‘league’ has two values
(i.e. m = 2). For level 1 (i.e. k=1) we will use
a code of -1 (i.e. λ1 = -1) and for level 2 (i.e.
k=2) we will use a code of +1 (i.e. λ2 = +1).

6. Contrast Codes
Contrast codes (which we will be using) are simply one of the many possible coding schemes for numerically representing categorical data. Contrast codes have two conditions:
The sum of lambdas equals zero:
They are orthogonal (covered later)
The second condition will be covered later.
Note that we are summing the categories together, not the individual scores.
There are more scores in the American League than in the National League
so that if we added -1 for each score in the National League and =1 for each
score in the American league the sum will not be zero. While that will affect
our analysis it does not affect whether the coding is a contrast code, in this case
it is a contrast code because we are summing the lambdas (one for each category).

7. Estimation and Inference
MODEL C: Ŷi = β0
Where β0 is the mean batting average for all teams.
MODEL A: Ŷi = β0 + β1Xi
Where Xi indicates to which league the team belongs (Xi = -1 or 1)

8. Least Squared Estimates
MODEL C: Ŷi =
MODEL A: Ŷi = Xi
is the mean batting average of all the teams.
and 5.05 are the intercept and slope of the regression equation for predicting Y based upon X (see next slide)

9. Scatter Plot
Ŷi = Xi
National = -1
American = +1

10. Analysis SSE(C) = 3266.61 SSE(A) = 2608.1 SSR = 3266.61-2608.1=658.51
PRE to F Method
p=.02

11. Compare to Chapters 6 & 7
So far everything is the same as in Chapters 6 & 7.
1) The intercept is still the predicted value of Y when X is zero.
2) The slope is still the amount the predicted value of Y changes when X changes by one.
3) And PRE is the measure of how much we gain by moving from moving from MODEL C (just using the mean of all the scores) to MODEL A (incorporating the information of which group the score belonged to).
The use of categorical predictors, however, adds some additional interpretations.

12. Additional Interpretations
1) The predicted value of each score is the mean of its group.
For a National League team:
Ŷi = (-1) = which is the mean of the National League teams.
For an American League team:
Ŷi = (+1) = which is the mean of the American League teams.
2) The intercept (260.38) is the mean of the two group means. This is an unweighted grand mean.

13. Unweighted Grand Mean
The unweighted grand mean is simple the mean of the group means, it is not necessarily the mean of all the scores unless each group has the same number of scores.
E.g. Group 1: 100,100,100,100
Group 2: 0, 0
unweighted grand mean = 50
mean of the 6 scores = 66.67

14. Unweighted vs Weighted Grand Means

15. Additional Interpretations (cont)
3) The slope can be used to determine the difference between the two group means. Slope is how much the means differ as X changes by one. There is a difference of 2 between X=-1 and X=+1, so the difference between the two group means is the slope*2.
The slope = 5.05, so 5.05 x 2 = which is the difference between the mean batting average of the two leagues.
Or, you could use -0.5 and +.05 as your values of delta, then the least squares
solution would be Xi, and the slope would be the distance
between the two group means.

16. H0 and HA
Look back at the scatter plot, if the two group means were identical then the slope would be zero. The F test for the regression model:
MODEL C: Ŷi = β0
MODEL A: Ŷi = β0 + β1Xi
Can be stated as either:
H0: β1 = HA: β1  0, or as
H0: μ1 = μ HA: μ1  μ2, or as
H0: μ1 - μ2 = 0 HA: μ1 - μ2  0
Thus this is a way to test whether a statistically significant difference exists between the two group means. This is testing a non-directional alternative hypothesis (one that simply predicts the means will be different but does not predict which mean will be greater and which will be less).

17. And More...
While in this chapter we are focusing on how to do a ‘traditional t test’ for the difference between group means using multiple regression, don’t forget that this could be part of a larger model that can be tested using the model comparison approach. For example, our model of Y might include which group the person was in (variable X1) along with their score on some pretest (X2) and their height (X3). We will look at such models in a later chapter.

18. SSE
MODEL C
a.k.a “SS Total”.
MODEL A
a.k.a “SS Within Groups”

19. SSE (Cont), & df
SSR = SSE(C) - SSE(A) = = or
a.k.a: “SS Between Groups”
d.f
dfTotal = N-PC...this goes with SSTotal which is SSE(C)
dfWithin = N-PA...this goes with SSWithin which is SSE(A)
dfBetween = PA-PC...this goes with SSBetween which is SSR

20. Summary table (MS to F Method)
PRE = SSR/SSE(C) = .202
Remember: 1) a ‘MS’ is a SS divided by its d.f., and 2) F in this case is MSBetween/MSWithin

21. ‘t’ and ‘F’
The analysis of Model A provides a value of F* that is equivalent to what you would obtain by performing a traditional one-factor, independent groups, ANOVA.
If you take the square root of the F* you get the value of t* (the t obtained value if you analyzed this as a t test), the square root of Fcritical = tcritical, and the ‘p value’ for ‘t’ and that for ‘F’ are identical when you test a non-directional alternative hypothesis.

22. Directional Hypotheses
Testing a directional alternative hypothesis can also be done using this approach.
First, remember that in a test of a directional hypothesis the predicted direction of the difference between the two group means is expressed in HA. Example, you are testing a theory which predicts that group 1 will have a greater mean than group 2:
H0: μ1  μ2
HA: μ1 > μ2

23. Directional (cont.) Now analyze the data and look at the group means:
HA: μ1 > μ2
Now analyze the data and look at the group means:
If they support HA then the p value of that difference is the p value from the non-directional hypothesis divided by 2.
If they support H0 then do not reject H0 (the p value is actually 1- p/2, which is much greater than .5, let alone .05).

24. Alternative Coding Scheme: ‘Dummy Codes’
λ1 = 0
λ2 = +1
Least squared estimates using dummy coding:
MODEL A: Ŷi = Xi
Some things stay the same:
The predicted values of Y stay the same, they still come out to be the mean of the group the score is in.
Thus the error of the model is the same, as are the values of the SS’s, df’s, MS’s, F, PRE, p, and the decision regarding H0

25. Dummy Codes (cont.) MODEL A: Ŷi = 255.33 + 10.10Xi
While dummy coding does not change the results of the analysis, the interpretation of the slope and intercept do change:
The intercept is the mean of group 1 (as we set the value of X to λ1 = 0 for scores in group 1).
The slope is now equal to the difference between the two group means as the difference between λ1 (i.e. 0) and λ2 (i.e. 1) is one (remember that the slope is how much the prediction of Y changes as the value of X changes by one).

26. Assumptions
With regressional analysis we have the same assumptions about the populations as we did with the t test last semester.
The scores are normally distributed around the regression line (in this case the regression line passes through the group data at the mean of the group).
The variance of the scores doesn’t change along the regression line (in this case the variance of group 1 around its mean is the same as the variance of group 2 along its mean).
The scores are independent.
In addition, outliers can effect the regression line exactly the same way they effect the means in the t test.

27. Assumption of Normality
The scores are normally distributed around the regression line (i.e.
the residuals are normally distributed). From last semester we know that the analysis is ‘robust’ in terms of normality if N is large.

28. Assumption of Homogeneity of Variance
The variance of the scores around the regression line (i.e. the
residuals) is the same for both groups. From last semester we know
that this assumption is not important if we have an equal N in the groups.

29. Measure of Effect Size
Last semester we used Cohen’s d as a measure of the size of the effect of the independent variable when performing a t test for independent groups. This semester we are using PRE as our measure of effect size. PRE is a measure of how much the differences between the scores can be accounted for by knowing which group they were in (National League or American League), which is a way of measuring the size of the effect of the independent variable. PRE=.202

30. Categorical Predictors with More Than Two Levels
Type of feedback given on some pre-task, then similar task done again and performance measured.

31. Traditional ANOVA
Here is the overall F test to see if a difference exists somewhere between the means, using the ANOVA techniques covered last semester.
H0: μ1 = μ2 = μ3
Ha: at least one μ is different than the rest.

32. Traditional ANOVA
To perform an ANOVA in SPSS we create a dummy variable to indicate to which group each score belongs. SPSS knows that the dummy variable is a nominal scale and that it simply serves to indicate group membership.

33. Multiple Regression
This dummy variable won’t work with multiple regression, for it will be treated as a cardinal variable (i.e. that the ‘no feedback group’ is 1 more than the ‘failure group’, and that the ‘success group’ is 1 more than the ‘no feedback group’). ‘Group’ really is a nominal variable and needs to be treated as such.

34. Solution: Contrast Codes
With more than two groups one variable cannot be used to code the independent variable. With ‘m’ number of groups we need to come up with m-1 contrast codes to completely code the independent variable so that we can analyze the data using multiple regression.
That gets us to the second condition of contrast codes: 1)
2) If you have two or more contrast codes then they all must be orthogonal to each other.
(the first condition, covered earlier)

35. Comparisons
Before we start looking at whether or not the contrast codes are orthogonal let’s consider what each contrast code is doing, each code is making a ‘comparison’ between two aspects of our data (essentially asking a question that involves comparing two means).

36. Selecting Contrast Codes
Last semester we looked at the use of comparisons to answer specific questions in designs that have more than two groups. For each comparison we dropped groups, or added groups together, until we had just two things to compare. If ‘m’ is the number of groups, we knew that we could come up with m-1 number of orthogonal comparisons. We are doing exactly the same thing again, but this time we are doing it at the beginning of the analysis rather than after the overall F test has been completed. As you will see, there are some theoretical advantages to this new approach.

37. Our Contrast Codes
We have three groups (i.e. m=3) in our experiment, we need to come up with m-1 (i.e. 2) contrast codes that are orthogonal to each other. One possible set is given below (we will let λjk represent the value we plug in for X for using the jth contrast code on the scores in group k):
Code 1:
λ11 = λ12 = λ13 = 1
Code 2:
λ21 = λ22 = λ23 = 1
Where these codes came from will be explained in a minute, first I want to make sure you understand how they will be implemented.

38. Implementation
We are going to regress Y on X1 and X2, together X1 and X2 completely code our independent variable (type of feedback). The values of X1 and X2 implement our contrast codes.

39. Comparison One
Let’s say we are interested in whether the mean of group 1 (the ‘failure feedback’ group) is different than the mean of other two groups combined (a group consisting of the ‘no feedback’ and ‘success feedback’ groups). The hypotheses for this comparison would be:
H0: μ1 = (μ2 + μ3 )/2 or equivalently μ1 - (μ2 + μ3 )/2=0
The contrast codes could be any of the following:
λ11 = -1 λ12 = .5 λ13 = .5 or
λ11 = 1 λ12 = λ13 = -.5 or
λ11 = -2 λ12 = 1 λ13 = 1 etc.
I will be using this latter form, where whole numbers are used rather than decimals, this will have certain advantages.

40. Comparison Two
Let’s say we are also interested in whether the mean of group 2 (the ‘no feedback’ group) is different than the mean of group 3 (the ‘success feedback’ group). The hypotheses for this comparison would be:
H0: μ2 = μ3 or equivalently μ2 - μ3 = 0
The contrast codes could be any of the following:
λ11 = 0 λ12 = 1 λ13 = -1 or
λ11 = 0 λ12 = -1 λ13 = 1 etc.

41. Determining Orthogonality (cont)
Code 1:
λ11 = λ12 = λ13 = 1
Code 2:
λ21 = λ22 = λ23 = 1
Both codes fit the first condition of contrast codes, in that in both cases the values of delta add to zero.
Before we check computationally for orthogonality, think about it conceptually. If we compare group 1 with groups 2 and 3 combined, would that help us predict what we would find if we subsequently compared group 2 with group 3?

42. Determining Orthogonality (cont)
The second condition of contrast codes (that they be orthogonal to each other) is determined as follows;
1) For each group you times the lambda of one code by the lambda of the other code (i.e. multiplying downward).
2) The two codes are orthogonal if the sum of those products is zero.
The two codes are orthogonal because = 0

43. Another example
We have three groups (i.e. m=3) in our experiment, we need to come up with m-1 (i.e. 2) contrast codes that are orthogonal to each other. Here is a possible of codes:
Code 1:
λ11 = λ12 = λ13 = 1
Code 2:
λ21 = λ22 = λ23 = 1
Code 1 compares the mean of group 1 with the mean of group 3.
Code 2 compares the mean of group 2 with the mean of group 3.

44. Determining Orthogonality (cont)
Code 1:
λ11 = λ12 = λ13 = 1
Code 2:
λ21 = λ22 = λ23 = 1
Both codes fit the first condition of contrast codes, in that both cases the values of delta add to zero.
Before we check computationally for orthogonality, think about it conceptually. If, for example, we found a significant difference between group 1 and group 3, would that influence your guess about whether or not there may also be a difference between group 2 and group 3?

45. Determining Orthogonality (cont)
The second condition of contrast codes (that they be orthogonal to each other):
The two codes are not orthogonal because
doesn’t equal 0.

46. Coding Categorical Variables
Contrast Coding: To completely code a categorical variable for multiple regression requires m-1 contrast codes. Thus a categorical variable with 3 levels (as in the previous example) can be completely coded with 2 contrast codes. A categorical variable with 4 levels can be completely coded with 3 contrast codes, etc.
Dummy Coding: You can code a categorical variable with any number of groups using just one variable using dummy coding (X=1 if in group 1, X=2 if in group 2, X=3 if in group 3, etc.) if the only purpose of the variable is to inform your statistical program which scores go into which group in an ANOVA, if you want to use multiple regression to analyze the data then go with contrast codes. We will always use contrast coding in this class.

47. Coding 4 levels
Four levels requires 4-1=3 contrast codes to completely code it. The
requirement is that every pair of codes be orthogonal (i.e. λ1k is orthogonal
to λ2k , and λ1k is orthogonal to λ3k , and λ2k is orthogonal to λ3k ).
It is impossible to come up with a set of more than m-1 codes that are all
orthogonal to each other.
One possible set of orthogonal codes

48. There are many possible sets of codes to use
Another set of orthogonal codes. This has the same general pattern as
the previous set, it may be easier to simply use the previous set and then
think about which group you want to call ‘Level 1’, which group to call
‘Level 2’, etc.

49. There are many possible sets of codes to use
A very different set of orthogonal codes. Note that once you have determined
all but the last code there is no freedom, there will be only one code left
that will complete the orthogonal set.

50. How to Select a Set of Codes
There are often many different sets of contrast codes from which you may choose. Select the one that contains the comparisons that best fit your a priori questions. For example, if you think that the ‘failure’ level should lead to worse performance than the ‘no feedback’ or ‘success’ level, then select a set of contrast codes that contains the contrast which compares the ‘failure level’ with the average of the other two.

51. The Big Picture
Where we are heading with this: we will be able to do the equivalent of an overall F test to determine whether a difference exists somewhere between the levels, but we are also going to be able to test the significance of specific contrasts, thus answering more specific questions at the same time.

52. More Options
If you have no a priori reason to select a specific set of contrast codes you
can always follow this general pattern (which we have seen in
previous examples) which can be adapted to fit any number of levels.
This pattern is known as a set of Helmert contrast codes.

53. More Options: Trends Group 1 Group 2 Linear -1 1 Two-Group Experiment
If your independent variable is an ordinal scale, rather than a nominal
scale, then you can select ‘orthogonal polynomial contrast codes’ to check
whether various sorts of orderings (e.g. linear, quadratic, cubic, etc.) exist between
your categories. What you can test for depends upon how many categories you have: two
categories can only detect a linear relationship, three categories can detect a
linear or a quadratic relationship, etc. In the following slides note that the contrast codes
are indeed orthogonal.
Two-Group Experiment
Group 1
Group 2
Linear -1 1

54. Group 1 Group 2 Group 3 Linear -1 1 Quadratic 2 Group 1 Group 2
Three-Group Experiment
Group 1
Group 2
Group 3
Linear -1 1
Quadratic 2
Four-Group Experiment
Group 1
Group 2
Group 3
Group 4
Linear -3 -1 1 3
Quadratic
Cubic

55. Five-Group Experiment

56. Parameter Estimation with a Multilevel Predictor Variable
Back to our example with a three level predictor variable (‘failure’, ‘no feedback’, ‘success’).
Code 1:
λ11 = λ12 = λ13 = 1
Code 2:
λ21 = λ22 = λ23 = 1
First contrast the mean of ‘failure’ with the average of the other two means, then contrast the ‘no feedback’ with ‘success’

57. Data
The data and the two contrast codes.

58. The following parameters result when we regress Y on X1 and X2:
MODEL A: Ŷi = Xi Xi2
If you use a complete set of m-1 contrast codes then:
1) The predicted value for each Y is the mean of the group it is in.
2) The intercept is the unweighted grand mean of the group means.
3) The slopes can be used to determine the difference in the means involved in each contrast.
X1 compares the mean of group 1 with the mean of the other two groups combined. The values of X1 are –2 and 1. The effect on Y of X1 changing by 1 is .35, since X1 changes by 3 when it moves from group 1 to the other two groups combined, then the difference in the means in that contrast is equal to .35x3=1.05
X2 is little easier to understand, it simply compares the mean of group 2 with the mean of group 3. The values of X2 are 0, -1, and 1. The effect on Y of X2 changing by 1 is .31, since X2 changes by 2 when it moves from group 2 to group 3, then the difference in those means is equal to 2x.31=.62

59. Inference with a Multilevel Predictor Variable
The first test we will look at is the overall test comparing a model that just uses the mean of Y to a model that uses both contrasts. Remember that the two contrasts completely code the three groups in the experiment, thus this is the equivalent to testing to see if the independent variable had an effect.
MODEL C: Ŷi = β0
MODEL A: Ŷi = β0 + β1Xi1 + β2Xi2
Model A (for any complete set of m-1 contrast codes) uses the mean of the group as a prediction for each score. If that reduces error compared to using the mean of all the scores (Model C) then that tells us that at least one group mean must differ from the mean of all the scores. Thus H0 can be written two ways:
H0: β1 = β2 = 0, (model comparison approach), or equivalently…
H0: μ1 = μ2 = μ3 (ANOVA approach)

60. Summary Table
Note that dfBetween = 2 (i.e. 1 d.f. for each contrast, or m-1).
Also note that the test for an overall effect of the independent variable is not statistically significant. The SS, df, MS, F, and p values are identical to when we analyzed the data with an ANOVA
(see the earlier slide).

61. A More Detailed Analysis
The previous analysis is the equivalent of the ‘overall F test’ to see if a difference exists somewhere among the group means. This has two disadvantages:
It doesn’t tell us where that difference lies.
It lacks power. By plugging all of the contrast codes in at once we are weakening our ability to see if one or more were worthwhile by themselves. I.e. Model A in the overall analysis has one parameter for each contrast, the PRE per parameter added may be less than the PRE for any individual contrast.
It will be more informative and more powerful to look at each contrast individually.

62. Testing Contrast #1
The general approach will be to use the full set of contrast codes as the augmented model, and all but the contrast code of interest as the compact model. Remember in this contrast X1 compares the mean of group 1 with the mean of groups 2 and 3 combined.
Test 1:
Model C: Ŷi = β0 + β2Xi Ŷi = Xi2
Model A: Ŷi = β0 + β2Xi2 + β1Xi Ŷi = Xi Xi1
H0: β1 =0, or alternatively,
H0: μ1 = (μ2 + μ3 )/2 or μ1 - (μ2 + μ3 )/2=0
Note:
1) PA-PC =1, thus this is a ‘Single-degree-of-freedom test’
2) The intercept and the slope for X2 don’t change from Model C to Model A, this is because the lambdas for X1 and X2 are orthogonal (i.e. contrast codes were used and the groups have equal n’s), this causes X1 and X2 to be completely nonredundant predictors.
3) The statistical analysis here is simply that of the partial regression coefficient.

63. From SPSS SPSS will give us the following for Contrast 1 (i.e. X1):
Partial regression coefficient (i.e. ‘b1’)=0.35
Partial correlation coefficient = .453
PRE = .453² = .205
t = 2.326
p = Remember, this p value is for many equivalent tests: whether the partial regression coefficient = 0; whether the partial correlation coefficient = 0; whether the PRE is statistically significant. Contrast 1 is statistically significant (worth adding to our model of Y).
From this we can also compute:
F = t² = 2.326² = 5.41
SPSS does not give us the SSR for the contrast but it can easily be computed. SScontrast = MSResidual x Fcontrast = x 5.41 = 6.021

64. Testing Contrast #2
Contrast X2 compares the mean of group 2 with the mean of group 3.
Test 1:
Model C: Ŷi = β0 + β1Xi Ŷi = Xi1
Model A: Ŷi = β0 + β1Xi1 + β2Xi Ŷi = Xi Xi2
H0: β2 =0, or alternatively,
H0: μ2 = μ3 or μ2 - μ3 = 0
From SPSS:
Partial regression coefficient (i.e. ‘b2’)=0.31
Partial correlation coefficient = .25
PRE = .25² = .063
Fcontrast = t²contrast = 1.185² = 1.40
SScontrast = MSresidual x Fcontrast = x 1.4 = 1.56
p = (not statistically significant).

65. Full Summary Table
Note that what we have done is to partition the SSbetween groups (7.68 = ) into that which is accounted for by contrast 1 and that which is accounted for by contrast 2. SSX1 + SSX2 = SSbetween only when the contrasts are orthogonal.

66. Full Summary Table
The overall F test was not statistically significant (p=.052), Contrast 1 was (p=.03), Contrast 2 was not (p=.249). The overall F test had the highest PRE but remember that the F value is based upon the PRE per parameters added, and the overall F test had two parameters (b1 and b2), the second parameter (b2) watered down the effect of the first parameter (b1)

67. Thoughts
Using multiple regression to perform an overall ANOVA requires us to first come up with a set of contrasts, this has a couple of advantages.
The first advantage is that each comparison has more power than the overall F as the comparison involves testing the addition of just one parameter. In our example if we had only performed the overall F test we would have simply concluded that the effect of our independent variable was not statistically significant.

68. Thoughts
The second advantage is conceptual. If we are satisfied in knowing that our independent variable had some sort of effect (i.e. H0 is rejected in the overall F test) than we are not doing very good science. Having to come up with a set of comparisons requires us to think about our experiment in a much more detailed way, leading to much more specificity in our theory building.

69. Other Contrast Codes
If we had selected a different set of contrast codes then the only
thing that would have changed is the information in the table
indicated by ... The choice of which set of contrast codes to use
should be based upon which set best answers your a priori questions.

70. Contrast Codes with Unequal N
A slight complication arises if you do not have an equal number of scores in each level of your categorical variable (i.e. the group n’s are not equal).
With unequal n’s your contrast coded predictor variables become somewhat redundant.

71. Unequal n’s (cont.) How this shows up:
1) The SS for the contrast codes no longer add up to equal SSbetween.
2) The value of the parameters change as you add more contrast coded predictors (e.g. the value of β1 may change from Model C to Model A).
3) You lose a little power, for redundancy takes away from the power of adding a new predictor.

72. Unequal n’s (cont.) Bottom Line:
1) You lose a little power, so it is better to have equal n’s if you can.
2) Everything concerning how you proceed and how you interpret the results stays the same, just don’t expect the SS of your contrast codes to sum up to equal the SS between groups (because of redundancies).

73. Nonorthogonal Comparisons
It might be the case that you can’t find one set of contrast codes that contain all of the comparisons you would like to perform (i.e. some of the comparisons you want to perform are not orthogonal with other comparisons you also want to perform).

74. Nonorthogonal Contrasts: The Problem of Error Rate
The problem with performing nonorthogonal comparisons is that of increasing error rate (Type 1 errors). The requirement that comparisons be orthogonal limits error rate in two ways:
1) It limits the number of contrasts being made to m-1.
2) The requirement of orthogonality decreases the ability of one weird mean to lead to many Type 1 errors.

75. Nonorthogonal Comparisons: Step 1
Write down all of the comparisons in which you have an a priori interest. Find a set of orthogonal contrast codes that contains the greatest number of those comparisons (or perhaps the set that contains the most important comparisons you would like to make). Proceed as previously described, don’t make any adjustments, interpret normally.
Now look at the comparisons you have left to perform…

76. Nonorthogonal Comparisons: Step 2
For every comparison that is not in your original set of contrast codes, create a complete* set of contrast codes that contains that comparison. Run that set like you did your original (getting a value of F and p for the contrast of interest), and then make one of the following adjustments...
* Make sure when you run this new analysis that your new Model A has a complete set of m-1, orthogonal, contrast codes. If you can find a set of contrast codes that contain more than of of these left-over desired comparisons that would be great.

77. Nonorthogonal Comparisons: Step 3a
If this comparison (the one that was nonorthogonal to the original set) is a ‘planned comparison’ then adjust its p value using the following formula:
k = the total number of contrasts being made on the data (include the contrasts made in the original set)
Then use (your significance level) / k as significance level for that contrast. For example if you are making 3 contrasts altogether, then the p value for the nonorthogonal contrast must be less than or equal to .05/3 = .017 to be statistically significant.
a.k.a.: Bonferronni method or Dunn method

78. Nonorthogonal Comparisons: Step 3b
If the nonorthogonal comparison is a ‘post hoc comparison’ then adjust the value of Fcritical for this contrast using the following formula:
new Fcritical = (m-1) Fcritical with m-1, n-PA df
Note: if you make every possible comparison using this new Fcritical value the probability of making at least one Type 1 error is equal to your significance level. Also, if the overall F test is significant than at least one comparison will be, if not then no comparison will be.
a.k.a.: Scheffe Method