1. Step 1: Specify a null hypothesis
We have 8 books and we wish to see whether the books differ on the basis of their average sentence count. If i represents the mean of the ith book, then our null hypothesis is
Statistical Data Analysis - Lecture11 28/03/03

2. Step 2: Specify an alternative hypothesis
Statistical Data Analysis - Lecture11 28/03/03
Step 2: Specify an alternative hypothesis
The null hypothesis is a fairly general one. The reason for this is to avoid multiple comparison problems.
To reject the null hypothesis, only one mean has to differ, i.e.
Statistical Data Analysis - Lecture11 28/03/03

3. Statistical Data Analysis - Lecture11 28/03/03
Model fit diagnostics
We did some “pre-fit” checking of the assumptions of ANOVA
However, we need to do some further checks after fitting our model
Both of these posthoc methods involve examination of the residuals
Recall the residuals are the small “errors” left after we subtract the estimated group means from each observation
The first plot is called a “pred-res” plot, or a residuals vs.. predicted plot.
Statistical Data Analysis - Lecture11 28/03/03

4. Statistical Data Analysis - Lecture11 28/03/03
Pred-res plots
Pred-res plots are highly patterned in a sense for ANOVA.
The reason is that the fitted value is the same for each observation in a group, i.e. it is the group mean
So, we expect to get a plot with the group means on the x-axis
Recall that we assumed that all the residuals have mean zero and standard devotion . Furthermore, we assumed that the residuals are normally distributed
Therefore we expect to the points for each group to be
roughly centred around zero (symmetric)
spread out about the same amount
not to have too many extreme points
Statistical Data Analysis - Lecture11 28/03/03

5. Statistical Data Analysis - Lecture11 28/03/03
1.8
2.0
2.2
2.4
2.6 -3 -2 -1 1 2
Fitted values
Residuals
Residuals vs. Fitted
300 41
286
aov(formula = log.length ~ book)
Statistical Data Analysis - Lecture11 28/03/03

6. Interpreting pred-res plots
We can see that things are basically okay with our assumptions
The data are symmetric around zero for the most part
There don’t appear to be any extreme values
The WGMS is Remember this is an estimate of the variance of the residuals, so the standard deviation is the square root of the WGMS.
This is 0.712, so we expect 99.9% of the residuals to be with in 3 standard deviations, or within +/
There is one observation that exceeds this bound. If we divide by the square root of the WGMS, we get an approximate Z-score of about –3.66.
This says that this point is 3.66 std. deviations from the mean, which is extreme, but not too bad.
Statistical Data Analysis - Lecture11 28/03/03

7. Statistical Data Analysis - Lecture11 28/03/03

8. Statistical Data Analysis - Lecture11 28/03/03
Norplots of residuals
Here is a situation where we need the use of a norplot
We use norplot to test the assumption that the residuals are normally distributed
From the pred-res plot we can determine things like symmetricity of the data and inequality of variance
The calculations on the previous slide are interesting, but in fact similar results are guaranteed by Komologorov’s theorem
A norplot plots the the theoretical quantiles from the normal distribution vs. quantiles for the residuals (which are really just the order statistics). A straight line => Normality
Statistical Data Analysis - Lecture11 28/03/03

9. Statistical Data Analysis - Lecture11 28/03/03

10. Interpreting the norplot
From the norplot,
even though the aspect ratio is not 1:1
we can see that the data pretty much follow a straight line,
therefore our assumption of normality is sound’
There is one outlying point, and it may be interesting to investigate the effect of removing it from the fit
Identifying points in R
Easiest way is to make a vector of labels and then use the text function to add the labels to the plot
Statistical Data Analysis - Lecture11 28/03/03

11. Interpreting the output
We saw a P-value in our ANOVA table of 1.055E-14 this is a very strong implication that there is a difference between the authors.
How do we find the differences?
They’re just the group means which we can get using the apply (or sapply) function
Whilst we might use this as a rough guide to identifying the differences, we really run into a multiple comparison problem
Statistical Data Analysis - Lecture11 28/03/03

12. Multiple comparisons problem
Every time we perform a hypothesis test, we set ourselves an artificial critical point or significance value, like 95% or 99%.
In setting such a significance value we accept a certain Type I error rate or size. For example, if we use 95% we accept that we will reject the null hypothesis when it is actually true on average about 5% of the time.
In a one way ANOVA we initially carry out one hypothesis test. So we have (assuming the use of a 95% significance level) a 5% chance of making the wrong conclusion if the null hypothesis is true
An acceptable level?
Statistical Data Analysis - Lecture11 28/03/03

13. Multiple comparisons problem
If we stopped at that point, then yes, the Type I error rate would be fine.
However, it is usually not sufficient to know that there is a difference, we want to know where the differences are
We could compare pairs of groups at at time and look for differences that way.
Is this a good thing to do?
Yes and no.
Every time we compare two groups, we have the same 5% chance of making an incorrect conclusion
Because these tests are essentially independent, the number of Type I errors is binomially distributed
Statistical Data Analysis - Lecture11 28/03/03

14. Type I error and Bonferroni corrections
E.g for 5 groups, there are 5*4/2=10 possible comparisons, so the probability of making at least one mistake has risen to 40%!
In general if there are k pairwise tests and the Type I error is  then the probability of making at least one mistake is
In general if there are k pairwise tests and the Type I error is  then the probability of making at exactly one mistake is
This can be approximated by k×
Statistical Data Analysis - Lecture11 28/03/03

15. Type I error and Bonferroni corrections
What can we do about it?
One solution is called the Bonferroni correction. It involves dividing the Type I error rate by k. E.g. if we want an overall Type I error rate of 5% and we have 10 tests to carry out we set  = 0.05/10 = 0.005
This test is very conservative
A conservative test means that it is very difficult to reject the null hypothesis.
Statistical Data Analysis - Lecture11 28/03/03

16. Conservative tests and LSD intervals
One analogy is that if your pants would have to be on fire before you noticed your house was burning down
Bonferroni, especially for large numbers of groups, is far too conservative to be acceptable.
What other procedures are there?
One other is known as Fisher’s Protected Least Significant Difference or LSD intervals
Statistical Data Analysis - Lecture11 28/03/03

17. Fisher’s protected LSD
Fisher’s method is as follows
If you have rejected the ANOVA null hypothesis of no difference between the means,
then calculate confidence intervals for the pairwise differences between the means
If two intervals overlap, then there is no evidence to suggest that there is a difference between the two groups (or treatments)
If two intervals do not overlap, then there is some evidence at the  level of significance.
How do we construct the intervals
Statistical Data Analysis - Lecture11 28/03/03

18. Statistical Data Analysis - Lecture11 28/03/03
LSD intervals/plots
For each of our k groups we calculate
where
We usually plot these intervals for each group and make our interpretations from there
This is a conservative approach and works well when the groups have similar numbers of observations.
If the groups have less than 10 observations in each group and the numbers in each group differ greatly then this rule should be used with caution.
Statistical Data Analysis - Lecture11 28/03/03

19. Interpreting LSD plots
For every group we have an LSD interval. We compare one of these intervals with an LSD interval for another group. This is an approximation of performing a two sample t-test for a difference in the group means. There are two things that can happen: (a) the intervals overlap and (b) the intervals don’t overlap.
When the intervals overlap (and given certain assumptions which we will discuss later) we can say at the 5% level of significance that the data provide no evidence against the null hypothesis, i.e. the data provide no evidence of a difference in the group means.
Statistical Data Analysis - Lecture11 28/03/03

20. Non-overlapping intervals
When the intervals don’t overlap (and given the same conditions), we can say at the 5% level of significance that the data provide sufficient evidence against the null hypothesis, i.e. the data provide evidence to suggest a difference in the means of the two groups.
Statistical Data Analysis - Lecture11 28/03/03

21. Assumptions in LSD interval comparisons
 When we use an LSD plot to make a pairwise comparison we assume two things
The numbers of observations per group are fairly similar – i.e. balanced or “near balanced” designs are better
The standard errors of the groups are approximately equal.
If there is any doubt about either of these assumptions, then any conclusions about differences or lack thereof should be backed up with a proper two sample t-test.
Statistical Data Analysis - Lecture11 28/03/03