Testing Differences Among Several Sample Means Multiple t Tests vs. Analysis of Variance

Several Sample Means What might we do if we had more than two samples? X 11 X 12. X 1n X 21 X 22. X 2n Sample 1Sample 2 means: X 31 X 32. X 3n Sample 3

Several Sample Means Specifically how many t Tests could you do? X 11 X 12. X 1n X 21 X 22. X 2n Sample 1Sample 2 means: X 31 X 32. X 3n Sample 3

Several Sample Means Specifically how many t Tests could you do? X 11 X 12. X 1n X 21 X 22. X 2n Sample 1Sample 2 means: X 31 X 32. X 3n Sample 3

Several Sample Means Specifically how many t Tests could you do? X 11 X 12. X 1n X 21 X 22. X 2n Sample 1Sample 2 means: X 31 X 32. X 3n Sample 3

Several Sample Means Specifically how many t Tests could you do? X 11 X 12. X 1n X 21 X 22. X 2n Sample 1Sample 2 means: X 31 X 32. X 3n Sample 3

Multiple t Tests and Family- wise Error Rate If you do all possible pair-wise comparisons (C), what happens to the overall probability of making a Type I error?

Multiple t Tests and Family- wise Error Rate If you do all possible pair-wise comparisons (C), what happens to the overall probability of making a Type I error? Family-wise Error Rate =

Multiple t Tests and Family- wise Error Rate How could we prevent the family-wise error rate from exceeding.05 ?

Multiple t Tests and Family- wise Error Rate How could we prevent the family-wise error rate from exceeding.05 ? Set the alpha-level for each pair-wise t test to be a fraction of.05; specifically:

Multiple t Tests and Familywise Error Rate This isn’t usually done in practice because only a few different alpha-levels appear in the t tables

Multiple t Tests and Familywise Error Rate This isn’t usually done in practice because only a few different alpha-levels appear in the t tables More importantly, consider that C increases dramatically as more samples are added –for 4 samples: C = 6 –for 5 samples: C = 10 –for 6 samples: C = 15 Which leads to a precipitous drop in power

Analysis of Variance What is needed is a technique that controls family-wise error rate while looking for one or more differences between several sample means

Analysis of Variance What is needed is a technique that controls family-wise error rate while looking for one or more differences between several sample means That technique is a one-way Analysis of Variance (ANOVA)

Analysis of Variance Here are three samples, each are measurements under different treatment conditions: X 11 X 12. X 1n X 21 X 22. X 2n Sample 1Sample 2 means: X 31 X 32. X 3n Sample 3 Each sample has a mean and variance and the 3 means are a sampling distribution of means

Analysis of Variance What would the null hypothesis be? What would the alternative hypothesis be? X 11 X 12. X 1n X 21 X 22. X 2n Sample 1Sample 2 means: X 31 X 32. X 3n Sample 3

Analysis of Variance What would the null hypothesis be? –All three samples are taken from the same population so:

Analysis of Variance What would the null hypothesis be? –All three samples are taken from the same population so: What would the alternative hypothesis be? –At least one of the samples is from a different population and hence has a different mean

Analysis of Variance We can estimate the variance of the “null hypothesis” population by averaging the j variance estimates

Analysis of Variance This is called the “Mean Square Error” or “Mean Square Within” in our example:

Analysis of Variance MS error is an estimate of the population variance

Analysis of Variance MS error is an estimate of the population variance What’s another way we could estimate the population variance (hint: assume the null hypothesis is true)?

Analysis of Variance Each sample has a mean and variance and the 3 means are a sampling distribution of means X 11 X 12. X 1n X 21 X 22. X 2n Sample 1Sample 2means: X 31 X 32. X 3n Sample 3

Analysis of Variance Recall that we estimated the variance of a sampling distribution of means (since we only had one sample) using the equation:

Analysis of Variance Now we’ve got more than one sample! So we can turn this equation around and make an estimate of the population variance called the “Mean Square Effect” or “Mean Square Between”:

Analysis of Variance We now have two different estimates of the population variance: MS error and MS effect Why might these two estimates disagree?

Analysis of Variance MS error is based on deviation scores within each sample but…

Analysis of Variance MS error is based on deviation scores within each sample but… MS effect is based on deviations between samples

Analysis of Variance MS error is based on deviation scores within each sample but… MS effect is based on deviations between samples MS effect would overestimate the population variance when…

Analysis of Variance MS error is based on deviation scores within each sample but… MS effect is based on deviations between samples MS effect would overestimate the population variance when…there is some effect of the treatment pushing the means of the different samples apart

Analysis of Variance We compare MS effect against MS error by constructing a statistic called F

Analysis of Variance We compare MS effect against MS error by constructing a statistic called F If the hull hypothesis: is true then we would expect: except for random sampling variation

Analysis of Variance F is the ratio of MS effect to MS error

Analysis of Variance F is the ratio of MS effect to MS error If the null hypothesis is true then F should equal 1.0

Analysis of Variance Of course there is a sampling distribution of F - if you repeated your experiment many times you would get a distribution of Fs

Analysis of Variance Of course there is a sampling distribution of F - if you repeated your experiment many times you would get a distribution of Fs The shape of that distribution depends on two different degrees of freedom: –MS effect has k-1 degrees of freedom –MS error has k(n-1) degrees of freedom

Analysis of Variance We can look up a critical F from an F table for any given number of degrees of freedom

Analysis of Variance We can look up a critical F from an F table for any given number of degrees of freedom If the F statistic we’ve obtained in our experiment exceeds F crit then we know that fewer than 5% of such experiments would be likely to obtain this F statistic if the null hypothesis was true

Analysis of Variance We can look up a critical F from an F table for any given number of degrees of freedom If the F statistic we’ve obtained in our experiment exceeds F crit then we know that fewer than 5% of such experiments would be likely to obtain this F statistic if the null hypothesis was true So we can reject the null and conclude that at least one pair of means is different