2. Testing Hypotheses about Differences among Several Means

3. The t-test compares independent means TWO AT A TIME. Whenever we encounter a research situation in which we need to compare THREE or MORE independent-sample means, we would have to make multiple t-tests. This is clear from the numerator of the t-test:

4. Statistically, this is undesirable. Recall that each t-test is conducted with a chance of incorrectly rejecting the null hypothesis. The magnitude of this chance is determined when we decide upon an alpha level. With alpha set at 0.05, there is a 5 percent chance of committing a Type I error each time we perform a t-test. When we perform 100 t-tests, we will INCORRECTLY reject the null hypothesis 5 times. Clearly, the chance of error increases in proportion to the number of t-tests we make.

5. To test the significance of difference among three or more randomly-selected subsamples, we must use a statistic known as the analysis of variance, or ANOVA. for short. The analysis of variance assumes that variations among scores come from only two sources: (1) random variations among the units sampled; and (2) variations across the treatments to which these units have been exposed. Algebraically: Y ij =  +  j X ij +  ij Here,  j is the symbol for the effect of exposure to the treatment (the X ij -variable) and  ij is the symbol for the variations among sampled units. The constant, , will be explained later. This is the expression for the general linear model.

6. Suppose we have four sections of PPD 404. We wish to know whether or not the quality of instructors (X ij ) makes a difference in the way students perform in this course. To measure performance, we give the same final examination (Y ij ) in all four sections. If only variations among students exist (  ij ), the four mean final examination scores should be about the same. However, if instructors make a difference (  j ), there should be a statistically significant difference among the four means. How great this difference would have to be before we rule out chance can then be decided in the usual way (i.e., by selecting an alpha level, identifying an appropriate sampling distribution, defining a region of rejection, locating the critical value, etc.)

7. In the example, exposure to the four individual PPD 404 instructors is the treatment, symbolized by X ij. The effect of instructors is represented by  j. Differences among individual students make up the “error” term, symbolized by  ij. If instructors do not vary in their effect on student performance, then the value of the treatment should be zero, i.e.,  j = 0.0. Therefore, the student test scores could be "predicted" statistically as follows: Y ij =  +  ij That is, each student's score could be predicted by knowing the value of  (the average exam performance of all PPD 404 students in general) plus knowing how students differ from one another in every respect except their instructor.

8. On the other hand, if instructors DO make a difference,  j would NOT equal 0.0, and our prediction would need to include the  j X ij component as well: Y ij =  +  j X ij +  ij The null hypothesis that we test (indirectly) is H 0 :  1 =  2 =  3 =...  j where j = number of categories of X ij.

9. In performing an analysis of variance, we do not test the differences among means directly. Instead, we make two different estimates of the population variance. One estimate of the population variance includes the subgroup means, the other includes only the differences among individual scores. By comparing the two estimated variances, we can decide if the subgroups differ significantly. Remember: Means are imbedded in the construction of the variance (i.e., deviations about the mean are the numerator of the variance statistic):

10. Notice that the calculation of the variance is the sum of squared deviations about the mean (sum of squares, or ss) in the numerator and degrees of freedom (df) in the numerator. Also, you should know that another name for the variance is the mean square. Thus, the variance can be rewritten as Sum of Squares ss Mean Square = —————————— = —— Degrees of Freedom df

11. It helps to understand that the analysis of variance developed for the analysis of results from experiments. Some subjects in experiments are “exposed” to a treatment (the “test group”) while other subjects were not so exposed (the “control group”). At the end of the experiment, the two groups are expected to differ on some measure (Y ij ). (Usually, more than one control group was involved, hence the inability to apply the t- test.) The key to any experiment is that the test and the control groups must differ only by chance. That is, there can be no systematic differences between the two groups, only random differences.

12. Control Group Test Group X ij t1t1 t2t2 R R Y ij

13. Subjects are (1) randomly selected; (2) randomly assigned to the test and control groups; and (3) conditions during the experiment are controlled in a laboratory. Thus, any differences between the two groups at the end of the experiment (t 2 ) could arise from only one of two sources: (1) chance differences in the selection and assignment of subjects; or (2) the differing effects of treatment versus non-treatment. With randomization (and proper experimental design), these are the only two possibilities. Chance differences are the random differences among subjects, referred to as random "errors." Systematic differences could only come from treatments.

14. If we calculate the mean for all subjects, we have a measure of average outcome that includes BOTH differences among subjects AND whatever effect the treatment had. That is, the overall mean cannot distinguish between these two sources of variation. In the analysis of variance, this statistic is called the grand mean so that it is not confused with other means. In the general linear model, the grand mean is the term, . With observations divided into j subgroups, the calculation of the grand mean is

15. The variance for the deviations of the individual scores about the various subgroup means represents the error term,  ij, and must be calculated for each subgroup, then added together. We lose one degree of freedom for each subgroup we have created, hence this calculation is This is called the mean square (variance) within, indicating that it captures the variations WITHIN groups without capturing the differences AMONG groups.

16. The second estimate of the population variances treats GROUPS MEANS as scores and computes variance in terms of the deviations of these means from the grand mean. This is called the mean square (variance) between (not grammatically correct) and is calculated as: This estimate contains BOTH the effects of individual differences (they are imbedded in the values of the subgroup means) and effects of the treatment (in the difference, if any, between the average group score and the grand mean).

17. If the groups ARE NOT significantly different, then Mean Square Between = Mean Square Within Hence, the ratio of the former to the latter will equal 1.0. If the groups differ, this difference will be reflected in their means, and Mean Square Between > Mean Square Within Hence, the ratio of the former to the latter will be GREATER THAN 1.0. How much greater does the Mean Square Between have to be relative to the Mean Square Within for the sample before we conclude that the differences hold in general and are not an artifact of chance (in random selection and random assignment)?

18. Hence, the test is known as the F-test or the F-ratio and statistically is Mean Square Between F = —————————— Mean Square Within

19. Two helpful things to know are that the sums of squares form an identity and the degrees of freedom form an identity. That is, Total SS = Between SS + Within SS and Total df = Between df + Within df Thus, the calculation of mean squares can be simplified by calculating the two simplest sums of squares and subtracting for the third. Next time, an example.