Department of Cognitive Science Inferential Statistics II: Introduction to ANOVA Michael J. Kalsher Department of Cognitive Science MGMT 6970 PSYCHOMETRICS © 2012, Michael Kalsher

ANOVA: A Framework Understand the basic principles of ANOVA Why it is done? What it tells us? Theory of one-way independent ANOVA Following up an ANOVA: Planned Contrasts/Comparisons Choosing Contrasts Coding Contrasts Post Hoc Tests Writing up Results

Why ANOVA? t tests are limited to situations in which there are only two levels of a single independent variable or two associated groups. There are many instances in which we’d like to compare more than two levels. But … performing multiple t tests can inflate the Type I error rate. The concept of “Omnibus test.” Familywise error: 1 - (.95)n

What is ANOVA Tests the null hypothesis that three or more means are roughly equal. Produces a test statistic termed the F-ratio. Systematic Variance (SSM) Unsystematic Variance (SSR) The F-ratio tells us only that the experimental manipulation has had an effect—not where the effect has occurred. -- Planned comparisons (before you collect the data) -- Post-hoc tests (after-the-fact “snooping”). -- Both conducted to control the overall Type I error rate at 5%. -- Guards against familywise error [1 - (.95)n ] -- For a study with 3 conditions, Type 1 error rate = 14.3%!

ANOVA as Regression Suppose we are interested in determining whether the drug Viagra is effective at making someone a better lover. A group of men are then randomly assigned to one of three groups: (1) Placebo (sugar pills); (2) Low-dose Viagra; or (3) High-dose Viagra. The dependent measure is an “objective” measure of libido. The data are below:

Outcomei = (model) + errori ANOVA as Regression To predict levels of libido from the different levels of Viagra, we can use the general equation: Outcomei = (model) + errori

Model for the Placebo group (base category) We are predicting the level of libido when both doses of Viagra are ignored. Thus, the predicted value will be the mean of the placebo group. The intercept, b0, is always equal to the mean of the base category

Model for the High-dose group b2 represents the difference between the means of the High-dose group and the Placebo (base) group.

Model for the Low-dose group b1 represents the difference between the means of the Low-dose group and the Placebo (base) group.

Regression Output Conclusion: Using group means to predict the outcome is significantly better than using the overall mean. b1 = Low Dose b2 = High Dose Note: b2 = the difference between the means of the high-dose group and the placebo group (5.0 - 2.2 = 2.8). b1 = the difference between the means of the low-dose group and the placebo group (3.2 - 2.2 = 1)

Theory of ANOVA: Partitioning Variance We compare the amount of variability explained by the Model (MSM), to the error in the model [individual differences] (MSR) This ratio is called the F-ratio. If the model explains a lot more variability than it can’t explain, then the experimental manipulation has had a significant effect on the outcome (DV).

Partitioning the Variance Total sum of squares (SST) Model sum of squares (SSM) Residual sum of squares (SSR)

SST = based on differences between each data point and the grand mean SSM = differences between predicted values (group means) and the grand mean Grand Mean = 3.467 XPlacebo = 2.20 XLowDose = 3.20 XHighDose = 5.00 SSR = based on differences between person’s score and their group mean. Basis for SST Basis for SSR Basis for SSM Partitioning the Variance Base Category

Total sum of squares (SST) Compute the difference between each data point and the grand mean, square these differences, then add them together. 1 Grand Mean = 3.467 Grand SD = 1.767 Grand Variance = 3.124 2 3 4

Model sum of squares (SSM) Calculate the difference between the mean of each group and the grand mean. Square each of these differences Multiply each result by the number of participants within that group. Add the values for each group together.

Residual sum of squares (SSR) SSR = SST - SSM … But, don’t rely on this relationship (e.g., might not work out if either SST or SSM is miscalculated) SSR is the difference between what the model predicts and what was actually observed. Therefore it is calculated by by looking at the difference between the score obtained by a person and the mean of the group to which the person belongs.

Total sum of squares (SSR)

Mean Squares (MSM and MSR) SSM = amount of variation explained by the model (exp. manipulation). SSR = amount of variation due to extraneous factors. These are “summed” scores and will therefore be influenced by the number of scores. To eliminate this bias we calculate the average sum of squares (mean squares) by dividing by the appropriate degrees of freedom. Calculating Degrees of Freedom (for one-way independent groups ANOVA) dftotal = N - 1 (number of all scores minus 1) dfM / between = k - 1 (number of groups minus 1) df R / within = N - k (number of all scores minus number of groups)

The F-ratio F = MSM/MSR and is a measure of the ratio of variation due to the experimental effect to the variation due to individual differences (unexplained variance). Therefore, we again return to the basic conceptual model: Test Statistic = systematic variance unsystematic variance

Assumptions The one-way independent groups ANOVA test requires the following statistical assumptions: Random and independent sampling. Data are from normally distributed populations. Note: This test is robust against violation of this assumption if n > 30 for both groups. Variances in these populations are roughly equal (Homogeneity of variance). Note: This test is robust against violation of this assumption if all group sizes are equal.

1-Way Independent-Groups ANOVA Characteristics: Used when testing 3 or more experimental groups. Each participant contributes only one score to the data. Used to test the (null) hypothesis that several means are equal. Example: Testing the idea that exposure to mobile phones will lead to brain tumors. IV: Hrs./day exposure DV: Size of brain tumor Mobile Phone Use (hours per day)

Steps in the Analysis -- Calculate the Model (or between-groups) sum of squares (SSM) and the Residual (or within-groups) sum of squares (SSR). -- The number of scores used to calculate SSM and SSR are different, so to make them comparable, we convert them to the average sums of square or “mean squares” (MS) by dividing by their appropriate degrees of freedom. -- F = MSM/MSR and is a measure of the ratio of variation due to the experimental effect to the variation due to individual differences (unexplained variance).

Degrees of Freedom (df): One-way Independent-Groups ANOVA Formulas for the number of degrees of freedom in one-way independent-groups ANOVA differ for each SS term. The degrees of freedom (df) reflect the number of values free to vary in any sum of squares term. Calculating Degrees of Freedom dftotal = N - 1 (number of all scores minus 1) dfbetween = k - 1 (number of groups minus 1) df within = N - k (number of all scores minus number of groups)

SPSS Summary Tables 95% Confidence Interval N Mean Std. Deviation Std. Error Lower Bound Upper Min. Max. 1 2 3 4 5 Total 20 120 .0175 .5149 1.2614 3.0216 4.8878 4.7306 2.4056 .01213 .28419 .49218 .76556 .69625 .78163 2.02662 .00271 .06355 .11005 .17118 .15569 .17478 .18500 .0119 .3819 1.0310 2.6633 4.5619 4.3648 2.0393 .0232 .6479 1.4917 3.3799 5.2137 5.0964 2.7720 .00 .48 1.77 3.04 2.70 .04 .94 2.34 4.31 6.05 6.14 Size of Tumor 95% Confidence Interval Hrs. of Exposure Sum of Squares df Mean Square F Sig. Between Groups Within Groups Total 450.664 38.094 488.758 5 114 119 90.133 .334 269.733 .000

Levene’s Test: Testing the Homogeneity of Variance Assumption If Levene’s test is significant (p<.05), then the variances are significantly different. What to do if Levene’s test is significant? Transform the data Report the F value and the Levene’s test results and allow your audience to assess the accuracy of your results for themselves. Levene Statistic df1 df2 Sig. 10.245 5 114 .000

Computing Effect Size (2 - eta squared and 2 - omega squared) Analogous to r2, eta-squared reflects the proportion of DV variance explained by each IV in the sample data. It tends to be upwardly biased (i.e., an overestimate). 2 = SSModel / SStotal One 2 for each effect (i.e., 1 per IV plus 1 per interaction) and the 2s sum to 1. SPSS reports partial eta squared (p2 = SSEffect / SSEffect+ SSError) In 1-way ANOVA, eta-squared and partial eta-squared are the same. An alternative, omega-squared (2), estimates the proportion of DV variance explained by the IVs in the population (see Field, 2005, p. 357-359).

Computing Effect Size (2 vs. 2 omega squared) 2 = SSM = 450.66 = .922 SST 488.76 2 = SSM - (dfM)(MSR) = 450.664 - (5 x .334) SST + MSR 488.758 + .334 = 449.01 = .918 489.09 (called eta squared, 2 ) (called omega squared, 2 )

Reporting the Results Levene’s test indicated that the assumption of homogeneity of variance had been violated, F(5, 114) = 10.25, p<.001. Transforming the data did not rectify this problem and so F tests are reported nonetheless. The results show that using a mobile phone significantly affected the size of brain tumor found in participants, F(5, 114) = 269.73, p<.001, 2 = .92. Tumor size (measured in mm) increased with the duration of exposure. Specifically, tumor size for 0, 1, 2, 3, 4 or 5 hours/day of exposure was .02 mm, .51 mm, 1.26 mm, 3.02 mm, 4.89 mm, and 4.73 mm, respectively. The effect size indicated that the effect of phone use on tumor size was substantial. Games-Howell post hoc tests revealed significant differences between all groups (p<.001 for all tests) except between 4- and 5-hours (ns).

Computing a One-way Independent-groups ANOVA by hand

Critical Values for F

Computing Effect Size (2 vs 2 Omega squared) 2 = SSM = 93.17 = .69 SST 134.67 2 = SSM - (dfM x MSR) = 93.17 - (2 x 4.61) MSR + SST 4.61 + 134.67 = 83.95 = .60 139.28

Computing the One-way Independent-groups ANOVA using SPSS

SPSS Variable View: Assign Value Labels

SPSS Data View: Type in the Data

SPSS Data Editor: Select Procedure

Specify IV and DV, then select “options” Select “Descriptives” and “Homogeneity test”. If the variances are unequal, choose “Brown-Forsythe” or “Welch” tests. 38

Choosing Follow-up tests: Contrasts vs. Post-Hoc Select “Contrasts” if you have apriori hypotheses. Select “Post-Hoc” if you do not. 39

Planned Contrasts (if we have specific hypotheses) The first contrast compares the combined “experimental drug groups” against the control (Saltwater) group. The second contrast compares the the two experimental groups, Fenfluramine vs. Amphetamine. 40

Post-Hoc Tests (if we do NOT have specific hypotheses) LSD (Least Significant Difference) does not control Type I error rate. Bonferroni tightly controls Type I error rate. Other post-hoc procedures have specific applications (see Field text). 41

SPSS ANOVA Output: Descriptives and ANOVA Summary Table

SPSS ANOVA Output: Planned Contrasts Levene’s test was not significant (p>.05) and so we can assume equal variances. In the Contrast Tests summary, use the “Assume equal variances” row values. Contrast 1: weight gain among rats receiving the experimental drugs is significantly different from rats receiving saltwater. Contrast 2: weight gain among rats taking fenfluramine and amphetamine are not significantly different.

Post-hoc tests After the Fact “Snooping”: Some Considerations LSD does not control Type I error rate, whereas the Bonferroni test tightly controls the Type I error rate. Given equal sample sizes and equal variances, use REGWQ or Tukey HSD, as both have good power and tight control over Type I error rate. If sample sizes across groups are slightly different, use Gabriel’s procedure because it has greater power, but if sample sizes are very different use Hochberg’s GT2. If there is any doubt that the population variances are equal, use the Games-Howell procedure, in place of, or in addition to other tests. 44

Post-Hoc tests: LSD vs Bonferroni

Reporting the Results The results show that the amount of weight gained by the rats was significantly affected by the type of drug they received, F(2,9) = 10.10, p < .01, 2 = .60. Specifically, weight gain was greatest for rats receiving Saltwater (M=10.0, SE=.82) and least for rats receiving Fenfluramine (M=3.25, SE=1.11). Weight gain for rats receiving Amphetamine was intermediate (M=5.75, SE=1.25). If you used planned comparisons … Planned comparisons revealed a significant difference in weight gain between the combined experimental groups and the saltwater control group, t(9)=-4.18, p<.05 (1-tailed). The difference in weight gain between the two experimental groups receiving fenfluramine and amphetamine, respectively, was not significant, t(9)= -1.65, p>.05 (1-tailed). If you used post-hoc tests … Bonferroni post hoc tests revealed a significant difference in weight gain between the Fenfluramine and Saltwater groups (p<.05). No other comparisons were significant (ps>.05).

Sample Problem: Your turn…