1. Stat 112: Lecture 22 Notes Chapter 9.1: One Way Analysis of Variance Chapter 9.2: Two Way Analysis of Variance

2. Milgram’s Obedience Experiment Y=Maximum voltage level subject would give before refusing to continue. X=Condition. Four categories: Remote feedback, voice feedback, proximity, touch proximity.

3. Testing whether each of the groups is different Naïve approach to deciding which groups have mean that is different from the average of the means of all groups: Do t- test for each group and look for groups that have p-value <0.05. Problem: Multiple comparisons.

5. Errors in Hypothesis Testing State of World Null Hypothesis True Alternative Hypothesis True Decision Based on Data Accept Null Hypothesis Correct Decision Type II error Reject Null Hypothesis Type I errror Correct Decision When we do one hypothesis test and reject null hypothesis if p-value <0.05, then the probability of making a Type I error when the null hypothesis is true is 0.05. We protect against falsely rejecting a null hypothesis by making probability of Type I error small.

6. Multiple Comparisons Problem Compound uncertainty: When doing more than one test, there is an increase chance of making a mistake. If we do multiple hypothesis tests and use the rule of rejecting the null hypothesis in each test if the p-value is 0.05.

7. Multiple Comparisons Simulation In multiplecomp.JMP, 20 groups are compared with sample sizes of ten for each group. The observations for each group are simulated from a standard normal distribution. Thus, in fact, Number of pairs found to have significantly different means using t-test at level Iterat ion 12345 # of Pairs

8. Multiple Comparison Simulation In multiplecomp.JMP, 20 groups are compared with sample sizes of ten for each group. The observations for each group are simulated from a standard normal distribution. Thus, in fact, Number of groups found to have means different than average using t-test and rejecting if p-value <0.05. Iteration12345 # of Groups

9. Individual vs. Familywise Error Rate When several tests are considered simultaneously, they constitute a family of tests. Individual Type I error rate: Probability for a single test that the null hypothesis will be rejected assuming that the null hypothesis is true. Familywise Type I error rate: Probability for a family of test that at least one null hypothesis will be rejected assuming that all of the null hypotheses are true. When we consider a family of tests, we want to make the familywise error rate small, say 0.05, to protect against falsely rejecting a null hypothesis.

10. Why is it important to make familywise error rate small? If we consider many tests, we will almost always reach some false conclusions if we do not make familywise error rate small: –Suppose we test the effect of eating a large amount of 100 different foods on preventing cancer and suppose that none of the foods helps. If we test each food with individual error rate 0.05, then we will on average find 5 foods that have a significan effect even though in truth, no food has a significant effect.

11. Suppose we look for a rule that reliably predicts the presidential election winner. If we look over enough rules for a rule that is significant at the 0.05 level, we will be able to find one even if no rule is reliable. Before this year’s election, the Washington Post reported this rule: When the Redskins win their final home game before the presidential election, the incumbent party stays in office. When the Redskins lose their final home game before the election the incumbent party loses, except in 2004. Pretty simple, right? Steve Hirdt of the Elias Sports Bureau discovered it's happened precisely that way 16 of 17 times since 1940. Anything close to that exact is more than a trend and is certainly politically unbiased.

12. Bonferroni Method General method for doing multiple comparisons for any family of k tests. Denote familywise type I error rate we want by p*, say p*=0.05. Compute p-values for each individual test -- Reject null hypothesis for ith test if Guarantees that familywise type I error rate is at most p*. Why Bonferroni works: If we do k tests and all null hypotheses are true, then using Bonferroni with p*=0.05, we have probability 0.05/k to make a Type I error for each test and expect to make k*(0.05/k)=0.05 errors in total.

14. Tukey’s Honest Significant Differences (HSD) Tukey’s HSD is a method that is specifically designed to control the familywise type I error rate (at 0.05) for analysis of variance. After Fit Y by X, click the red triangle next to Response, click Compare Means and click All Pairs, Tukey’s HSD. If using Fit Model, click the red triangle next to the X variable in the Leverage Plot for the X variable and click LSMeans Tukey HSD.

15. Tukey’s HSD for Milgram Data

16. Assumptions in one-way ANOVA Assumptions needed for validity of one- way analysis of variance p-values and CIs: –Linearity: automatically satisfied. –Constant variance: Spread within each group is the same. –Normality: Distribution within each group is normally distributed. –Independence: Sample consists of independent observations.

17. Rule of thumb for checking constant variance Constant variance: Look at standard deviation of different groups by using Fit Y by X and clicking Means and Std Dev. Rule of Thumb: Check whether (highest group standard deviation/lowest group standard deviation) is greater than 2. If greater than 2, then constant variance is not reasonable and transformation should be considered.. If less than 2, then constant variance is reasonable. (Highest group standard deviation/lowest group standard deviation) =(131.874/63.640)=2.07. Thus, constant variance is not reasonable for Milgram’s data.

18. Transformations to correct for nonconstant variance If standard deviation is highest for high groups with high means, try transforming Y to log Y or. If standard deviation is highest for groups with low means, try transforming Y to Y 2. SD is particularly low for group with highest mean. Try transforming to Y 2. To make the transformation, right click in new column, click New Column and then right click again in the created column and click Formula and enter the appropriate formula for the transformation.

19. Transformation of Milgram’s data to Squared Voltage Level Check of constant variance for transformed data: (Highest group standard deviation/lowest group standard deviation) = 1.63. Constant variance assumption is reasonable for voltage squared. Analysis of variance tests are approximately valid for voltage squared data; reanalyzed data using voltage squared.

20. Analysis using Voltage Squared Strong evidence that the group mean voltage squared levels are not all the same. Strong evidence that remote has higher mean voltage squared level than proximity and touch-proximity and that voice-feedback has higher mean voltage squared level than touch-proximity, taking into account the multiple comparisons.

21. Rule of Thumb for Checking Normality in ANOVA The normality assumption for ANOVA is that the distribution in each group is normal. Can be checked by looking at the boxplot, histogram and normal quantile plot for each group. If there are more than 30 observations in each group, then the normality assumption is not important; ANOVA p-values and CIs will still be approximately valid even for nonnormal data if there are more than 30 observations in each group. If there are less than 30 observations per group, then we can check normality by clicking Analyze, Distribution and then putting the Y variable in the Y, Columns box and the categorical variable denoting the group in the By box. We can then create normal quantile plots for each group and check that for each group, the points in the normal quantile plot are in the confidence bands. If there is nonnormality, we can try to use a transformation such as log Y and see if the transformed data is approximately normally distributed in each group.

22. One way Analysis of Variance: Steps in Analysis 1.Check assumptions (constant variance, normality, independence). If constant variance is violated, try transformations. 2.Use the effect test (commonly called the F- test) to test whether all group means are the same. 3.If it is found that at least two group means differ from the effect test, use Tukey’s HSD procedure to investigate which groups are different, taking into account the fact multiple comparisons are being done.

23. Analysis of Variance Terminology Analysis of variance is generally concerned with comparing the means of different groups and is a special case of regression in which all the explanatory variables are categorical variables. The criterion (criteria) by which we classify the groups in analysis of variance is called a factor. In one-way analysis of variance, we have one factor. The possible values of the factor are levels. Milgram’s study: Factor is experimental condition with levels remote, voice-feedback, proximity and touch- proximity. Two-way analysis of variance: Groups are classified by two factors.

24. Two-way Analysis of Variance Examples Milgram’s study: In thinking about the Obedience to Authority study, many people have thought that women would react differently than men. Two-way analysis of variance setup in which the two factors are experimental condition (levels remote, voice-feedback, proximity, touch-proximity) and sex (levels male, female). Package Design Experiment: Several new types of cereal packages were designed. Two colors and two styles of lettering were considering. Each combination of lettering/color was used to produce a package, and each of these combinations was test marketed in 12 comparable stores and sales in the stores were recorded.. Two-way analysis of variance in which two factors are color (levels red, green) and lettering (levels block, script). Goal of two-way analysis of variance: Find out how the mean response in a group depends on the levels of both factors and find the best combination.

25. Two-way Analysis of Variance The mean of the group with the ith level of factor 1 and the jth level of factor 2 is denoted, e.g., in package-design experiment, the four group means are As with one-way analysis of variance, two-way analysis of variance can be seen as a a special case of multiple regression. For two-way analysis of variance, we have two categorical explanatory variables for the two factors and also include an interaction between the factors.

26. Two-way ANOVA in JMP Use Analyze, Fit Model with a categorical variable for the first factor, a categorical variable for the second factor and an interaction variable that crosses the first factor and the second factor. The LS Means Plots (which show how the means of the groups vary as the levels of the factor vary) are produced by going to the output in JMP for each variable that is to the right of the main output, clicking the red triangle next to each variable (for package design, the vairables are Color, TypeStyle, Typestyle*Color) and clicking LS Means Plot.

27. Estimated Mean for Red Block group = 144.92+9.83-11.17+4.5 = 148.08 Estimated Mean for Red Script group = 144.92+9.83+11.17-4.5= 161.42

28. The LS Means Plots show how the means of the groups vary as the levels of the factors vary. For the top plot for color, green refers to the mean of the two green groups (green block and green script) and red refers to the mean of the two red groups (red block and red script). Similarly for the second plot for TypeStyle, block refers to the mean of the two block groups (red block and green block). The third plot for TypeStyle*Color shows the mean of all four groups.