1. Statistical Analysis of Data1 of 38 1 of 42 Department of Cognitive Science Adv. Experimental Methods & Statistics PSYC 4310 / COGS 6310 MANOVA Multivariate Analysis of Variance Michael J. Kalsher PSYC 4310 Advanced Experimental Methods and Statistics © 2012, Michael Kalsher

2. Statistical Analysis of Data2 of 38 2 of 42 MANOVA: What is it? Used to determine main and interaction effects of categorical variables on multiple DVs. ANOVA tests the differences in means of a single DV for two or more categories of IVs. MANOVA tests whether the vectors of means for the two or more groups are sampled from the same sampling distribution. In other words, MANOVA gives a measure of the overall likelihood of picking two or more random vectors of means out of the same hat. Purposes for MANOVA To compare groups formed by categorical IVs on group differences in a set of interval-level DVs. To use lack of difference for a set of DVs as a criterion for reducing a set of IVs to a smaller, more easily modeled number of variables. To identify the IVs which differentiate a set of DVs the most.

3. Statistical Analysis of Data3 of 38 3 of 42 Why Use MANOVA? Advantages: Improves chances of discovering changes as a result of different variables and their interactions. Protects against inflated Type I error due to multiple tests of correlated dependent variables. Can detect whether groups differ along a combination of variables (i.e., a variate), whereas ANOVA can detect only if groups differ along a single variable. Disadvantages: More complicated and less powerful than ANOVA. Analysis often ambiguous in interpretation of the effects of IVs on any single DV.

4. Statistical Analysis of Data4 of 38 4 of 42 MANOVA: Parts of the Analysis Main Analysis Four commonly used ways of assessing the overall significance of a MANOVA: Pillai’s trace (V); Hotelling’s trace, Wilks’s lambda (  ), and Roy’s largest root. Debate over which method is best in terms of power and sample size considerations. Rules of thumb: If group differences are concentrated on the first variate, Roy’s statistic most powerful followed by Hotelling’s trace, Wilks’s lambda and Pillai’s trace. When groups differ along more than one variate, the power ordering is reversed. Approaches to Follow-up Analysis Perform separate ANOVAs for each DV (controlling for Type I error). Transform linear combinations of DVs to z scores, add them together, then evaluate the combined scores using ANOVA. Use discriminant function analysis (DFA) (yields one or more uncorrelated linear combinations of DVs that maximize differences among the groups)

5. Statistical Analysis of Data5 of 38 5 of 42 MANOVA test statistics: A Comparison

6. Statistical Analysis of Data6 of 38 6 of 42 Pillai-Bartlett trace (V) (Pillai’s trace) Given by the equation below in which represents the eigenvalues for each of the discriminant variates, and s represents the number of variates. Pillai’s trace is the sum of the proportion of explained variance on the discriminant functions and is similar to the ratio of SS M /SS T i 1 + i  i=1 s V =

7. Statistical Analysis of Data7 of 38 7 of 42 Hotelling’s trace The Hotelling-Lawley trace is the sum of the eigenvalues for each variate and is computed by the equation below. This test statistic is the sum of SS M /SS R for each of the variates and so it compares directly to the F ratio in ANOVA. i  i=1 s T =

8. Statistical Analysis of Data8 of 38 8 of 42 Wilks’s lambda (  ) Wilks’ lambda is the product of the unexplained variance on each of the variates. The  symbol is similar to the summation symbol (  ) except that it means multiply rather than add up. Wilks’s lambda represents the ratio of error variance to total variance (SS R /SS T ) for each variate.  i=1 s  = 1 1 + i

9. Statistical Analysis of Data9 of 38 9 of 42 Roy’s largest root Roy’s largest root is the eigenvalue for the first variate. In a sense, it is the same as the Hotelling-Lawley trace, except for the first variate only. This statistic represents the proportion of explained variance to unexplained variance (SS M /SS R ) for the first discriminant function. This value is conceptually the same as the F-ratio in univariate ANOVA and represents the maximum possible between-group difference given the data collected. Largest root = largest

10. Statistical Analysis of Data10 of 38 10 of 42 MANOVA: Assumptions Independence: Observations should be statistically independent. Random Sampling: Data should be score level and randomly sampled from the population of interest. Multivariate Normality: In ANOVA, we assume that our DV is normally distributed within each group. In MANOVA, we assume that the DVs (collectively) have multivariate normality within groups (cannot be tested directly by SPSS). Homogeneity of Covariance Matrices: In ANOVA, it is assumed that the variances in each group are roughly equal (homogeneity of variance). In MANOVA, we assume this is true for each DV, but also that the correlation between any two DVs is the same in all groups. As a preliminary test, Levene’s test should not be significant for any of the DVs. Since Levene’s test doesn’t take account of the covariances, Box’s M test should be used to test whether the population variance-covariance matrices of the different groups in the analysis are equal Note: The F test from Box’s M statistics should be interpreted cautiously in that a significant result may be due to violation of the multivariate normality assumption for the Box’s M test, and a non-significant result may be due to a lack of power.

11. Statistical Analysis of Data11 of 38 11 of 42 Effect Size Statistics for MANOVA The multivariate GLM procedure computes a multivariate effect size index. The multivariate effect size associated with Wilks’s lambda (  ) is the multivariate eta square. 1s1s Multivariate  2 = 1 -  Here, s is equal to the number of levels of the factor minus 1 or the number of DVs, whichever is smaller. This statistic should be interpreted similar to a univariate eta square and ranges in value from 0 to 1. A 0 represents no relationship between the factor and the DV, while a 1 indicates the strongest possible relationship. Unclear what should be considered a small, medium, and large effect size for this statistic.

12. Statistical Analysis of Data12 of 38 12 of 42 Controversies and MANOVA Ability of MANOVA to detect true effects – Ramsey (1982): As correlation between DVs increases, power of MANOVA decreases. – Tabachnick & Fidell (2001): MANOVA works best with highly negatively correlated DVs and acceptably well with moderately correlated DVs in either direction; MANOVA wasteful when DVs are uncorrelated. – Cole et al. (1994): Power of MANOVA depends on combination of the correlation between DVs and effect size. Expecting large effect: MANOVA most powerful if the measures are somewhat different and if the group differences are in the same direction for each measure. If 2 DVs differ in terms of group differences (one large, one small), then power increased if DVs are highly correlated.

13. Statistical Analysis of Data13 of 38 13 of 42 The Matrix: Revisited The MANOVA test statistic is derived by comparing the ratio of a matrix representing the systematic variance of all DVs to a matrix representing the unsystematic variance of all DVs 53610 31246 6427 10676 Square Matrix 1000 0100 0010 0001 Identity Matrix Diagonal components Off-diagonal components 2648 Row Vector Single person’s score on four different variables 8 6 10 15 Column Vector Four participants’ score on one variable

14. Statistical Analysis of Data14 of 38 14 of 42 Partitioning the Variance SS T = Sums of Squares Total SS m = Sums of Squares Model (Systematic Variance) SS R = Sums of Squares Error (Unexplained Variance) SS T SS M SS R MANOVA Test Statistic Sum of squares and cross-products matrices Systematic Variance HHypothesis sum of squares and cross-product matrix or hypothesis SSCP Unsystematic Variance EError sum of squares and cross-products matrix or error SSCP Total VarianceTTotal sum of squares and cross-products matrix or total SSCP Cross-product deviations represent a total value for the combined error between two variables, so in some sense, they represent an unstandardized estimate of the total correlation between two variables.

15. Statistical Analysis of Data15 of 38 15 of 42 MANOVA: Performing MANOVA Using SPSS

16. Statistical Analysis of Data16 of 38 16 of 42 Move “Social Dimension Summed Scale” and “Pragmatic Dimension Summed Scale” to the Dependent Variables box. Move “Gender (q047_r01)” to the Fixed Factors box. Then, click “Options”.

17. Statistical Analysis of Data17 of 38 17 of 42 Move “Gender (q047_r01” to the Display Means box. Click “Descriptive statistics”, “Estimates of effect size”, and “Homogeneity tests” in the Display box. Click “Continue”

18. Statistical Analysis of Data18 of 38 18 of 42

19. Statistical Analysis of Data19 of 38 19 of 42 SPSS Output

20. Statistical Analysis of Data20 of 38 20 of 42 SPSS Output: Main Analysis

21. Statistical Analysis of Data21 of 38 21 of 42 SPSS Output: Univariate ANOVAs

22. Statistical Analysis of Data22 of 38 22 of 42 Results Section A one-way MANOVA on the two dependent variables, the summed scales for the Social and Pragmatic dimensions, was significant for gender, Wilks’s  =.995, F(2,1340) = 3.289, p<.05. Table 1 contains the means and the standard deviations on the dependent variables for males and females. Analyses of variance (ANOVAs) on each dependent variable were conducted as follow-up tests to the MANOVA. The ANOVA on the Social Dimension summed scale scores was significant, F(1,1341) = 6.29, p.05.

23. Statistical Analysis of Data23 of 38 23 of 42 Practice Problem: Effects of study strategies on learning A researcher investigates the effectiveness of different study strategies on learning. Thirty undergraduates are randomly assigned to one of three study conditions. All participants receive the same set of study questions, but each group receives different instructions about how to study. The write group is instructed to write responses to each question, the think group is instructed to think about answers to the questions, and the talk group is instructed to develop a talk that they could deliver centering on the answers to the questions. At the completion of the study session, all students take a quiz consisting of two types of questions: recall and application.

24. Statistical Analysis of Data24 of 38 24 of 42 SubjectGroupApplicationRecall 1131 2144 3143 4145 5123 6132 7143 8133 9135 10133 11267 12274 13256 14263 15252 SubjectGroupApplicationRecall 16257 17254 18255 19287 20265 21343 22366 23344 24343 25356 26355 27355 28324 29333 30345 Data Set: Effects of study strategies on learning

25. Statistical Analysis of Data25 of 38 25 of 42

26. Statistical Analysis of Data26 of 38 26 of 42 Move “recall” and “applicat” to the Dependent Variables box. Move “Study Strategies” to the Fixed Factors box. Then, click “Options”.

27. Statistical Analysis of Data27 of 38 27 of 42 Move “group” to the Display Means box. Click “Descriptive statistics”, “Estimates of effect size”, and “Homogeneity tests” in the Display box. Click “Continue”

28. Statistical Analysis of Data28 of 38 28 of 42 Select “Post Hoc” from the Multivariate Screen Move “group” to the Post Hoc Test for box. Select “Bonferroni” Select “Games Howell” Click “Continue” Click “OK”

29. Statistical Analysis of Data29 of 38 29 of 42 SPSS Output for MANOVA

30. Statistical Analysis of Data30 of 38 30 of 42 SPSS Output for MANOVA

31. Statistical Analysis of Data31 of 38 31 of 42 SPSS Output: Univariate ANOVAs

32. Statistical Analysis of Data32 of 38 32 of 42 SPSS Output for Post-Hoc Tests

33. Statistical Analysis of Data33 of 38 33 of 42 Results Section A one-way Multivariate analysis of variance (MANOVA) was conducted to determine the effect of the three types of study strategies (thinking, writing, and talking) on the two dependent variables, the recall and the application test scores. Significant differences were found among the three study strategies on the dependent measures, Wilks’s  =.42, F(4,52) = 7.03, p<.01. The multivariate  2 based on Wilks’s  was.35. Table 1 contains the means and the standard deviations on the dependent variables for the three groups.

34. Statistical Analysis of Data34 of 38 34 of 42 Results Section - continued Univariate ANOVAs on each dependent variable were conducted as follow-up tests to the MANOVA. The ANOVA on the recall scores was significant, F(2,27) = 17.11, p<.01,  2 =.56, as was the ANOVA on the application scores, F(2,27) = 4.20, p =.026,  2 =.24. Post-hoc analyses to the univariate ANOVA for the recall and application scores consisted of conducting pair-wise comparisons to find which study strategy affected performance most strongly. With respect to the recall scores, the writing group produced significantly superior performance on the recall questions in comparison with either of the other two groups (ps.05). With respect to the application scores, the writing group produced significantly better performance on the application questions than the thinking group (p.05).

35. Statistical Analysis of Data35 of 38 35 of 42 One More Step … Following up MANOVA with Discriminant Analysis The practice of conducting ANOVAs as follow-up tests to significant MANOVA has been criticized because univariate ANOVAs do not take into account the multivariate nature of MANOVA An alternative is to conduct follow-up analyses using discriminant function analysis (DFA). DFA can be used after MANOVA to see how the dependent variables discriminate the groups. DFA yields one or more uncorrelated linear combinations of dependent variables (variates) that maximize differences among the groups.

36. Statistical Analysis of Data36 of 38 36 of 42 Discriminant Analysis: Step by Step

37. Statistical Analysis of Data37 of 38 37 of 42 Discriminant Analysis: Step by Step Predictors

38. Statistical Analysis of Data38 of 38 38 of 42 Discriminant Analysis: Step by Step Already in the MANOVA output Useful for gaining insight into the relationships between DVs for each group Produces the bs for each variate Finally, click on “Continue”, then “Classify” on the main dialog box for discriminant analysis

39. Statistical Analysis of Data39 of 38 39 of 42 Discriminant Analysis: Step by Step Plots the variate scores for each participant groups according to the study strategy they were given Provides an overall gauge of how well the discriminant variates classify the actual participants

40. Statistical Analysis of Data40 of 38 40 of 42 Discriminant Analysis: Step by Step Click on “Save” These scores can be useful because the variates that the analysis identifies may represent important underlying constructs

41. Statistical Analysis of Data41 of 38 41 of 42 Discriminant Analysis Output Shows that only one of the variates is significant. Thus, the group differences shown by the MANOVA can be explained in terms of one underlying dimension

42. Statistical Analysis of Data42 of 38 42 of 42 Discriminant Analysis Output These values are comparable to factor loadings. They represent the relative contribution of each DV to group separation. These coefficients tell us the relative contribution of each variable to the variates.

43. Statistical Analysis of Data43 of 38 43 of 42 Results The MANOVA was followed up with discriminant analysis, which revealed two discriminant functions. The first explained 96.7% of the variance, canonical R 2 =.56, whereas the second explained only 3.3% of the variance, canonical R 2 =.04. In combination, these discriminant functions significantly differentiated the treatment groups, Λ = 0.42,  2 (4) = 22.90, p.05. The correlations between outcomes and the discriminant functions revealed that total correct on recall questions loaded almost exclusively on the first function (r =.99); total correct on application questions loaded more highly on the second function (r =.89) than on the first function (r =.47).