1. Stats 346.3 Multivariate Data Analysis Stats 848.3

2. Instructor:W.H.Laverty Office:235 McLean Hall Phone:966-6096 Lectures: M W F 12:30am - 1:20pm Biol 123 Evaluation:Assignments, Term tests - 40% Final Examination - 60%

3. Dates for midterm tests: 1.Friday, February 06 2.Friday, March 20 Each test and the Final Exam are Open Book Students are allowed to take in Notes, texts, formula sheets, calculators (laptop computers.)

4. Text: Stat 346 –Multivariate Statistical Methods – Donald Morrison Not Required - I will give a list of other useful texts that will be in the library

5. Bibliography 1.Cooley, W.W., and Lohnes P.R. (1962). Multivariate Procedures for the Behavioural Sciences, Wiley, New York. 2.Fienberg, S. (1980), Analysis of Cross-Classified Data, MIT Press, Cambridge, Mass. 3.Fingelton, B. (1984), Models for Category Counts, Cambridge University Press. 4.Johnson, R.A. and Wichern D.W. Applied Multivariate Statistical Analysis, Prentice Hall. 5.Morrison, D.F. (1976), Multivariate Statistical Methods, McGraw-Hill, New York. 6.Seal, H. (1968), Multivariate Statistical Analysis for Biologists, Metheun, London 7. Alan Agresti (1990) Categorical Data Analysis, Wiley, New York.

6. The lectures will be given in Power Point They are now posted on the Stats 346 web page

7. Course Outline

8. Introduction

9. Review of Linear Algebra and Matrix Analysis Review of Linear Statistical Theory Chapter 2 Chapter 1

10. Multivariate Normal distribution Multivariate Data plots Correlation - sample estimates and tests Canonical Correlation Chapter 3

11. Mean Vectors and Covariance matrices Single sample procedures Two sample procedures Profile Analysis Chapter 4

12. Multivariate Analysis of Variance (MANOVA) Chapter 5

13. Classification and Discrimination Discriminant Analysis Logistic Regression (if time permits) Cluster Analysis Chapters 6

14. The structure of correlation Principal Components Analysis (PCA) Factor Analysis Chapter 9

15. Multivariate Multiple Regression (if time permits) References TBA

16. Discrete Multivariate Analysis (if time permits) References: TBA

17. Introduction

18. Multivariate Data We have collected data for each case in the sample or population on not just one variable but on several variables – X 1, X 2, … X p This is likely the situation – very rarely do you collect data on a single variable. The variables maybe 1.Discrete (Categorical) 2.Continuous (Numerical) The variables may be 1.Dependent (Response variables) 2.Independent (Predictor variables)

19. Independent variables Dependent Variables CategoricalContinuousContinuous & Categorical Categorical Multiway frequency Analysis (Log Linear Model) Discriminant Analysis Continuous ANOVA (single dep var) MANOVA (Mult dep var) MULTIPLE REGRESSION (single dep variable) MULTIVARIATE MULTIPLE REGRESSION (multiple dependent variable) ANACOVA (single dep var) MANACOVA (Mult dep var) Continuous & Categorical ?? A chart illustrating Statistical Procedures

20. Multivariate Techniques Multivariate Techniques can be classified as follows: 1.Techniques that are direct analogues of univariate procedures. There are univariate techniques that are then generalized to the multivariate situarion e. g. The two independent sample t test, generalized to Hotelling’s T 2 test ANOVA (Analysis of Variance) generalized to MANOVA (Multivariate Analysis of Variance)

21. 2.Techniques that are purely multivariate procedures. Correlation, Partial correlation, Multiple correlation, Canonical Correlation Principle component Analysis, Factor Analysis -These are techniques for studying complicated correlation structure amongst a collection of variables

22. 3.Techniques for which a univariate procedures could exist but these techniques become much more interesting in the multivariate setting. Cluster Analysis and Classification -Here we try to identify subpopulations from the data Discriminant Analysis -In Discriminant Analysis, we attempt to use a collection of variables to identify the unknown population for which a case is a member

23. An Example: A survey was given to 132 students Male=35, Female=97 They rated, on a Likert scale 1 to 5 their agreement with each of 40 statements. All statements are related to the Meaning of Life

24. Questions and Statements

25. Statements - continued

28. Cluster Analysis of n = 132 university students using responses from Meaning of Life questionnaire (40 questions)

29. Discriminant Analysis of n = 132 university students into the three identified populations

30. A Review of Linear Algebra With some Additions

31. Matrix Algebra Definition An n × m matrix, A, is a rectangular array of elements n = # of columns m = # of rows dimensions = n × m

32. Definition A vector, v, of dimension n is an n × 1 matrix rectangular array of elements vectors will be column vectors (they may also be row vectors)

33. A vector, v, of dimension n can be thought a point in n dimensional space

34. v2v2 v1v1 v3v3

35. Matrix Operations Addition Let A = (a ij ) and B = (b ij ) denote two n × m matrices Then the sum, A + B, is the matrix The dimensions of A and B are required to be both n × m.

36. Scalar Multiplication Let A = (a ij ) denote an n × m matrix and let c be any scalar. Then cA is the matrix

37. v2v2 v1v1 v3v3 Addition for vectors

38. v2v2 v1v1 v3v3 Scalar Multiplication for vectors

39. Matrix multiplication Let A = (a ij ) denote an n × m matrix and B = (b jl ) denote an m × k matrix Then the n × k matrix C = (c il ) where is called the product of A and B and is denoted by A∙B

40. In the case that A = (a ij ) is an n × m matrix and B = v = (v j ) is an m × 1 vector Then w = A∙v = (w i ) where is an n × 1 vector v2v2 v1v1 v3v3 w2w2 w1w1 w3w3

41. Definition An n × n identity matrix, I, is the square matrix Note: 1. AI = A 2. IA = A.

42. Definition (The inverse of an n × n matrix) AB = BA = I, If the matrix B exists then A is called invertible Also B is called the inverse of A and is denoted by A -1 Let A denote the n × n matrix Let B denote an n × n matrix such that

43. The Woodbury Theorem where the inverses

44. Then all we need to show is that H(A + BCD) = (A + BCD) H = I. Proof: Let

46. The Woodbury theorem can be used to find the inverse of some pattern matrices: Example: Find the inverse of the n × n matrix

47. where hence and

48. Thus Now using the Woodbury theorem

49. Thus

50. where

51. Note: for n = 2

52. Also

53. Now

54. and This verifies that we have calculated the inverse

55. Block Matrices Let the n × m matrix be partitioned into sub-matrices A 11, A 12, A 21, A 22, Similarly partition the m × k matrix

56. Product of Blocked Matrices Then

57. The Inverse of Blocked Matrices Let the n × n matrix be partitioned into sub-matrices A 11, A 12, A 21, A 22, Similarly partition the n × n matrix Suppose that B = A -1

58. Product of Blocked Matrices Then

59. Hence From (1) From (3)

60. Hence using the Woodbury Theorem or Similarly

61. From and similarly

62. Summarizing Let Suppose that A -1 = B then

63. Example Let Find A -1 = B

65. The transpose of a matrix Consider the n × m matrix, A is called the transpose of A then the m × n matrix, (also denoted by A T )

66. Symmetric Matrices An n × n matrix, A, is said to be symmetric if Note:

67. The trace and the determinant of a square matrix Let A denote then n × n matrix Then

68. also where

69. Some properties

70. Some additional Linear Algebra

71. Inner product of vectors Let denote two p × 1 vectors. Then.

72. Note: Let denote two p × 1 vectors. Then.

73. Note: Let denote two p × 1 vectors. Then. 0

74. Special Types of Matrices 1.Orthogonal matrices –A matrix is orthogonal if P'P = PP' = I –In this cases P -1 =P'. –Also the rows (columns) of P have length 1 and are orthogonal to each other

75. then Suppose P is an orthogonal matrix Let denote p × 1 vectors. Orthogonal transformation preserve length and angles – Rotations about the origin, Reflections

76. The following matrix P is orthogonal Example

77. Special Types of Matrices (continued) 2.Positive definite matrices –A symmetric matrix, A, is called positive definite if: –A symmetric matrix, A, is called positive semi definite if:

78. If the matrix A is positive definite then

79. Theorem The matrix A is positive definite if

80. Special Types of Matrices (continued) 3.Idempotent matrices –A symmetric matrix, E, is called idempotent if: –Idempotent matrices project vectors onto a linear subspace

81. Definition Let A be an n × n matrix Let then is called an eigenvalue of A and and is called an eigenvector of A and

82. Note:

83. = polynomial of degree n in. Hence there are n possible eigenvalues 1, …, n

84. Proof A is positive definite if be an eigenvalue and Thereom If the matrix A is symmetric then the eigenvalues of A, 1, …, n,are real. Thereom If the matrix A is positive definite then the eigenvalues of A, 1, …, n, are positive. Let corresponding eigenvector of A.

85. Proof: Note Thereom If the matrix A is symmetric and the eigenvalues of A are 1, …, n, with corresponding eigenvectors If i ≠ j then

86. Thereom If the matrix A is symmetric with distinct eigenvalues, 1, …, n, with corresponding eigenvectors Assume

87. proof Noteand P is called an orthogonal matrix

88. therefore thus

89. Comment The previous result is also true if the eigenvalues are not distinct. Namely if the matrix A is symmetric with eigenvalues, 1, …, n, with corresponding eigenvectors of unit length

90. An algorithm for computing eigenvectors, eigenvalues of positive definite matrices Generally to compute eigenvalues of a matrix we need to first solve the equation for all values of. –|A – I| = 0 (a polynomial of degree n in ) Then solve the equation for the eigenvector

91. Recall that if A is positive definite then It can be shown that and that

92. Thus for large values of m The algorithim 1.Compute powers of A - A 2, A 4, A 8, A 16,... 2.Rescale (so that largest element is 1 (say)) 3.Continue until there is no change, The resulting matrix will be 4.Find 5. Find

93. To find 6.Repeat steps 1 to 5 with the above matrix to find 7.Continue to find

94. Example A =

95. Differentiation with respect to a vector, matrix

96. Differentiation with respect to a vector Let denote a p × 1 vector. Let denote a function of the components of.

97. 1. Suppose Rules

98. 2. Suppose

99. Example 1. Determine when is a maximum or minimum. solution

100. 2. Determine whenis a maximum if  is the Lagrange multiplier. solution Assume A is a positive definite matrix. This shows that is an eigenvector of A. Thus is the eigenvector of A associated with the largest eigenvalue,.

101. Differentiation with respect to a matrix Let X denote a q × p matrix. Let f (X) denote a function of the components of X then:

102. Example Let X denote a p × p matrix. Let f (X) = ln |X| Solution = (i,j) th element of X -1 Note X ij are cofactors

103. Example Let X and A denote p × p matrices. Solution Let f (X) = tr (AX)

104. Differentiation of a matrix of functions Let U = (u ij ) denote a q × p matrix of functions of x then:

105. Rules:

106. Proof:

109. The Generalized Inverse of a matrix

110. Recall B (denoted by A -1 ) is called the inverse of A if AB = BA = I A -1 does not exist for all matrices A A -1 exists only if A is a square matrix and |A| ≠ 0 If A -1 exists then the system of linear equations has a unique solution

111. Definition B (denoted by A - ) is called the generalized inverse (Moore – Penrose inverse) of A if 1. ABA = A 2. BAB = B 3. (AB)' = AB 4. (BA)' = BA Note: A - is unique Proof: Let B 1 and B 2 satisfying 1. AB i A = A 2. B i AB i = B i 3. (AB i ) ' = AB i 4. (B i A) ' = B i A

112. Hence B 1 = B 1 AB 1 = B 1 AB 2 AB 1 = B 1 (AB 2 ) ' (AB 1 ) ' = B 1 B 2 ' A ' B 1 ' A ' = B 1 B 2 ' A ' = B 1 AB 2 = B 1 AB 2 AB 2 = (B 1 A)(B 2 A)B 2 = (B 1 A) ' (B 2 A) ' B 2 = A ' B 1 ' A ' B 2 ' B 2 = A ' B 2 ' B 2 = (B 2 A) ' B 2 = B 2 AB 2 = B 2 The general solution of a system of Equations The general solution where is arbitrary

113. Suppose a solution exists Let

114. Calculation of the Moore-Penrose g-inverse Let A be a p×q matrix of rank q < p, Proof thus also

115. Let B be a p×q matrix of rank p < q, Proof thus also

116. Let C be a p×q matrix of rank k < min(p,q), Proof is symmetric, as well as then C = AB where A is a p×k matrix of rank k and B is a k×q matrix of rank k

117. References 1. Matrix Algebra Useful for Statistics, Shayle R. Searle 2. Mathematical Tools for Applied Multivariate Analysis, J. Douglas Carroll, Paul E. Green