1. Lecture 3: A brief background to multivariate statistics
Univariate versus multivariate statistics
The material of multivariate analysis
Displaying multivariate data
The uses of multivariate statistics
A refresher of matrix algebra
Bio 8100s Applied Multivariate Biostatistics
2001

2. Multivariate versus univariate statistics
In univariate statistical analysis, we are concerned with analyzing variation in a single random variable.
In multivariate statistical analysis, we are concerned with analyzing variation in several random variables which may or may not be related.
Bio 8100s Applied Multivariate Biostatistics
2001

3. The material of multivariate analysis
Multivariate data consists of a set of measurements (usually related) of P variables X1, X2, …, XP on n sample units.
The variables Xj may be ratio, ordinal, or nominal.
Bio 8100s Applied Multivariate Biostatistics
2001

4. Example 1: Bumpus’ sparrow data
5 morphological measurements (in mm) of 49 sparrows recovered from a storm in 1898.
...
Bio 8100s Applied Multivariate Biostatistics
2001

5. Example 2: Biodiversity of SE Ontario wetlands
Species richness (number of species) of 5 different taxa in 57 wetlands in southeastern Ontario.
...
...
...
...
...
...
Bio 8100s Applied Multivariate Biostatistics
2001

6. The material of multivariate analysis
In some applications, the measured variables comprise both dependent (X) and independent (Y) variables.
The material of multivariate analysis
Bio 8100s Applied Multivariate Biostatistics
2001

7. Example 1: Pgi frequencies in California Euphydras editha colonies in relation to environmental factors.
Bio 8100s Applied Multivariate Biostatistics
2001

8. Example 2: Anurans in SE Ontario wetlands in relation to surrounding forest cover and road densities
Bio 8100s Applied Multivariate Biostatistics
2001

9. Multivariate LS estimators
The vector of sample means, variances and covariances is an estimate of the true (“population”) means, variances and covariances.
As such, inferences to the latter based on the former assume random sampling.
Population
Sample
Bio 8100s Applied Multivariate Biostatistics
2001

10. The sample covariance matrix
The sample covariance matrix is a square matrix whose diagonal elements give the sample variances for each measured variable (si2), and whose off-diagonal elements are the sample covariances between pairs of variables (cik).
Bio 8100s Applied Multivariate Biostatistics
2001

11. A review of matrix algebra
A matrix of size m x n is an array of numbers (either real or complex) with m rows and n columns.
Matrices with one column are column vectors, matrices with one row are row vectors.
Bio 8100s Applied Multivariate Biostatistics
2001

12. Special matrices A zero matrix 0 has all elements equal to zero.
A diagonal matrix T is a square matrix (m = n) with all elements equal to zero except the main diagonal.
An identity matrix I is a diagonal matrix with all diagonal terms equal to zero.
Bio 8100s Applied Multivariate Biostatistics
2001

13. Matrix operations
The transpose of a matrix A (AT) is obtained by interchanging rows and columns.
The transpose of a row vector is a column vector, and the transpose of a column vector is a row vector.
Bio 8100s Applied Multivariate Biostatistics
2001

14. The trace of a matrix
The trace of a matrix A, denoted tr(A), is the sum of the diagonal elements.
The trace is defined only for square matrices.
Bio 8100s Applied Multivariate Biostatistics
2001

15. Matrix addition and subtraction
Two matrices A and B are conformable for addition if they are of the same size (same numbers of rows and columns).
The resulting matrix A + B (A - B) is obtained by adding (subtracting) individual matrix elements.
Bio 8100s Applied Multivariate Biostatistics
2001

16. Matrix multiplication by a scalar
The multiplication of a matrix A by a scalar k involves multiplying each element of A by k.
Bio 8100s Applied Multivariate Biostatistics
2001

17. Matrix multiplication
Two matrices A (m x n) and B (n x p) are conformable for multiplication (A • B) if the number of columns in A equals the number of rows in B.
A • B and B • A are both defined only when both A and B are square, but even when true, in general A • B  B • A .
Bio 8100s Applied Multivariate Biostatistics
2001

18. Matrix inversion
The inverse of a matrix A, denoted A-1, is the matrix solving the matrix equation
where I is the identity matrix.
Only square matrices are invertible, and some matrices cannot be inverted (“singular” matrices)
Bio 8100s Applied Multivariate Biostatistics
2001

19. The covariance matrix
A multivariate sample is described by a covariance matrix, whose diagonal elements give the sample variances for each measured variable (si2), and whose off-diagonal elements are the sample covariances between pairs of variables (cik).
Bio 8100s Applied Multivariate Biostatistics
2001

20. Calculating the sample covariance matrixd = N M Q P - 1 3 2 4 7
Bio 8100s Applied Multivariate Biostatistics
2001

21. The determinant of a matrix: 2 X 2 matrices
The determinant of a matrix A, denoted det(A) or |A|, is a unique number associated with every square matrix.
In multivariate statistics, the determinant of the sample covariance matrix C plays a crucial role in hypothesis testing.
Bio 8100s Applied Multivariate Biostatistics
2001

22. Matrix inversion and the determinant: 2 X 2 matrices
If a 2 X 2 matrix A is invertible, the elements of its inverse A-1 are obtained by dividing modified elements of A by |A|
Hence, if |A| = 0, the division is undefined and the matrix is non-invertible or singular.
Bio 8100s Applied Multivariate Biostatistics
2001

23. Multivariate variance: a geometric interpretation
Larger
variance
Smaller
variance
Univariate variance is a measure of the “volume” occupied by sample points in one dimension.
Multivariate variance involving m variables is the volume occupied by sample points in an m -dimensional space. X X X1 X2
Occupied
volume
Bio 8100s Applied Multivariate Biostatistics
2001

24. Multivariate variance: effects of correlations among variables
No correlation
Multivariate variance: effects of correlations among variables X1 X2
Correlations between pairs of variables reduce the volume occupied by sample points…
…and hence, reduce the multivariate variance.
Positive
correlation
Negative
correlation X1
Occupied
volume X2
Bio 8100s Applied Multivariate Biostatistics
2001

25. C and the generalized multivariate variance
The determinant of the sample covariance matrix C is a generalized multivariate variance…
… because area2 of a parallelogram with sides given by the individual standard deviations and angle determined by the correlation between variables equals the determinant of C.
Bio 8100s Applied Multivariate Biostatistics
2001

26. The use of determinants in multivariate analysis
Univariate single-classification
ANOVA, k groups
For a univariate sample variance sa2, the multivariate analog is the determinant of the corresponding sample covariance matrix Ca, i.e., | Ca|…
… and these variances are often used in the calculation of multivariate test statistics, e.g., Wilk’s L.
Multivariate single-classification
ANOVA (MANOVA)
Bio 8100s Applied Multivariate Biostatistics
2001

27. Eigenvalues
The eigenvalues of a p X p matrix A are the p solutions, some of which may be zero, to the equation |A - lI| = 0.
The trace of a matrix is the sum of its eigenvalues…
… and the determinant of a matrix is the product of its eigenvalues.
Bio 8100s Applied Multivariate Biostatistics
2001

28. Eigenvalues and eigenvectors I
Suppose v is a vector, and L a linear transformation. If L(v) = lv, then v is an eigenvector of L associated with the eigenvalue l.
e.g., if L is the reflection in the line y = mx, then a is the eigenvector associated with eigenvalue 1, b with -1.
Note that a and b are orthogonal!
Bio 8100s Applied Multivariate Biostatistics
2001

29. Eigenvalues and eigenvectors of C
No correlation
Eigenvectors of the covariance matrix C are orthogonal directed line segments that “span” the variation in the data, and the corresponding (unsigned) eigenvalues are the length of these segments.
… so the product of the eigenvalues is the “volume” occupied by the data, i.e. the determinant of the covariance matrix. X1
Positive
correlation X2
Negative
correlation X1 X2
Bio 8100s Applied Multivariate Biostatistics
2001

30. Displaying multivariate data I: Draftman’s plots (SPLOM)
Plot pairs of variables against one another.
Advantages: need only 2 plotting dimensions, bivariate relationships among variables is clear.
Problems: no direct information on relationships in higher than 2 dimensions, relationships between objects unclear.
Bio 8100s Applied Multivariate Biostatistics
2001

31. Displaying multivariate data II: multiple 3-D plots
Plot 3 variables against one another.
Advantages: trivariate relationships among variables is clear.
Problems: no direct information on relationships in higher than 3 dimensions, relationships between objects unclear.
Bio 8100s Applied Multivariate Biostatistics
2001

32. Displaying multivariate data III: plotting index variables
Generate index variables that combine information from several measured variables, then plot these variables.
Advantages: 2- D plots make relationships among variables clear.
Disadvantages: relationships among objects unclear, key information may be lost in data reduction
Bio 8100s Applied Multivariate Biostatistics
2001

33. Displaying multivariate data IV: Icon plots
Cuon
Dingo
Prehistoric
dog
Chinese
wolf
Golden
jackal
Modern
Displaying multivariate data IV: Icon plots
Used to visualize relationships among objects, e.g. different canine groups.
Advantages: All variables displayed simultaneously.
Problems: order of display of variables arbitrary, and impressions may depend on order. Relationships among variables may be unclear. X4 X3 X5 X6 X2 X1
Bio 8100s Applied Multivariate Biostatistics
2001

34. Displaying multivariate data V: profile plots
Represent objects by lines, histograms or Fourier plots.
Advantages: All variables displayed simultaneously.
Problems: order of display of variables arbitrary, and impressions may depend on order. Relationships among variables may be unclear.
Bio 8100s Applied Multivariate Biostatistics
2001