1. Principal Components Analysis BMTRY 726 3/27/14

2. Uses Goal: Explain the variability of a set of variables using a “small” set of linear combinations of those variables Why: There are several reasons we may want to do this (1) Dimension Reduction (use k of p components) -Note, total variability still requires p components (2) Identify “hidden” underlying relationships (i.e. patterns in the data) -Use these relationships in further analyses (3) Select subsets of variables

3. “Exact” Principal Components We can represent data X as linear combinations of p random measurements on j = 1,2,…,n subjects

4. “Exact” Principal Components Principal components are those combinations that are: (1) Uncorrelated (linear combinations Y 1, Y 2,…, Y p ) (2) Variance as large as possible (3) Subject to:

5. Finding PC’s Under Constraints So how do we find PC’s that meet the constraints we just discussed? We want to maximize subject to the constraint that This constrained maximization problem can be done using the method of Lagrange multipliers Thus we want to maximize the function

6. Finding PC’s Under Constraints Differentiate w.r.t a i :

7. Finding PC’s Under Constraints But how do we choose our eigenvector (i.e. which eigenvector corresponds to which PC?) We can see that what we want to maximize is So we choose i to be as large as possible If 1 is our largest eigenvalue with corresponding eigenvector e i then the solution for our max is

8. Finding PC’s Under Constraints Recall we had a second constraint We could conduct a second Lagrangian maximization to find our second PC However we already know that eigenvectors are independent (so this constraint is met) We choose the order of the PCs by the magnitude of the eigenvalues

9. “Exact” Principal Components So we can compute the PCs from the variance matrix of X,  :

10. Properties We can also find the moments of our PC’s

11. Properties We can also find the moments of our PC’s

12. Properties Normality assumption not required to find PC’s If X j ~ N p (  ) then: Total Variance:

13. Principal Components Consider data with p random measures on j = 1,2,…,n subjects For the j th subject we then have the random vector X1X1 X2X2 11 22

14. Graphic Representation

15. Now suppose X 1, X 2 ~ N 2 ( ,  ) Y 1 axis selected to maximize variation in the scores Y 2 axis must be orthogonal to Y 1 and maximize variation in the scores Y2Y2 X2X2 Y1Y1 X1X1

16. Dimension Reduction Proportion of total variance accounted for by the first k components is If the proportion of variance accounted for by the first k principal components is large, we might want to restrict our attention to only these first k components Keep in mind, components are simply linear combinations of the original p measurements Ideally look for meaningful interpretations of our choose k components

17. PC’s from Standardized Variables We may want to standardize our variables before finding PCs

18. PC’s from Standardized Variables So the covariance of V equals the correlation of X We can define our PC’s for Z the same way as before….

19. Compare Standardized/Non-standardized PCs

20. Estimation In general we do not know what  is- we must estimate if from the sample So what are our estimated principal components?

21. Sample Properties In general we do not know what  is- estimate it from sample So what are our estimated principal components?

22. Centering We often center our observations before defining our PCs The centered PCs are found according to:

23. Example Jolicoeur and Mosimann (1960) conducted a study looking at the relationship between size and shape of painted turtle carapaces. We can develop PC’s for natural log of length, width, and height of female turtles’ carapaces

24. Example The first PC is: This might be interpreted as an overall size component Shell dimensions small Small values y 1 Shell dimensions large Large values y 1

25. Example The second PC is: Emphasizes contrast between length and height of the shell Small values y 2 Large values y 2

26. Example The third PC is: Emphasizes contrast between width and length of the shell Small values y 3 Large values y 3

27. Example Consider the proportion of variability accounted for by each PC

28. Example How are the PCs correlated with each of the x’ s? Then Trait x1x1 0.990.09-0.08 X2X2 0.990.060.15 X3X3 0.99-0.14-0.01

29. Interpretation of PCs Consider data x 1, x 2, …., x p : PCs are actually projections onto the estimated eigenvectors -1 st PC is the one with the largest projection -For data reduction, only use PCA if the eigenvalues vary -If x ’s are uncorrelated, we can’t really do data reduction

30. Choosing Number of PCs Often the goal of PCA is dimension reduction of data Select a limited number of PCs that capture majority of the variability in the data How do we decide how many PCs to include: 1. Scree plot: plot of versus i 2. Select all PCs with (for standardized observations) 3. Choose some proportion of the variance you want to account for

31. Scree Plots

32. Choosing Number of PCs Should principal components that only account for a small proportion of variance always be ignored? Not necessarily, they may indicate near perfect colinearities among traits In the turtle example, this is true-very little variation of the variation in shell measurements can be attributed to the 2 nd and 3 rd components

33. Large Sample Properties If n is large, there are nice properties we can use

34. Large Sample Properties Also for our estimated eigenvectors These results assume that X 1, X 2, …., X n are N( ,  )

35. Summary Principal component analysis most useful for dimensionality reduction Can also be used for identifying colinear variables Note, use of PCA in a regression setting is therefore one way to handle multi-colinearity A caveat… principal components can be difficult to interpret and should therefore be used with caution