1. Object Orie’d Data Analysis, Last Time Cornea Data –Images (on the disk) as data objects –Zernike basis representations Outliers in PCA (have major influence) Robust PCA (downweight outliers) –Eigen-analysis of robust covariance matrix –Projection Pursuit –Spherical PCA

2. Cornea Data Cornea: Outer surface of the eye Driver of Vision: Curvature of Cornea Sequence of Images Objects: Images on the unit disk Curvature as “ Heat Map ” Special Thanks to K. L. Cohen, N. Tripoli, UNC Ophthalmology

3. PCA of Cornea Data PC2PC2 Affected by Outlier: How bad is this problem? View 1: Statistician: Arrggghh!!!! Outliers are very dangerous Can give arbitrary and meaningless dir ’ ns What does 4% of MR SS mean???

4. Robust PCA What is multivariate median? There are several! ( “ median ” generalizes in different ways) i.Coordinate-wise median Often worst Not rotation invariant (2-d data uniform on “ L ” ) Can lie on convex hull of data (same example) Thus poor notion of “ center ”

5. Robust PCA M-estimate (cont.): “ Slide sphere around until mean (of projected data) is at center ”

6. Robust PCA M-estimate (cont.): Additional literature: Called “ geometric median ” (long before Huber) by: Haldane (1948) Shown unique for by: Milasevic and Ducharme (1987) Useful iterative algorithm: Gower (1974) (see also Sec. 3.2 of Huber). Cornea Data experience: works well for

7. Robust PCA Robust PCA 3: Spherical PCA

8. Robust PCA Recall M-estimate for Cornea Data: Sample Mean M-estimate Definite improvement But outliers still have some influence Projection onto sphere distorts the data

9. Robust PCA Useful View: Parallel Coordinates Plot X-axis: Zernike Coefficient Number Y-axis: Coefficient

10. Robust PCA Spherical PCA Problem: Magnification of High Freq. Coeff ’ s Solution: Elliptical Analysis Main idea: project data onto suitable ellipse, not sphere Which ellipse? (in general, this is problem that PCA solves!) Simplification: Consider ellipses parallel to coordinate axes

11. Robust PCA Rescale Coords Unscale Coords Spherical PCA

12. Robust PCA Elliptical Analysis (cont.): Simple Implementation, via coordinate axis rescaling Divide each axis by MAD Project Data to sphere (in transformed space) Return to original space (mul ’ ply by orig ’ l MAD) for analysis Where MAD = Median Absolute Deviation (from median) (simple, high breakdown, outlier resistant measure of “ scale ” )

13. Robust PCA Elliptical Estimate of “ center ” : Do M-estimation in transformed space (then transform back) Results for cornea data: Sample Mean Spherical Center Elliptical Center Elliptical clearly best Nearly no edge effect

14. Robust PCA Elliptical PCA for cornea data: Original PC1Original PC1, Elliptical PC1Elliptical PC1 Still finds overall curvature & correlated astigmatism Minor edge effects almost completely gone Original PC2Original PC2, Elliptical PC2Elliptical PC2 Huge edge effects dramatically reduced Still find steeper superior vs. inferior

15. Robust PCA Elliptical PCA for Cornea Data (cont.): Original PC3Original PC3, Elliptical PC3Elliptical PC3 -Edge effects greatly diminished But some of against the rule astigmatism also lost Price paid for robustness Original PC4Original PC4, Elliptical PC4Elliptical PC4 Now looks more like variation on astigmatism???

16. Robust PCA Current state of the art: Spherical & Elliptical PCA are a kludge Put together by Robustness Amateurs To solve this HDLSS problem Good News: Robustness Pros are now in the game: Hubert, et al (2005)

17. Robust PCA Disclaimer on robust analy ’ s of Cornea Data: Critical parameter is “ radius of analysis ”, : Shown above, Elliptical PCA very effective : Stronger edge effects, Elliptical PCA less useful : Edge effects weaker, don ’ t need robust PCA

18. Big Picture View of PCA Above View: PCA finds optimal directions in point cloud Maximize projected variation Minimize residual variation (same by Pythagorean Theorem) Notes: Get useful insights about data Shows can compute for any point cloud But there are other views.

19. Big Picture View of PCA Alternate Viewpoint: Gaussian Likelihood When data are multivariate Gaussian PCA finds major axes of ellipt ’ al contours of Probability Density Maximum Likelihood Estimate Mistaken idea: PCA only useful for Gaussian data

20. Big Picture View of PCA Simple check for Gaussian distribution: Standardized parallel coordinate plot Subtract coordinate wise median (robust version of mean) (not good as “ point cloud center ”, but now only looking at coordinates) Divide by MAD / MAD(N(0,1)) (put on same scale as “ standard deviation ” ) See if data stays in range – 3 to +3

21. Big Picture View of PCA E.g. Cornea Data: Standardized Parallel Coordinate Plot

22. Big Picture View of PCA Check for Gaussian dist ’ n: Stand ’ zed Parallel Coord. Plot E.g. Cornea data (recall image view of data)image view of data Several data points > 20 “ s.d.s ” from the center Distribution clearly not Gaussian Strong kurtosis But PCA still gave strong insights

23. Correlation PCA A related (& better known) variation of PCA: Replace cov. matrix with correlation matrix I.e. do eigen analysis of Where

24. Correlation PCA Why use correlation matrix? Reason 1: makes features “ unit free ” e.g. M-reps: mix “ lengths ” with “ angles ” (degrees? radians?) Are “ directions in point cloud ” meaningful or useful? Will unimportant directions dominate?

25. Correlation PCA Alternate view of correlation PCA: Ordinary PCA on standardized (whitened) data I.e. SVD of data matrix Distorts “ point cloud ” along coord. dir ’ ns

26. Correlation PCA Reason 2 for correlation PCA: “ Whitening ” can be a useful operation (e.g. M-rep Corp. Call. data)M-rep Corp. Call. data Caution: sometimes this is not helpful (can lose important structure this way) E.g. 1: Cornea data Elliptical vs. Spherical PCA

27. Correlation PCA E.g. 2: Fourier Boundary Corp. Call. DataFourier Boundary Corp. Call. Data Recall Standard PC1, PC2, PC3:PC1PC2PC3 –Gave useful insights Correlation PC1, PC2, PC3PC1PC2PC3 –Not useful directions –No insights about population –Driven by high frequency noise artifacts –Reason: whitening has damped the important structure –By magnifying high frequency noise

28. Correlation PCA Parallel coordinates show what happened: Most Variation in low frequencies Whitening gives major distortion Skews PCA towards noise directions

29. Correlation PCA Summary on correlation PCA: Can be very useful (especially with noncommensurate units) Not always, can hide important structure To make choice: Decide whether whitening is useful My personal use of correlat ’ n PCA is rare Other people use it most of the time

30. PCA to find clusters Toy Example (2 clusters)

31. PCA to find clusters Toy Example (2 clusters) Dominant direction finds very distinct clusters Skewer through meatballs (in point cloud space) Shows up clearly in scores plot An important use of scores plot is finding such structure

32. PCA to find clusters Recall Toy Example with more clusters:

33. PCA to find clusters Best revealed by 2d scatterplots (4 clusters):

34. PCA to find clusters Caution: there are limitations … Revealed by NCI60 data –Recall Microarray data –With 8 known different cancer types Could separate out clusters –Using specialized DWD directions But can these be found using PCA? –Recall only finds dirn ’ s of max variation –Does not use class label information

35. PCA to find clusters Specific DWD directions, for NCI60 Data: Good Separation Of Cancer Types

36. PCA to find clusters PCA directions, for NCI60 Data: See clusters? PC2? PC1-3?

37. PCA to find clusters PCA directions, for NCI60 Data: PC2: Melanoma PC1-3 Leukemia

38. PCA to find clusters PC5-8 directions, for NCI60 Data: See clusters? PC5? Others?

39. PCA to find clusters PC5-8 directions, for NCI60 Data: PC5: Leukemia & Renal PC6: CNS???

40. PCA to find clusters PC9-12 directions, for NCI60 Data: See clusters? Maybe Not???

41. PCA to find clusters PC9-12 directions, for NCI60 Data: Some New Combos ???

42. PCA to find clusters Off Diagonal PC1-4 & 5-8, NCI60 Data: See clusters? PC1 & 5 PC2

43. PCA to find clusters Off Diagonal PC1-4 & 5-8, NCI60 Data: PC1 & 5 Renal Leukemia (separated) PC2: Melanoma

44. PCA to find clusters Off Diagonal PC1-4 & 9-12, NCI60 Data: See clusters? PC2 Others???

45. PCA to find clusters Off Diagonal PC1-4 & 9-12, NCI60 Data: PC2 Melanoma PC1 Colon???

46. PCA to find clusters Off Diagonal PC5-8 & 9-12, NCI60 Data: See clusters? PC5 Others???

47. PCA to find clusters Off Diagonal PC5-8 & 9-12, NCI60 Data: PC5 Leukemia & Renal Others???

48. PCA to find clusters Main Lesson: PCA is limited a finding clusters Revealed by NCI60 data –Recall Microarray data –With 8 known different cancer types PCA does not find all 8 clusters –Recall only finds dirn ’ s of max variation –Does not use class label information

49. PCA to find clusters A deeper example: Mass Flux Data Data from Enrica Bellone, –National Center for Atmospheric Research Mass Flux for quantifying cloud types How does mass change when moving into a cloud

50. PCA to find clusters PCA of Mass Flux Data:

51. PCA to find clusters Summary of PCA of Mass Flux Data: Mean: Captures General mountain shape PC1: Generally overall height of peak –shows up nicely in mean +- plot (2nd col) –3 apparent clusters in scores plot –Are those “ really there ” ? –If so, could lead to interesting discovery –If not, could waste effort in investigation

52. PCA to find clusters Summary of PCA of Mass Flux Data: PC2: Location of peak again mean +- plot very useful here PC3: Width adjustment again see most clearly in mean +- plot Maybe non-linear modes of variation???

53. PCA to find clusters Return to Investigation of PC1 Clusters: Can see 3 bumps in smooth histogram Main Question: Important structure or sampling variability? Approach: SiZer (SIgnificance of ZERo crossings of deriv.)

54. SiZer Background Two Major Settings: 2-d scatterplot smoothing 1-d histograms (continuous data, not discrete bar plots) Central Question: Which features are “ really there ” ?

55. SiZer Background Two Major Settings: 2-d scatterplot smoothing 1-d histograms (continuous data, not discrete bar plots) Central Question: Which features are really there? Solution, Part 1: Scale space Solution, Part 2: SiZer

56. SiZer Background Smoothing Setting 1: 2-d Scatterplots Fossil Data from T. Bralower, Dept. Geological Sciences, UNC (now Penn. State) Strontium Ratio in fossil shells reflects global sea level surrogate for climate over millions of years

57. SiZer Background Bralower ’ s Fossil Data - Global Climate

58. SiZer Background Smooths - Suggest Structure - Real?

59. SiZer Background Smooths of Fossil Data (details given later) Dotted line: undersmoothed (feels sampling variability) Dashed line: oversmoothed (important features missed?) Solid line: smoothed about right? Central question: Which features are “ really there ” ?

60. SiZer Background My favorite scatterplot smoothing method: (others disagree) Local Linear Smoothing Main idea: (illustrated by toy example)toy example Use kernel window to determine neighborhood Then fit a line within the window Then slide window along Window Width,, is critical

61. SiZer Background Smoothing Setting 2: Histograms Family Income Data: British Family Expenditure Survey For the year 1975 Distribution of Family Incomes ~ 7000 families