1. Object Orie’d Data Analysis, Last Time
Mildly Non-Euclidean Spaces
Strongly Non-Euclidean Spaces
Tree spaces
No Tangent Plane
Classification - Discrimination
Mean Difference (Centroid method)
Fisher Linear Discrimination
Graphical Viewpoint (nonparametric)
Maximum Likelihood Derivation (Gaussian based)

2. Classification - Discrimination
Background: Two Class (Binary) version:
Using “training data” from
Class +1 and Class -1
Develop a “rule” for
assigning new data to a Class
Canonical Example: Disease Diagnosis
New Patients are “Healthy” or “Ill”
Determined based on measurements

3. Classification - Discrimination
Important Distinction: Classification vs. Clustering Classification: Class labels are known, Goal: understand differences Clustering: Goal: Find class labels (to be similar) Both are about clumps of similar data, but much different goals

4. Classification - Discrimination
Important Distinction: Classification vs. Clustering Useful terminology: Classification: supervised learning Clustering: unsupervised learning

5. Fisher Linear Discrimination
Graphical Introduction (non-Gaussian):
HDLSSod1egFLD.ps

6. Classical Discrimination
Above derivation of FLD was:
Nonstandard
Not in any textbooks(?)
Nonparametric (don’t need Gaussian data)
I.e. Used no probability distributions
More Machine Learning than Statistics

7. Classical Discrimination
Summary of FLD vs. GLR:
Tilted Point Clouds Data
FLD good
GLR good
Donut Data
FLD bad
X Data
GLR OK, not great
Classical Conclusion: GLR generally better
(will see a different answer for HDLSS data)

8. Classical Discrimination
FLD Generalization II (Gen. I was GLR)
Different prior probabilities
Main idea:
Give different weights to 2 classes
I.e. assume not a priori equally likely
Development is “straightforward”
Modified likelihood
Change intercept in FLD
Won’t explore further here

9. Classical Discrimination
FLD Generalization III
Principal Discriminant Analysis
Idea: FLD-like approach to > two classes
Assumption: Class covariance matrices are the same (similar)
(but not Gaussian, same situation as for FLD)
Main idea:
Quantify “location of classes” by their means

10. Classical Discrimination
Principal Discriminant Analysis (cont.) Simple way to find “interesting directions” among the means: PCA on set of means i.e. Eigen-analysis of “between class covariance matrix” Where Aside: can show: overall

11. Classical Discrimination
Principal Discriminant Analysis (cont.) But PCA only works like Mean Difference, Expect can improve by taking covariance into account. Blind application of above ideas suggests eigen-analysis of:

12. Classical Discrimination
Principal Discriminant Analysis (cont.)
There are:
smarter ways to compute
(“generalized eigenvalue”)
other representations
(this solves optimization prob’s)
Special case: 2 classes,
reduces to standard FLD
Good reference for more: Section 3.8 of:
Duda, Hart & Stork (2001)

13. Classical Discrimination
Summary of Classical Ideas:
Among “Simple Methods”
MD and FLD sometimes similar
Sometimes FLD better
So FLD is preferred
Among Complicated Methods
GLR is best
So always use that
Caution:
Story changes for HDLSS settings

14. (requires root inverse covariance)
HDLSS Discrimination
Recall main HDLSS issues:
Sample Size, n < Dimension, d
Singular covariance matrix
So can’t use matrix inverse
I.e. can’t standardize (sphere) the data
(requires root inverse covariance)
Can’t do classical multivariate analysis

15. HDLSS Discrimination An approach to non-invertible covariances:
Replace by generalized inverses
Sometimes called pseudo inverses
Note: there are several
Here use Moore Penrose inverse
As used by Matlab (pinv.m)
Often provides useful results
(but not always)
Recall Linear Algebra Review…

16. Recall Linear Algebra Eigenvalue Decomposition:
For a (symmetric) square matrix
Find a diagonal matrix
And an orthonormal matrix
(i.e )
So that: , i.e.

17. Recall Linear Algebra (Cont.)
Eigenvalue Decomp. solves matrix problems:
Inversion:
Square Root:
is positive (nonn’ve, i.e. semi) definite all

18. Recall Linear Algebra (Cont.)
Moore-Penrose Generalized Inverse:
For

19. Recall Linear Algebra (Cont.)
Easy to see this satisfies the definition of
Generalized (Pseudo) Inverse
symmetric

20. Recall Linear Algebra (Cont.)
Moore-Penrose Generalized Inverse:
Idea: matrix inverse on non-null space
of linear transformation
Reduces to ordinary inverse, in full rank case,
i.e. for r = d, so could just always use this
Tricky aspect:
“>0 vs. = 0” & floating point arithmetic

21. HDLSS Discrimination
Application of Generalized Inverse to FLD: Direction (Normal) Vector: Intercept: Have replaced by

22. HDLSS Discrimination
Toy Example: Increasing Dimension

24. Next Topics:
HDLSS Properties of FLD
Generalized inverses???
Think about HDLSS, as on 04/03/02
Maximal Data Piling
Embedding and Kernel Spaces