1. Principal Component Analysis (PCA)
J.-S Roger Jang (張智星)
MIR Lab, CSIE Dept
National Taiwan University

2. Introduction to PCA PCA (Principal Component Analysis)
An effective method for reducing dataset dimensions while keeping spatial characteristics as much as possible
Characteristics:
For unlabeled data
A linear transform with solid mathematical foundation
Applications
Line/plane fitting
Face recognition

3. Problem Definition Input Output
2017/4/21
Problem Definition
Input
A dataset of n d-dim points which are zero justified:
Output
A unity vector u such that the square sum of the dataset’s projection onto u is maximized.
2017/4/21

4. Projection Angle between two vectors Projection of x onto u 2017/4/21

5. Mathematical Formulation
2017/4/21
Mathematical Formulation
Dataset representation:
X is d by n, with n>d
Projection of each column of X onto u:
Square sum:
Objective function with a constraint on u:
2017/4/21

6. Optimization of the Obj. Function
2017/4/21
Optimization of the Obj. Function
Set the gradient to zero:
 u is the eigenvector while l is the eigenvalue
When u is the eigenvector:
If we arrange eigenvalues such that:
Max of J(u) is l1, which occurs at u=u1
Min of J(u) is ld, which occurs at u=ud
2017/4/21

7. Facts about Symmetric Matrices
2017/4/21
Facts about Symmetric Matrices
A square symmetric matrix have orthogonal eigenvectors corresponding to different eigenvalues
2017/4/21

8. 2017/4/21
Conversion
Conversion between orthonormal bases
2017/4/21

9. Steps for PCA Find the sample mean: Compute the covariance matrix:
2017/4/21
Steps for PCA
Find the sample mean:
Compute the covariance matrix:
Find the eigenvalues of C and arrange them into descending order, with the corresponding eigenvectors
The transformation is , with
2017/4/21

10. PCA for TLS Problem for ordinary LS (least squares)
Not robust if the fitting line has a large slope
PCA can be used for TLS (total least squares)
PCA for TLS of lines in 2D
Zero adjustment (Prove that the TLS line goes through the mean of the dataset.)
Find the u1 & u2. Use u2 as the normal vector.
Can be extend to surfaces in 3D.

11. 2017/4/21
Tidbits
PCA is designed for unlabeled data. For classification problem, we usually resort to LDA (linear discriminant analysis) for dimension reduction.
If d>>n, then we need to have a workaround for computing the eigenvectors
2017/4/21

12. 2017/4/21
Example of PCA
IRIS dataset projection
2017/4/21

13. 2017/4/21
Weakness of PCA
Not designed for classification problem (with labeled
training data)
Ideal situation
Adversary situation
2017/4/21

14. Linear Discriminant Analysis
2017/4/21
Linear Discriminant Analysis
LDA projection onto directions that can best separate data
of different classes.
Adversary situation
for PCA
Ideal situation
for LDA
2017/4/21