1. Outline
Variance Matrix of Stochastic Variables and Orthogonal Transforms
Principle Component Analysis
Generalized Eigenvalue Decomposition

2. Random Variable Transform
Stochastic Variables
Mean: E[x] = mx
Variance: E[(x – mx)(x – mx)H] = Cx
Eigenvalue Decomposition: Cx =UΣUH
Properties on Covariance Matrix
Hermitian Matrix
Semi-positive Definite

3. Random Variable Transform
Isotropic circular transform
y = Σ–1/2UH(x – mx)
Properties on variable: y
Zero mean, E[y] = 0
Independent distributed stochastic variables with unit variance
Gaussian Random Variable
Gaussian variable: x ~ CN(mx, Cx)
Gaussian variable: y ~ CN(0, I)

4. Principal Component Analysis
Zero Mean Stochastic Variable x
Lower Rank Representation of x
Approximate random variable x using another low- rank random variable x0
Low rank random variable: low rank covariance matrix
Minimize the Distortion E||x – x0||2

5. Principal Component Analysis
Lower Rank Representation of x
Transform x w, where components of w are independent
Linear transform: w = QHx, x = Qw,
Represent x, using the M most important component of w, w[1:k], with the largest variance
Transform: x0 = Q[1:k]w[1:k]
The Above Processes Minimizes the Distortion
E||x – x0||2

6. Principal Component Analysis
Recall the linear transform: w = QHx,
Minimize the distortion E||x – Q[1:k]w[1:k]||2
Solution:
Eigenvalue decomposition of Cx
Results: Q[1:k] the eigenvectors corresponding to the k maximum eigenvalues
Mostly Effective when Cx Some Eigenvalues of x Are Negligible

7. Principal Component Analysis
PCA Maximizes the Trace under Orthogonal Transform, E[x] = 0
Transform y = Bx, B orthogonal matrix, size p*n
B corresponds to the maximum p eigenvalues
Minimizing the Trace under Orthogonal Transform, E[x] = 0
B corresponds to the minimum p eigenvalues

8. Principal Component Analysis
Algorithm for a Data Set, for a Data Set of vectors x1, x2, …, xN
Obtain the empirical mean
Obtain the deviations from the empirical mean
Obtain the empirical covariance matrix
Eigenvalue decomposition of the empirical covariance matrix
Obtain the largest K eigenvalues and the corresponding eigenvectors

9. KL Analysis on the Stochastic Processes
KL Expansion
Expansion
Orthogonal basis
Orthogonal coefficients
The time-domain correlation
Eigenvalue decomposition of the function of the time- domain correlation