Outline Variance Matrix of Stochastic Variables and Orthogonal Transforms Principle Component Analysis Generalized Eigenvalue Decomposition

Random Variable Transform Stochastic Variables Mean: E[x] = mx Variance: E[(x – mx)(x – mx)H] = Cx Eigenvalue Decomposition: Cx =UΣUH Properties on Covariance Matrix Hermitian Matrix Semi-positive Definite

Random Variable Transform Isotropic circular transform y = Σ–1/2UH(x – mx) Properties on variable: y Zero mean, E[y] = 0 Independent distributed stochastic variables with unit variance Gaussian Random Variable Gaussian variable: x ~ CN(mx, Cx) Gaussian variable: y ~ CN(0, I)

Principal Component Analysis Zero Mean Stochastic Variable x Lower Rank Representation of x Approximate random variable x using another low- rank random variable x0 Low rank random variable: low rank covariance matrix Minimize the Distortion E||x – x0||2

Principal Component Analysis Lower Rank Representation of x Transform x w, where components of w are independent Linear transform: w = QHx, x = Qw, Represent x, using the M most important component of w, w[1:k], with the largest variance Transform: x0 = Q[1:k]w[1:k] The Above Processes Minimizes the Distortion E||x – x0||2

Principal Component Analysis Recall the linear transform: w = QHx, Minimize the distortion E||x – Q[1:k]w[1:k]||2 Solution: Eigenvalue decomposition of Cx Results: Q[1:k] the eigenvectors corresponding to the k maximum eigenvalues Mostly Effective when Cx Some Eigenvalues of x Are Negligible

Principal Component Analysis PCA Maximizes the Trace under Orthogonal Transform, E[x] = 0 Transform y = Bx, B orthogonal matrix, size p*n B corresponds to the maximum p eigenvalues Minimizing the Trace under Orthogonal Transform, E[x] = 0 B corresponds to the minimum p eigenvalues

Principal Component Analysis Algorithm for a Data Set, for a Data Set of vectors x1, x2, …, xN Obtain the empirical mean Obtain the deviations from the empirical mean Obtain the empirical covariance matrix Eigenvalue decomposition of the empirical covariance matrix Obtain the largest K eigenvalues and the corresponding eigenvectors

KL Analysis on the Stochastic Processes KL Expansion Expansion Orthogonal basis Orthogonal coefficients The time-domain correlation Eigenvalue decomposition of the function of the time- domain correlation