Outline Singular Value Decomposition Example of PCA: Eigenfaces

Singular Value Decomposition A sample set of M N-dimensional points can be written as a matrix each row of which represents a sample point. PCA of the sample is then equivalent to solving the SVD problem for this matrix; that is, finding the decomposition W is diagonal matrix, V is orthogonal square matrix, columns of U are orthogonal, columns of V are orthogonal : Also

SVD-remarks Meaning: columns of V represent the KL transform axes, ordered by respective values in W (singular values), which are amount of variation, in descending order. The new axes (columns of V) are also eigenvectors of XXT if X is square matrix From the orthogonal basis vectors given as columns of V we omit those to which correspond small values in W. SVD provides unique decomposition for the given data. Taking the first m<N eigenvectors (rows in VT) we get the optimal approximation in the sense of L2 norm.

SVD-remarks When M<N the singular values wj for j=M+1,…,N are zero.