Presentation on theme: "Eigen Decomposition and Singular Value Decomposition"— Presentation transcript:
1 Eigen Decomposition and Singular Value Decomposition Based on the slides by Mani ThomasModified and extended by Longin Jan Latecki
2 Introduction Eigenvalue decomposition Spectral decomposition theoremPhysical interpretation of eigenvalue/eigenvectorsSingular Value DecompositionImportance of SVDMatrix inversionSolution to linear system of equationsSolution to a homogeneous system of equationsSVD application
3 A(x) = (Ax) = (x) = (x) What are eigenvalues?Given a matrix, A, x is the eigenvector and is the corresponding eigenvalue if Ax = xA must be square and the determinant of A - I must be equal to zeroAx - x = 0 ! (A - I) x = 0Trivial solution is if x = 0The non trivial solution occurs when det(A - I) = 0Are eigenvectors are unique?If x is an eigenvector, then x is also an eigenvector and is an eigenvalueA(x) = (Ax) = (x) = (x)
4 Calculating the Eigenvectors/values Expand the det(A - I) = 0 for a 2 £ 2 matrixFor a 2 £ 2 matrix, this is a simple quadratic equation with two solutions (maybe complex)This “characteristic equation” can be used to solve for x
5 Eigenvalue example Consider, The corresponding eigenvectors can be computed asFor = 0, one possible solution is x = (2, -1)For = 5, one possible solution is x = (1, 2)For more information: Demos in Linear algebra by G. Strang,
6 Physical interpretation Consider a covariance matrix, A, i.e., A = 1/n S ST for some SError ellipse with the major axis as the larger eigenvalue and the minor axis as the smaller eigenvalue
7 Physical interpretation Original Variable AOriginal Variable BPC 1PC 2Orthogonal directions of greatest variance in dataProjections along PC1 (Principal Component) discriminate the data most along any one axis
8 Physical interpretation First principal component is the direction of greatest variability (covariance) in the dataSecond is the next orthogonal (uncorrelated) direction of greatest variabilitySo first remove all the variability along the first component, and then find the next direction of greatest variabilityAnd so on …Thus each eigenvectors provides the directions of data variances in decreasing order of eigenvaluesFor more information: See Gram-Schmidt Orthogonalization in G. Strang’s lectures
12 Eigen/diagonal Decomposition Let be a square matrix with m linearly independent eigenvectors (a “non-defective” matrix)Theorem: Exists an eigen decomposition(cf. matrix diagonalization theorem)Columns of U are eigenvectors of SDiagonal elements of are eigenvalues ofUnique for distinct eigen-valuesdiagonal
13 Diagonal decomposition: why/how Let U have the eigenvectors as columns:Then, SU can be writtenThus SU=U, or U–1SU=And S=UU–1.
14 Diagonal decomposition - example RecallThe eigenvectors and formRecallUU–1 =1.Inverting, we haveThen, S=UU–1 =
15 Example continued Let’s divide U (and multiply U–1) by Then, S= Q (Q-1= QT )Why? Stay tuned …
16 Symmetric Eigen Decomposition If is a symmetric matrix:Theorem: Exists a (unique) eigen decompositionwhere Q is orthogonal:Q-1= QTColumns of Q are normalized eigenvectorsColumns are orthogonal.(everything is real)
17 Spectral Decomposition theorem If A is a symmetric and positive definite k £ k matrix (xTAx > 0) with i (i > 0) and ei, i = 1 k being the k eigenvector and eigenvalue pairs, thenThis is also called the eigen decomposition theoremAny symmetric matrix can be reconstructed using its eigenvalues and eigenvectors
18 Example for spectral decomposition Let A be a symmetric, positive definite matrixThe eigenvectors for the corresponding eigenvalues areConsequently,
19 Singular Value Decomposition If A is a rectangular m £ k matrix of real numbers, then there exists an m £ m orthogonal matrix U and a k £ k orthogonal matrix V such that is an m £ k matrix where the (i, j)th entry i ¸ 0, i = 1 min(m, k) and the other entries are zeroThe positive constants i are the singular values of AIf A has rank r, then there exists r positive constants 1, 2,r, r orthogonal m £ 1 unit vectors u1,u2,,ur and r orthogonal k £ 1 unit vectors v1,v2,,vr such thatSimilar to the spectral decomposition theorem
20 Singular Value Decomposition (contd.) If A is a symmetric and positive definite thenSVD = Eigen decompositionEIG(i) = SVD(i2)Here AAT has an eigenvalue-eigenvector pair (i2,ui)Alternatively, the vi are the eigenvectors of ATA with the same non zero eigenvalue i2
21 Example for SVD Let A be a symmetric, positive definite matrix U can be computed asV can be computed as
22 Example for SVDTaking 21=12 and 22=10, the singular value decomposition of A isThus the U, V and are computed by performing eigen decomposition of AAT and ATAAny matrix has a singular value decomposition but only symmetric, positive definite matrices have an eigen decomposition
23 Applications of SVD in Linear Algebra Inverse of a n £ n square matrix, AIf A is non-singular, then A-1 = (UVT)-1= V-1UT where-1=diag(1/1, 1/1,, 1/n)If A is singular, then A-1 = (UVT)-1¼ V0-1UT where0-1=diag(1/1, 1/2,, 1/i,0,0,,0)Least squares solutions of a m£n systemAx=b (A is m£n, m¸n) =(ATA)x=ATb ) x=(ATA)-1 ATb=A+bIf ATA is singular, x=A+b¼ (V0-1UT)b where 0-1 = diag(1/1, 1/2,, 1/i,0,0,,0)Condition of a matrixCondition number measures the degree of singularity of ALarger the value of 1/n, closer A is to being singular
24 Applications of SVD in Linear Algebra Homogeneous equations, Ax = 0Minimum-norm solution is x=0 (trivial solution)Impose a constraint,“Constrained” optimization problemSpecial CaseIf rank(A)=n-1 (m ¸ n-1, n=0) then x= vn ( is a constant)Genera CaseIf rank(A)=n-k (m ¸ n-k, n-k+1== n=0) then x=1vn-k+1++kvn with 21++2n=1Has appeared beforeHomogeneous solution of a linear system of equationsComputation of Homogrpahy using DLTEstimation of Fundamental matrixFor proof: Johnson and Wichern, “Applied Multivariate Statistical Analysis”, pg 79
25 What is the use of SVD?SVD can be used to compute optimal low-rank approximations of arbitrary matrices.Face recognitionRepresent the face images as eigenfaces and compute distance between the query face image in the principal component spaceData miningLatent Semantic Indexing for document extractionImage compressionKarhunen Loeve (KL) transform performs the best image compressionIn MPEG, Discrete Cosine Transform (DCT) has the closest approximation to the KL transform in PSNR
26 Singular Value Decomposition Illustration of SVD dimensions and sparseness
27 SVD example Let Thus m=3, n=2. Its SVD is Typically, the singular values arranged in decreasing order.
28 Low-rank Approximation SVD can be used to compute optimal low-rank approximations.Approximation problem: Find Ak of rank k such thatAk and X are both mn matrices.Typically, want k << r.Frobenius norm
29 Low-rank Approximation Solution via SVDset smallest r-ksingular values to zerokcolumn notation: sumof rank 1 matrices
30 Approximation error How good (bad) is this approximation? It’s the best possible, measured by the Frobenius norm of the error:where the i are ordered such that i i+1.Suggests why Frobenius error drops as k increased.
Your consent to our cookies if you continue to use this website.