E IGEN & S INGULAR V ALUE D ECOMPOSITION قسمتی از درس ریاضی مهندسی پیشرفته ( ارشد و دکتری ) دوشنبه، 3/9/90 1 دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

R ECAP : C LUSTERING 2 Hierarchical clustering Agglomerative clustering techniques Evaluation Term vs. document space clustering Multi-lingual docs Feature selection Labeling دوشنبه، 3 / 9 / 90 2

E IGENVALUES & E IGENVECTORS Eigenvectors (for a square m  m matrix S ) How many eigenvalues are there at most? دوشنبه، 3 / 9 / 90 3 only has a non-zero solution if this is a m -th order equation in λ which can have at most m distinct solutions (roots of the characteristic polynomial) – can be complex even though S is real. eigenvalue (right) eigenvector Example

M ATRIX - VECTOR MULTIPLICATION دوشنبه، 3 / 9 / 90 4 has eigenvalues 3, 2, 0 with corresponding eigenvectors On each eigenvector, S acts as a multiple of the identity matrix: but as a different multiple on each. Any vector (say x = ) can be viewed as a combination of the eigenvectors: x = 2 v v v 3

M ATRIX VECTOR MULTIPLICATION Thus a matrix-vector multiplication such as Sx ( S, x as in the previous slide) can be rewritten in terms of the eigenvalues/vectors: Even though x is an arbitrary vector, the action of S on x is determined by the eigenvalues/vectors. Suggestion: the effect of “small” eigenvalues is small. دوشنبه، 3 / 9 / 90 5

E IGENVALUES & E IGENVECTORS دوشنبه، 3 / 9 / 90 6 For symmetric matrices, eigenvectors for distinct eigenvalues are orthogonal All eigenvalues of a real symmetric matrix are real. All eigenvalues of a positive semidefinite matrix are non-negative

E XAMPLE Let Then The eigenvalues are 1 and 3 (nonnegative, real). The eigenvectors are orthogonal (and real): دوشنبه، 3 / 9 / 90 7 Real, symmetric. Plug in these values and solve for eigenvectors.

E IGEN / DIAGONAL D ECOMPOSITION Let be a square matrix with m linearly independent eigenvectors (a “non- defective” matrix) Theorem : Exists an eigen decomposition (cf. matrix diagonalization theorem) Columns of U are eigenvectors of S Diagonal elements of are eigenvalues of دوشنبه، 3 / 9 / 90 8diagonal Unique for distinct eigen- values

D IAGONAL DECOMPOSITION : WHY / HOW دوشنبه، 3 / 9 / 90 9 Let U have the eigenvectors as columns: Then, SU can be written And S=U  U –1. Thus SU=U , or U –1 SU= 

D IAGONAL DECOMPOSITION - EXAMPLE دوشنبه، 3 / 9 / Recall The eigenvectors and form Inverting, we have Then, S=U  U –1 = Recall UU – 1 =1.

E XAMPLE CONTINUED دوشنبه، 3 / 9 / Let ’ s divide U (and multiply U –1 ) by Then, S= Q (Q -1 = Q T )  Why? Stay tuned …

S YMMETRIC E IGEN D ECOMPOSITION If is a symmetric matrix: Theorem : Exists a (unique) eigen decomposition where Q is orthogonal: Q -1 = Q T Columns of Q are normalized eigenvectors Columns are orthogonal. (everything is real) دوشنبه، 3 / 9 / 90 12

E XERCISE Examine the symmetric eigen decomposition, if any, for each of the following matrices: دوشنبه، 3 / 9 / 90 13

S INGULAR V ALUE D ECOMPOSITION دوشنبه، 3/9/90 14 دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

U NDERCONSTRAINED L EAST S QUARES What if you have fewer data points than parameters in your function? Intuitively, can’t do standard least squares Recall that solution takes the form A T Ax = A T b When A has more columns than rows, A T A is singular: can’t take its inverse, etc. دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

U NDERCONSTRAINED L EAST S QUARES More subtle version: more data points than unknowns, but data poorly constrains function Example: fitting to y=ax 2 +bx+c دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

U NDERCONSTRAINED L EAST S QUARES Problem: if problem very close to singular, round off error can have a huge effect Even on “well-determined” values! Can detect this: Uncertainty proportional to covariance C = (A T A) -1 In other words, unstable if A T A has small values More precisely, care if x T (A T A)x is small for any x Idea: if part of solution unstable, set answer to 0 Avoid corrupting good parts of answer دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

S INGULAR V ALUE D ECOMPOSITION (SVD) Handy mathematical technique that has application to many problems Given any m  n matrix A, algorithm to find matrices U, V, and S such that A = U S V T U is m  m and Orthonormal S is m  n and Diagonal V is n  n and Orthonormal دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

SVD Treat as black box: code widely available In Matlab: [U,S,V]=svd(A,0) دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

SVD The S i are called the singular values of A If A is singular, some of the S i will be 0 In general rank ( A ) = number of nonzero s i SVD is mostly unique (up to permutation of singular values, or if some S i are equal) دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

S INGULAR V ALUE D ECOMPOSITION دوشنبه، 3 / 9 / 90 21

T HE S INGULAR V ALUE D ECOMPOSITION دوشنبه، 3 / 9 / r = the rank of A = number of linearly independent = number of linearly independent columns/rows columns/rows AU VTVTVTVT m x n m x m m x n n x n ··=  0 0

T HE S INGULAR V ALUE D ECOMPOSITION دوشنبه، 3 / 9 / r = the rank of A = number of linearly independent = number of linearly independent columns/rows columns/rows AU m x n m x m m x n n x n =  0 0 VTVTVTVT

SVD P ROPERTIES U, V give us orthonormal bases for the subspaces of A : 1st r columns of U : Column space of A Last m - r columns of U : Left nullspace of A 1st r columns of V : Row space of A 1st n - r columns of V : Nullspace of A IMPLICATION: Rank( A ) = r دوشنبه، 3 / 9 / 90 24

S INGULAR V ALUE D ECOMPOSITION دوشنبه، 3 / 9 / where u 1 … u r are the r orthonormal vectors that are basis of C(A) and u 1 … u r are the r orthonormal vectors that are basis of C(A) and v 1 … v r are the r orthonormal vectors that are basis of C(A T ) v 1 … v r are the r orthonormal vectors that are basis of C(A T )

M ATLAB E XAMPLE >> A = rand(3,5) دوشنبه، 3 / 9 / 90 26

M ATLAB E XAMPLE >> [U,S,V] = svd (A) دوشنبه، 3 / 9 / 90 27

SVD P ROOF Any m x n matrix A has two symmetric covariant matrices (m x m) AA T (n x n) A T A دوشنبه، 3 / 9 / 90 28

W HY IS SVD SO USEFUL ? موارد استفاده : 1. Inverses 2. Pseudo Inverse 3. Eigen values and Eigenvectors 4. Matrix equivalent using SVD as Similarity transform 5. Frobenius Norm of a Matrix 6. Matrix Liklihood دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

C ONTINUED موارد استفاده : 7. Principal Components Analysis (PCA) on: Faces and Recognition 8. Total Least Squares 9. Constrained Optimization دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

SVD AND I NVERSES 1. Application #1: inverses A -1 =( V T ) -1 S -1 U -1 = V S -1 U T Using fact that inverse = transpose for orthogonal matrices Since S is diagonal, S -1 also diagonal with reciprocals of entries of S دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

SVD AND I NVERSES A -1 =( V T ) -1 S -1 U -1 = V S -1 U T This fails when some s i are 0 It’s supposed to fail for singular matrix 2. Pseudo inverse: if s i =0, set 1/ s i to 0 (!) “Closest” matrix to inverse Defined for all (even non-square, singular, etc.) matrices Equal to ( A T A ) -1 A T if A T A invertible دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

SVD AND L EAST S QUARES Solving Ax = b by least squares x =pseudoinverse( A ) times b Compute pseudoinverse using SVD Lets you see if data is singular Even if not singular, ratio of max to min singular values (condition number) tells you how stable the solution will be Set 1/ s i to 0 if s i is small (even if not exactly 0) دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

SVD AND E IGENVECTORS Let A = USV T, and let x i be i th column of V Consider A T A x i : 3. So elements of S are sqrt(eigenvalues) and columns of V are eigenvectors of A T A What we wanted for robust least squares fitting! دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

SVD AND M ATRIX S IMILARITY 4. One common equivalent of matrix similarity in linear system of : Can be deduced using. This changes the linear system to : This means of a similarity transform for the system using SVD(A). دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

SVD AND M ATRIX N ORM 5. One common definition for the norm of a matrix is the Frobenius norm: Frobenius norm can be computed from SVD So changes to a matrix can be evaluated by looking at changes to singular values دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

SVD AND M ATRIX L IKLIHOOD 6. Suppose you want to find best rank- k approximation to A Answer: set all but the largest k singular values to zero Can form compact representation by eliminating columns of U and V corresponding to zeroed s i دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

SVD AND PCA 7. Principal Components Analysis (PCA): approximating a high-dimensional data set with a lower-dimensional subspace دوشنبه، 3 / 9 / Original axes * * * * * * * *** * * *** * * * * * * * * * Data points First principal component Second principal component دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

SVD AND PCA Data matrix with points as rows, take SVD Subtract out mean (“whitening”) Columns of V k are principal components Value of s i gives importance of each component دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

P HYSICAL INTERPRETATION Consider a correlation matrix, A Error ellipse with the major axis as the larger eigenvalue and the minor axis as the smaller eigenvalue دوشنبه، 3 / 9 / 90 40

P HYSICAL INTERPRETATION Orthogonal directions of greatest variance in data Projections along PC1 (Principal Component) discriminate the data most along any one axis دوشنبه، 3 / 9 / Original Variable A Original Variable B PC 1 PC 2

دوشنبه، 3 / 9 / 90 42

PCA ON F ACES : “E IGENFACES ” دوشنبه، 3 / 9 / Average face First principal component Other components For all except average, “gray” = 0, “white” > 0, “black” < 0 دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

I MAGE C OMPRESSION USING SVD The image is stored as a 256 X 264 matrix M with entries between 0 and 1 The matrix M has rank 256 Select r X 256 as an approximation to the original M As r in increased from 1 all the way to 256 the reconstruction of M would improve i.e. approximation error would reduce Advantage To send the matrix M, need to send 256 X 264 = numbers To send an r = 36 approximation of M, need to send * *264 = numbers 36 singular values 36 left vectors, each having 256 entries 36 right vectors, each having 264 entries دوشنبه، 3 / 9 / Courtesy:

U SING PCA FOR R ECOGNITION Store each person as coefficients of projection onto first few principal components Compute projections of target image, compare to database (“nearest neighbor classifier”) دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

T OTAL L EAST S QUARES 8. One final least squares application Fitting a line: vertical vs. perpendicular error دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

T OTAL L EAST S QUARES Distance from point to line: where n is normal vector to line, a is a constant Minimize: دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

T OTAL L EAST S QUARES First, let’s pretend we know n, solve for a Then دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

T OTAL L EAST S QUARES So, let’s define and minimize دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

T OTAL L EAST S QUARES Write as linear system Have An=0 Problem: lots of n are solutions, including n=0 Standard least squares will, in fact, return n=0 دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

C ONSTRAINED O PTIMIZATION 9. Solution: constrain n to be unit length So, try to minimize |An| 2 subject to |n| 2 =1 Expand in eigenvectors e i of A T A: where the i are eigenvalues of A T A دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

C ONSTRAINED O PTIMIZATION To minimize subject to set  min = 1, all other  i = 0 That is, n is eigenvector of A T A with the smallest corresponding eigenvalue دوشنبه، 3 / 9 / دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق

A PPLICATIONS OF SVD IN L INEAR A LGEBRA Homogeneous equations, Ax = 0 Minimum-norm solution is x=0 (trivial solution) Impose a constraint, “Constrained” optimization problem Special Case If rank(A)=n-1 (m ¸ n-1, n =0) then x=  v n (  is a constant) Genera Case If rank(A)=n-k (m ¸ n-k, n- k+1 =  = n =0) then x=  1 v n- k+1 +  +  k v n with   +  2 n =1 دوشنبه، 3 / 9 / For proof: Johnson and Wichern, “Applied Multivariate Statistical Analysis”, pg 79 Has appeared before Has appeared before  Homogeneous solution of a linear system of equations  Computation of Homogrpahy using DLT  Estimation of Fundamental matrix