Solving Linear Systems Solving linear systems Ax = b is one part of numerical linear algebra, and involves manipulating the rows of a matrix. Solving linear systems Ax = b is one part of numerical linear algebra, and involves manipulating the rows of a matrix. The second main part of numerical linear algebra is about transforming a matrix to leave its eigenvalues unchanged. The second main part of numerical linear algebra is about transforming a matrix to leave its eigenvalues unchanged. Ax = x where is an eigenvalue of A and non-zero x is the corresponding eigenvector. The symbol is the Greek symbol for Lambda The symbol is the Greek symbol for Lambda

What are Eigenvalues? Eigenvalues are important in physical, biological, and financial problems (and others) that can be represented by ordinary differential equations. Eigenvalues are important in physical, biological, and financial problems (and others) that can be represented by ordinary differential equations. Eigenvalues often represent things like natural frequencies of vibration, energy levels in quantum mechanical systems, stability parameters. Eigenvalues often represent things like natural frequencies of vibration, energy levels in quantum mechanical systems, stability parameters.

What are Eigenvectors Mathematically speaking, the eigenvectors of matrix A are those vectors that when multiplied by A are parallel to themselves. Mathematically speaking, the eigenvectors of matrix A are those vectors that when multiplied by A are parallel to themselves. Finding the eigenvalues and eigenvectors is equivalent to transforming the underlying system of equations into a special set of coordinate axes in which the matrix is diagonal. Finding the eigenvalues and eigenvectors is equivalent to transforming the underlying system of equations into a special set of coordinate axes in which the matrix is diagonal. The eigenvalues are the entries of the diagonal matrix. The eigenvalues are the entries of the diagonal matrix. The eigenvectors are the new set of coordinate axes. The eigenvectors are the new set of coordinate axes.

Determinants Suppose we if eliminate the components of x from Ax=0 using Gaussian Elimination without pivoting. Suppose we if eliminate the components of x from Ax=0 using Gaussian Elimination without pivoting. We do the kth forward elimination step by subtracting a jk times row k from a kk times row j, for j=k,k+1,…n. We do the kth forward elimination step by subtracting a jk times row k from a kk times row j, for j=k,k+1,…n. Then at the end of forward elimination we have Ux=0, and U nn is the determinant of A, det(A). Then at the end of forward elimination we have Ux=0, and U nn is the determinant of A, det(A). For nonzero solutions to Ax=0 we must have det(A) = 0. For nonzero solutions to Ax=0 we must have det(A) = 0. Determinants are defined only for square matrices. Determinants are defined only for square matrices.

Determinant Example Suppose we have a 3x3 matrix. Suppose we have a 3x3 matrix. So Ax=0 is the same as: So Ax=0 is the same as: a 11 x 1 +a 12 x 2 +a 13 x 3 = 0 a 21 x 1 +a 22 x 2 +a 23 x 3 = 0 a 31 x 1 +a 32 x 2 +a 33 x 3 = 0

Determinant Example (continued) Step k=1: Step k=1: –subtract a 21 times equation 1 from a 11 times equation 2. –subtract a 31 times equation 1 from a 11 times equation 3. So we have: So we have: (a 11 a 22 -a 21 a 12 )x 2 +(a 11 a 23 -a 21 a 13 )x 3 = 0 (a 11 a 32 -a 31 a 12 )x 2 +(a 11 a 33 -a 31 a 13 )x 3 = 0

Determinant Example (continued) Step k=2: Step k=2: –subtract (a 11 a 32 -a 31 a 12 ) times equation 2 from (a 11 a 22 - a 21 a 12 ) times equation 3. So we have: So we have: [(a 11 a 22 -a 21 a 12 )(a 11 a 33 -a 31 a 13 )- (a 11 a 32 -a 31 a 12 )(a 11 a 23 -a 21 a 13 )]x 3 = 0 which becomes: [a 11 (a 22 a 33 –a 23 a 32 ) – a 12 (a 21 a 33 -a 23 a 31 ) + a 13 (a 21 a 32 -a 22 a 31 )]x 3 = 0 and so: det(A) = a 11 (a 22 a 33 –a 23 a 32 ) – a 12 (a 21 a 33 -a 23 a 31 ) + a 13 (a 21 a 32 -a 22 a 31)

Definitions Minor M ij of matrix A is the determinant of the matrix obtained by removing row i and column j from A. Minor M ij of matrix A is the determinant of the matrix obtained by removing row i and column j from A. Cofactor C ij = (-1) i+j M ij Cofactor C ij = (-1) i+j M ij If A is a 1x1 matrix then det(A)=a 11. If A is a 1x1 matrix then det(A)=a 11. In general, In general, where i can be any value i=1,…n.

A Few Important Properties det(AB) = det(A)det(B) det(AB) = det(A)det(B) If T is a triangular matrix, If T is a triangular matrix, det(T) = t 11 t 22 …t nn det(A T ) = det(A) det(A T ) = det(A) If A is singular then det(A)=0. If A is singular then det(A)=0. If A is invertible then det(A)=0. If A is invertible then det(A)=0. If the pivots from Gaussian Elimination are d 1, d 2,…,d n then If the pivots from Gaussian Elimination are d 1, d 2,…,d n then det(A) = d 1 d 2= d n where the plus or minus sign depends on whether the number of row exchanges is even or odd.

Characteristic Equation Ax = x can be written as Ax = x can be written as (A- I)x = 0 which holds for x=0, so (A- I) is singular and det(A- I) = 0 This is called the characteristic polynomial. If A is nxn the polynomial is of degree n and so A has n eigenvalues, counting multiplicities. This is called the characteristic polynomial. If A is nxn the polynomial is of degree n and so A has n eigenvalues, counting multiplicities.

Example Hence the two eigenvalues are 1 and 5.

Example (continued) Once we have the eigenvalues, the eigenvectors can be obtained by substituting back into Once we have the eigenvalues, the eigenvectors can be obtained by substituting back into (A- I)x = 0. This gives eigenvectors (1 -1) T and (1 1/3) T This gives eigenvectors (1 -1) T and (1 1/3) T Note that we can scale the eigenvectors any way we want. Note that we can scale the eigenvectors any way we want. Determinant are not used for finding the eigenvalues of large matrices. Determinant are not used for finding the eigenvalues of large matrices.

Positive Definite Matrices A complex matrix A is positive definite if for every nonzero complex vector x the quadratic form x H Ax is real and: A complex matrix A is positive definite if for every nonzero complex vector x the quadratic form x H Ax is real and: x H Ax > 0 x H Ax > 0 where x H denotes the conjugate transpose of x (i.e., change the sign of the imaginary part of each component of x and then transpose).

Eigenvalues of Positive Definite Matrices If A is positive definite and and x are an eigenvalue/eigenvector pair, then: If A is positive definite and and x are an eigenvalue/eigenvector pair, then: Ax = x  x H Ax = x H x Since x H Ax and x H x are both real and positive it follows that is real and positive. Since x H Ax and x H x are both real and positive it follows that is real and positive.

Properties of Positive Definite Matrices If A is a positive definite matrix then: If A is a positive definite matrix then: –A is nonsingular. –The inverse of A is positive definite. –Gaussian elimination can be performed on A without pivoting. –The eigenvalues of A are positive.