1. Orthogonal matrices
based on excelent video lectures by Gilbert Strang, MIT
Lecture 17

2. Orthogonality we’ve met so far today we’ll meet orthogonal vectors
When are two vector orthogonal?
orthogonal subspaces
When are two subspaces orthogonal? Do you remember some examples?
today we’ll meet
orthogonal basis q1, …, qn
orthogonal matrix Q
- vectors dot product a.b = aTb = 0
- orth subspaces: C(A) and N(A^T), C(A^T) and N(A)

3. Orthonormal vectors they are orthogonal, i.e. qiTqj = 0 if i ≠ j
the length of each of them is one
I can write this like condition:
qiTqj = 0 if i ≠ j
qiTqj = 1 if i = j
for general vector a, how do I get the same vector of unity length?
I divide every of its component by it’s norm |a| = (aTa)-1/2
dot product a.b = a^Tb = |a||b|cos(\phi), where || is the norm of the vector
if vectots have norm of one (we know how to achieve this), then a.b = cos(\phi)
i.e. if a and b are the same (i.e. phi=0), cos (\phi) = 1, if they are in oposite directions (phi=180), cos(phi) = -1
so a.b with unit vectors can be used as a measure of their similarity

4. orthonormal vectors are independent
i.e. there is no linear combination (except 0) giving 0
having orthonormal basis is nice, it makes all the calculations better
and I can put orthonormal vectors q1 … qn into matrix Q (when I use Q, I always mean orthonormal columns!!!)
Generally, Q is rectangular matrix, we can have e.g. just two columns with many rows.
Such a matrix Q is called matrix with orthonormal columns.
The name orthogonal matrix (should better be orthonormal matrix, but this is used rarely) is used when Q is square.
- basis … vectors are independent and they span the space

5. This can be done by Gram-Schmidt procedure: A → Q
However, suppose my columns of A are not orthogonal. How do I make them orthonormal?
This can be done by Gram-Schmidt procedure: A → Q
In the last lecture we’ve been talking about ATA, it’s natural to look at QTQ.
What is QTQ?
QTQ = I
QT = matrix with q1T…qnT in rows, Q = matrix with q1…qn in columns, row times column is a number (0, if i=j it is a number 1)
therefore QTQ = I

6. Its inverse is its transpose … Q-1 = QT Example:
If Q is square and it holds, that QTQ = I, what can you tell me about its inverse?
Its inverse is its transpose … Q-1 = QT
Example:
X1 is not orthogonal, it has not normal columns, they must be normalized
A-1A = I, QTQ=I, thus Q-1=QT
is Q1 orthogonal?
does it have orthogonal columns? Yes, it does
are the columns of the unit length? No, they’re not, I must divide each of them by its length, i.e. by sqrt(2). So I must multiply the whole matrix by sqrt(2).

7. What’s the good of having Q? What formulas become easier?
I want to project to Q’s column space.
What’s the projection matrix P = A(ATA)-1AT?
P = QQT
So all the messy equations from the last lecture become nice and cute when I have matrix Q.
What do I mean by that?
to compute ATA I must compute a lot of inner products
But if I have Q instead of A, what happens?
- QTQ = I, thus Q(QTQ)-1QT = QI-1QT = QQT ^
so i-th component of x is qiT times b

8. If I have just matrix A with independent columns, I use Gram-Schmidt orthonormalization procedure to calculate Q.
The column space of A and Q are the same.
And the connection between A and Q is a matrix R. And it can be shown, that R is upper triangular.
A = QR

9. Determinants based on excelent video lectures by Gilbert Strang, MIT
Lecture 18

10. determinant is a test for invertibility
det A = |A| … determinant of a matrix is a number associated with every square matrix A
determinant is a test for invertibility
matrix is invertible if |A| ≠ 0
matrix is singular if |A| = 0

11. In our quest for determinants, we start with three key properties.

12. Property 1 Property 2 Property 3 det I = 1
Exchange rows of a matrix: reverse sign of
det
Property 3
The determinant is a linear function of each
row separately.
multiply row #1 by number t
add row #1 of A to row #1 of A’

13. det of triangular matrix (except for the sign)
U matrix is a result of Gauss elimination. So now we know, how any reasonable software (Matlab, e.g.) calculates det. Do the elimination, get U and multiply the diagonal elements.
det (A) = det (AT)
because of this property, all properties about rows (exchanging, all rows zero) are also valid about columns
upper triangular matrix

14. There exist an ugly formula (Leibniz formula) involving the sum of the terms (products of certain elements of A), determining the sign of the products involves permutations of numbers (i.e. combinatorics).
For those (not many, I guess) interested, I refer to

15. Eigenvalues and eigenvectors
based on excelent video lectures by Gilbert Strang, MIT
Lecture 21, 22

16. Introduction
Matrices A are square and we’re looking for some special numbers, eigenvalues, and some special vectors, eigenvectors.
Matrix A acts on vector x, the outcome is vector Ax.
And the vectors I'm specially interested in are the ones the come out in the same direction that they went in.
That won’t be typical. Most Ax vectors point into different direction.
But some vectors Ax will be parallel to x. These are called eigenvectors.
By parallel I mean
λ is called eigenvalue, and there is not just one eigenvalue for the given A.
- Ax is some multiple of x, the multiple is always called lambda

17. Example: eigenvalues and eigevectors of the projection matrix PPb x b x
is the vector b eigenvector?
no, it isn’t, its projection Pb is in another direction
propose some eigenvector x
any vector x in the plane is unchaged by P (Px = x)
and this is telling me, that λ = 1
Are there any other eigenvectors? I expect the
answer to be yes, I am in 3D and I hope for getting
three independent eigenvectors, two of them are in
the plane, but where is the third?
perpendicular to the plane, Px = 0, λ = 0
- of course, the plane is subspace, so it goes through zero
λ = 0, 1

18. I will jump ahead a bit and will tell you some nice things about eigenvalues.
n x n matrix has n eigenvalues
The sum of eigenvalues is equal to the sum of diagonal elements (trace).
The product of eigenvalues is determinant.
Eigenvalues can be complex numbers, but eigenvalues of symmetric matrices are always real numbers.
And let me introduce you antisymmetric matrices. These are matrices for which AT = -A.
And they have all eigenvalues complex .

19. What else can go wrong?
n x n matrix has n eigenvalues. But they don’t have to be different (algebraic multiplicity of the eigenvalue, two eigenvalues of the same value  2).
number of eigenvalues with the same value (for a given value), multiplicity of the corresponding root
If some of them are equal, then the eigenvectors may (but don’t have to!) also be equal (geometric multiplicity of the eigenvalue, two same eigenvectors  1). Such a matrix is called degenerate, and again it’s a bad thing.
number of linearly independent eigenvectors with that eigenvalue

20. Now the question: how do I find eigenvalues and eigenvectors?
How to solve Ax = λx, where I have two unknowns (λ and x) in one equation.
rewrite as (A - λI)x = 0
I don’t know λ, x, but I do know something here. I know, that matrix (A - λI) (i.e. matrix A shifted on the diagonal by some λ) must be singular. Otherwise the only solution would be just x = 0, not particularly interesting eigenvector.
And what I know about singular matrices?
Their determinant is zero!

21. det(A - λI) = 0
This is called characteristic (or eigenvalue) equation. And I got rid of x in the equation!
Now I am in the position to find λ first (I find n eigenvalues, they don’t necessarily differ), and then to find eigenvectors.
Eigenvectors are found by Gauss elimination, as the solution of (A - λI)x = 0.

22. Example Find eigen for matrix Form A – λI. How will it look like?
What is det(A – λI)?
characteristic polynomial

23. Find the roots of the polynomial, it must be equal to zero (determinant must be equal to zero).
What are the roots of λ2 – λ – 2?
Isn’t it (λ + 1)(λ - 2)? So λ1 = -1, λ2 = 2

24. For each λ find eigenvector, generally by Gauss elimination, as the solution of (A - λI)x = 0.
However, we can guess in our case. What is A – λI for λ1 = -1?
and solve (A – λI)x = 0
so what is vector x1 = [bž, vž]T?
x1 = [1, 1]T (eigenvector corresponding to λ1 = -1)
and similarly we proceed for λ2 = 2

25. If I have Ax = λx and Bx = αx, what about eigenvalues of A + B?
(A + B)x = (λ + α)x. Right?
WRONG!
What’s wrong?
B will, generally, have different eigenvectors than A. A and B are different matrices.
Or A · B. Also ≠ λ·α
So that was like caution. In these cases, the igenvalue problem must be solved for A+B (A·B).

26. What to do with them?
We’ve eigenvalues/eigenvectors, but what do we do with them?
A small terminological diversion
Square matrix A is called diagonalizable if there exists an invertible matrix P such that P −1AP is a diagonal matrix.
Diagonalization is the process of finding a corresponding diagonal matrix for a diagonalizable matrix.
Eigendecomposition is used for diagonalization.

27. Construct matrix S, it will have eigenvectors as columns.
Start with AS multiplication.
But I can factor out eigenvalues from this matrix. How?
AS = SΛ

28. And now I can multiply by S-1, but only if S is invertible, i. e
And now I can multiply by S-1, but only if S is invertible, i.e. if n eigenvectors are independent.
S-1AS = Λ A = SΛS-1
There is a small number of matrices that don’t have independent eigenvectors. Such matrices ARE NOT diagonalizable (deffective matrix).
But if eigenvalues are distinct, their eigenvectors are independent and EVERY SUCH a matrix is diagonalizable.

29. Which matrices are diagonalizable (i.e. eigendecomposed)?
all λ’s are different (λ can be zero, the matrix is still diagonalizable. But it is singular.)
if some λ’s are same, I must be careful, I must check the eigenvectors.
If I have repeated eigenvalues, I may or may not have n independent eigenvectors.
If the eigenvectors are independent, than it is diagonalizable (even if I have some repeating eigenvalues) … geometric multiplicity = algebraic
Diagonalizability is concerned with eigenvectors. (independent)
Invertibility is concerned with eigenvalues. (λ=0)

30. Identity matrix I (n x n):
Example of matrix with equivalent eigenvalues, but with different eigenvectors
Identity matrix I (n x n):
eigenvalues?
all ones (n ones)
but there is no shortage in eigenvectors
its got plenty of eigenvectors, I choose n independent

31. symmetric matrix (n x n)
diagonal matrix
eigenvalues are on the diagonal
eigenvectors are columns with 0 and 1
triangular matrix
eigenvalues are sitting along the diagonal
symmetric matrix (n x n)
real eigenvalues, real eigenvectors
n independent eigenvectors
they can be chosen such that they are orthonormal (A = QΛQT)
orthogonal matrix
eigenvalues are +1 or -1

32. Some facts about eigen What does it mean, if we have λ = 0?
Ax = λx = 0
So what follows from this?
A has linearly dependent rows/columns.
if A has λ = 0, then A is singular
A has λi, then A-1 has λi-1, eigenvectors of A and A-1 are the same
The sum of eigenvalues is equal to the sum of diagonal elements (trace).
The product of eigenvalues is determinant.

33. By some unfortunate accident a new species of frog has been introduced into an area where it has too few natural predators. In an attempt to restore the ecological balance, a team of scientists is considering introducing a species of bird which feeds on this frog. Experimental data suggests that the population of frogs and birds from one year to the next can be modeled by linear relationships. Specifically, it has been found that if the quantities Fk and Bk represent the populations of the frogs and birds in the kth year, then
The question is this: in the long run, will the introduction of the birds reduce or eliminate the frog population growth?
- priklad na mocniny matice: Applied_Linear_Algebra_And_Matrix_Analysis_-_Thomas_S._Shores.pdf, strana 232

34. So this system evolves in time according to x(k+1) = Ax(k) (k = 0, 1, …). Such a system is called discrete linear dynamical system, matrix A is called transition matrix.
So if we need to know the state of the system in time k = 50, we have to compute x(50) = A50 x(0). 50th power of matrix A, that’s a lot matrix multiplication.
- solution

35. However, it can be easily proved, tha if matrix A is diagonalizable (i
However, it can be easily proved, tha if matrix A is diagonalizable (i.e. A = PDP −1, where D is a diagonal matrix), then
Ak = PDkP-1
and Dk is easy peasy.
That’s one practical application of matrix diagonalization – matrix powers.
And in the last lecture we have shown, that solving eigenproblem and diagonalization is equivalent!
A = SΛS-1
Whenever you read “matrix was diagonalized”, it means its eigenvalues/vectors were found.
- LAMBDA – diagonal matrix with eigenvalues, S – matrix with eigenvectors

36. Eigendecomposition demonstration

37. Symmetric matrices
eigenproblem in symmetric matrices is one big chapter in applied numerical linear algebra
there are whole books just about symmetric problem
based on excelent video lectures by Gilbert Strang, MIT
Lecture 25
Lecture 27

38. Eigenvalues/eigenvectors
eigenvalues are real
eigenvectors are perpendicular (orthogonal)
what I mean by this?
have a look at identity matrix I, it certainly is symmetric, what are its eigenvalues?
ones
and eigenvectors?
every vector is an eigenvector
so I may choose these that are orthogonal
If eigenvalues of symmetric matrix are different, its eigenvectors are orthogonal. If some eigenvalues are equal, orthogonal vectors may be chosen.

39. symmetric case: A = QΛQT
usual case: A = SΛS-1
symmetric case: A = QΛQT
so symmetric matrix can be factored into this form: orthogonal times diagonal times transpose of orthogonal
this factorization completely displays eigenvalues, eigenvectors, and the symmetry (QΛQT is clearly symmetric)
This is called spectral theorem.
Spectrum is the set of eigenvalues of matrix.
- Spectral theorem. There is an orthonormal basis of V consisting of eigenvectors of A. Each eigenvalue is real.

40. q1q1T – what kind of matrix is this matrix?
Every symmetric matrix breaks up into these pieces with real λ and orthonormal eigenvectors.
q1q1T – what kind of matrix is this matrix?
OK, I don’t expect you remember what we were talking about two weeks later.
It was about matrices of this form:
In our case qTq is what?
one
And q1q1T is a projection matrix !
- this and the slide can be skipped

41. That's another way that people like to think of the spectral theorem.
Every symmetric matrix is a combination of perpendicular projection matrices.
That's another way that people like to think of the spectral theorem.
I’d like to avoid computation, that can be a lot of work.
Lectur 25, 30 min, why eigenvalues are not computed by characteristic equation?

42. What’s the sign of the determinant of such matrices?
For symmetric matrices eigenvalues are real. Question: are they positive or negative?
Symmetric matrices with all the eigenvalues positive are called positive definite.
Can I answer this question without actually computing the whole spectra?
What’s the sign of the determinant of such matrices?
The determinant is positive.
But that’s not enough!
det is a product of eigenvalues
this matrix has eigenvalue -1 and -3 (it is not positive definite), but det is still positive

43. So I have to check all subdeterminants, they must all be positive.
Matrices with eigenvalues equal or higher than zero are called positive semidefinite.
- det = ac – bb, first det (a) must be positive

44. So we have three tests of positive semidefinitness:
Definition: positive definite matrices have xTAx > 0 for every x (where x is any non-zero real vector).
So we have three tests of positive semidefinitness:
all eigenvalues positive
determinant and all subdeterminants positive
xTAx > 0 for every nonzero x
- pure quadratic form – there are no linear terms

45. Positive definite matrices are very important in applications.
Symmetric matrices are good, but positive semidefinite matrices are the best !
Positive definite matrices are very important in applications.
Positive definite matrices can be factorized using Cholesky decomposition M = LLT = RTR (L – lower triangular, R – upper).
So we need to store just one factor (L or R).
This factorization takes into account the symmetry of M.
In least squares ATAx = ATb, ATA is always symmetric positive definite provided that all columns of A are independent.