1. Last lecture summary
Fundamental system in linear algebra : system of linear equations Ax = b. nice case – n equations, n unknowns
matrix notation
row picture
column picture
linear combinations

2. For our matrix, can I solve
Ax = b for every b?
Yes
And what it means geometrically?
RHS fills the whole 2D space.
But when this can go wrong?
If two columns are on the same line, then their combination is also on this line.
singular matrix, not invertible
invertible matrix, columns are independent

3. Matrix by matrix multiplication
row times column
shape of matrices?
m x n . n x p = m x p
column picture
AB = C
Columns of C are linear combinations of columns of A
row picture
Rows of C are linear combinations of rows of B
column times row
set of full size matrices

4. Inverse Rules I = AA-1 = A-1A
When is square matrix invertible (i.e. nonsingular)?
If it does not have dependent columns
If you can’t find non-zero x such that Ax = 0
Rules
CT = (AB)T = BTAT
(A-1)T = (AT)-1

5. Transpose What is it? What is symmetric matrix?
RTR is symmetric, RRT is also symmetric.

6. Vector space contains vectors – objects I can
add together
multiply by number
zero must belong to the space, otherwise it can’t be called space !
linear combination
- unusual vector space – all real matrices

7. Subspace Something smaller within the space.
This smaller bit is also space.
Subspaces in R2:
all of R2
all lines through zero
Is this line the same as R1?
zero vector

8. Column space subspaces come out of matrices column space C(A)
take columns of A
all their linear combinations
They all together form a space.
Original columns as well as zero vector are all covered by the term “all linear combinations”.

9. New stuff

10. Column space We’ll be interested in the size of the column space now.
What do you think, is that space the whole four dimensional space? Just use your feeling, if we start with three vectors and take their combinations, can we get the whole four dimensional space? NO
So somehow we get a smaller space, but how much smaller? That’s not immediate.
Let’s make me first the critical connection with linear equations.

11. Column space is a subspace of R what?
What’s in the column space of A?
The columns (vectors) and all their linear combinations.
R^4

12. And then I'm going to ask which right-hand sides are okay?
Two questions
Does Ax = b always have a solution for every b? I guess that's going to be a yes or no question.
And then I'm going to ask which right-hand sides are okay?
OK, let’s write Ax = b for our A.
What is the answer to the 1st question?
Apparently the answer for 1. is No. Why?
the combinations of the columns don't fill the whole four dimensional space
There's going to be some vectors b, a lot of vectors b, that are not combinations of these three columns, because the combinations of these columns are going to be just a little plane inside R4.

13. However, for some RHSs I can solve that Ax = b.
Which RHSs allow me to solve this? Which vectors b allow this system to be solved? This is the critical question.
Tell me one RHS I can solve this system for?
All zeros.
Tell me another RHS I can solve for?
[ ]
And another?
[ ]
So apparently I can solve Ax = b when b is a linear combination of the columns.
In other words?

14. I can solve Ax=b exactly when b is in the column space.
If b is not a combination of the columns, then there is no x. There's no way to solve Ax = b.
Now the question is are all the columns independent? Do they all contribute something new, or can I drop one of them and still have the same column space?

15. So, would you throw some column away without changing a column space?
Col3 = Col1 + 4
The column space of this matrix is a two dimensional subspace of R4.

16. Exercise Describe the column spaces for I … whole space R2
A … line, the equation Ax is solvable only if b is on that line
B … whole space R2

17. Null space That’s a completely different space.
What's in it? It contains not right-hand sides b. It contains x’s.
All solutions (x) to the equation Ax = 0.
So where is the null space for this example?
All x’s form a subspace of what?
of R3
- null space – lecture 6, 28:22

18. Column space is in R4, while null space is in different R (R3).
OK, we’ll try to find a null space N(A) of our matrix A. Help me, find at least three x’s.
[0 0 0] Now try to find another?
[1 4 -1] Now try to find yet another?
[2 8 -2]
In other words, c-multiples of [1 4 -1]
vector x has three components = R3

19. So geometrically, how would you describe a null space?
It’s a line. Line in R3, through the origin.
We have to show, that this line is a subspace.
Show, that if I have two solutions v and w, their sum v + w is also a solution. So if Av = 0 and Aw = 0, then A(v + w) = 0
This is actually one of the matrix laws (distributive law), that I can split A(v + w) into two pieces: Av + Aw = 0
40:15

20. And similarly I have to show that if Av = 0 then A times any multiple v is zero.
A(12.v) = 0 because 12 x Av = 12 x 0 = 0
Now, to fully understand the vector space, let’s change the RHS
What’s the solution?
[1 0 0]
Are there any other solutions?
[0 -4 1], etc.
Do the solutions form a subspace?
They do NOT, why?
zero vector is not a solution
What are the solutions geometrically?
They form a plane, not going through the origin.
the solutions do not form a subspace as the zero vector is not a solution

21. Independence, basis, dimension
lecture 5, second half, lecture 6
based on excelent video lectures by Gilbert Strang, MIT
Lecture 9

22. Independence When x1, x2, …, xn are independent?
No linear combination gives zero vector, except 0.
- 7:08 (lecture 9)

23. I have three nonzero vectors in 2D space (plane).
I can arrange such vectors in a 2 x 3 matrix A. Then, the vectors are dependent, if Ax = 0 (except x = 0), i.e. if there is something in the null space of A.
In othe words, columns are independent, if there is only 0 in its null space.
The number of independent columns in matrix A is called the rank (hodnost) of matrix A.

24. Span
When we had a columns in a matrix, we took all their combinations and that gave us the column space.
Those vectors that we started with span (definují) that column space.
So now I can say in shorthand the columns of a matrix span the column space.
- span the space = generovat prostor

25. Columns of matrix A span the column space.
Are the columns independent?
It depends on that particular columns.
But obviously we’re highly interested in a set of vectors that spans a space and is independent.
If we didn’t have them all, we wouldn’t have our whole space
If we had more, they wouldn’t be independent.
Such a bunch of vector is called a basis for a vector space.

26. Basis is a set of vectors they have two properties:
I’ve got enough vectors.
And not too many.
Well, mathematician way of saying the same:
they span the space
they are independent
From now, whenever I look at a subspace, if you give me a basis for that subspace, you've told me everything I need to know about that subspace.
Basis is
minimum spanning set
maximum independent set
max. indep. set – cannot be made larger without losing independence
min span. set – cannot be made smaller and still span the space

27. What would be a basis for a 3D space:
Examples – R3 space
What would be a basis for a 3D space:
[1,0,0]T, [0,1,0]T, [0,0,1]T, identity matrix, null space is 0
n vectors give a basis if the n x n matrix with those columns is invertible
why?
Matrix is invertible, if Ax = 0 only for x = 0
Identity matrix the vectors are independent (no zero vector for Ax)

28. the one they span, i.e. their combination what is their combination?
are these vectors independent?
yes
which space they span?
R3 (3D)
do they form a basis?
yes, for R3
Are these two vectors a
basis for any space?
yes, and for what space?
the one they span, i.e. their combination
what is their combination?
plane inside 3D space
next, remove one column

29. basis is not unique
there are zilions of bases
columns of any invertible 3x3 matrix form bases for 3D
however, all bases for a given space have the same number of vectors (3 for R3, n for Rn)
If we're talking about some other space, the column space of some matrix, or the null space of some matrix, or some other space that we haven't even thought of, then that still is true that there're lots of bases but every basis has the same number of vectors.

30. this number is called dimension (how big is the space?)
Let me repeat the four terms we’ve got now defined
Independence - looks at combinations not being zero
Spanning - looks at all the combinations
Basis - combines independence and spanning
Dimension - the number of vectors in any basis, because all bases have the same number.

31. Example do the columns span the column space of this matrix?
yes, by definition what the column space is
form they a basis for the column space?
no, they are not independent, there’s something
in the null space
Look at the null space N(A)
Tell me some vector in the nullspace (solution of Ax = 0)
[ ]T
Tell me the basis for that column space. There are many
answers, give me the most natural answer.
first two columns
And the rank of the matrix is?
two
Great theorem comes !!!!
rank is the number of independent columns
rank is the number of vectors in the basis
rank is WHAT?
dimension!

32. The rank of A is the dimension of the column space.
dim C(A) = r
About words
I am talking about the rank of the matrix
and I am talking about the dimension of the vector space/subspace
I am not talking about the dimension of a matrix, I am not talking about a rank of space
And there is a link between the rank of matrix and the dimension of its column space.

33. Null space dim N(A) = n – r
we already have one vector there: [ ]T
Are there other vectors in the null space?
Yes. So our only vector is not a basis, because it does not span.
Tell me one vector more
[ ]T
The vectors in the null space are telling me in what way the columns are dependent. That's what the null space is doing.
Now, what is the nullspace?
These are two vectors in the null space. They're independent. Are they a basis for the null space? What's the dimension of the null space?
they are independent, they form a basis, the null space is two-dimensional
dim N(A) = n – r
n is the number of columns

34. Four fundamental subspaces
lecture 5, second half, lecture 6
based on excelent video lectures by Gilbert Strang, MIT
Lecture 10

35. null space of AT ... N(AT) (called the left null space of A)
column space C(A)
null space N(A)
row space
all combinations of rows of A
rows span a row space
they are/they are not a basis for a row space
But I don’t like to work with row vectors, I’d like to stay with column vectors. How to get column vectors out of the rows?
transpose
So row space is all combinations of columns of AT - C(AT)
null space of AT ... N(AT) (called the left null space of A)

36. Where are these spaces? A is m x n
N(A) – vectors with n components (it is in Rn), solutions to Ax = 0
C(A) – columns of A, each column has m components, it is in Rm
C(AT) – the row space is in Rn
N(AT) – the left null space is in Rm
And now we want to understand these spaces, i.e. we’d like to know a basis for those spaces. And what’s their dimension?

37. Dimensions A = m x n dim C(A) = r dim C(AT) = r dim N(A) = n – r
dim N(AT) = m – r
dim C(A) + dim N(A) = n
dim C(AT) + dim N(AT) = m
The row space and the null space are in Rn. Their dimensions add to n.
The column space and the left null space are in Rm, and their dimensions add to m.

38. E. g. is this matrix singular?
Please, pay attention to the fact that dimension (i.e. rank) of the column space and row space is the same.
E. g. is this matrix singular?
Yes, it is, because two rows are the same. Thus rank both of column and row space is two.

39. Different type of vectors space
All our vector spaces have been subspaces of some real n dimensional space.
A new vector space – all 3 x 3 matrices.
i.e. my matrices are vectors.
They don’t look like vectors, they’re matrices, but they are vectors in my vector space because they obey the rules.
end of Lecture 10, Lecture 11

40. How about subspace of this matrix space M?
upper triangular matrices U
symmetric matrices
intersection of two subspaces is also a subspace
what is the intersection of U and symmetric?
diagonal – this is smaller subspace

41. Dimension = number of members in the basis 3x3 matrices
Now we will intuitively investigate a bases (and dimensions) of these three matrix spaces: all 3x3 matrices, 3x3 upper triangular, 3x3 symmetric and 3x3 diagonal matrices
Dimension = number of members in the basis
3x3 matrices
basis?
matrices, each with 1 at different positions and the rest zeros
dimension?
nine

42. upper triangular matrices
dimension?
six
symmetric
dimension is again six
diagonal
dimension is three

43. Orthogonality lecture 5, second half, lecture 6
based on excelent video lectures by Gilbert Strang, MIT
Lecture 14

44. Welcome to the world of orthoganility. This is a ninety degree lecture.
What it means for vectors to be orthogonal?
What it means for subspaces to be orthogonal?

45. Two vectors
orthogonal = perpendicular, the angle between two vectors is 90o
How to find whether two vectors are orthogonal? Use Pythagoras.
what is this vector in terms of a and b?
length of the vector

46. This leads to the inner product rule about orthogonality. Here is why:So
This leads to the inner product rule about orthogonality. Here is why:
aTa + bTb = (a+b)T(a+b) = aTa + bTb + aTb + bTa
 aTb + bTa = 0  2 aTb = 0  aTb = 0
Two vectors are orthogonal if aTb = bTa = 0
Zero vector is orthogonal to any vector.

47. Orthogonality of subspaces
Subspace S is orthogonal to subspace T. What it means?
It means that every vector in S is orthogonal to every vector in T.
Wall is one subspace in 3D, floor is another subspace in 3D.
Are they orthogonal? No
And why not?
if two subspaces meet at some vector, well then for sure they're not orthogonal, because that vector is in one and it's in the other, and it's not orthogonal to itself unless it's zero.

48. row space is orthogonal to the null space (in Rn) Why?
if x is in the null space, then Ax = 0
So I'm saying that a vector in the row space is perpendicular to the x in the null space.
row 1 is orthogonal to x, their inner product is zero

49. Similarly, column space is orthogonal to the left null space (in Rm)
OK, so far we have shown that rows of A are orthogonal to x, but what else is in the row space?
all their linear combinations
To show that the linear combinations of rows are also zero is pretty easy. Help me
row1. x = 0, so c . row1 . x = 0
Similarly, column space is orthogonal to the left null space (in Rm)

50. Next comes another definition, without proving.
Row space and null space are orthogonal complements (in Rn).
The orthogonal complement of a row space contains not just some vectors that are orthogonal to it, but all.
That means that the null space contains all, not just some but all, vectors that are perpendicular to the row space.
- 14. lecture, 35:03 a dale, navozeni least squares, str. 150

51. G. Strang, Introduction to linear algebra