1. CHAPTER FIVE Orthogonality Why orthogonal? Least square problem Accuracy of Numerical computation

2. Least square problem Motivation: Curve fitting Problem formulation: Given, Find such that

3. Outlines Results of Orthogonal subspace Inner product spaces Least square problems Orthonormal sets Gram-Schmidt Orthogonalization process

4. The scalar product in Def: Let,.the inner product (or scalar product) of and is defined to be The norm of is defined as

5. Theorem5.1.1 : Let. Then pf: By the law of cosines, Note: If is the angle between, then Thus Def:

6. Cor: (Cauchy-Schwartz Inequality) Let. Then Moreover, equality holds for some

7. Scalar and Vector Projections Let. Then the quantify is the scalar projection of onto and the vector is called the vector projection of onto If,then the scalar projection of onto is and the vector projection of onto is

8. Example: Find the point on the line that is closest to the point (1,4) Sol: Note that the vector is on the line Then the desired point is

9. Example: Find the equation of the plane passing through and normal to Sol:

10. Example: Find the distance form to the plane Sol: a normal vector to the plane is desired distance

11. Orthogonal Subspace Def: Let be two subspace of. We say that if, and. Example: Let but does not orthogonal to

12. Def: Let be a subspace of. Then the orthogonal complement of is defined as Example: In,

13. Lemma: (i) Two subspaces (ii) If is a subspace of,then is also a subspace of. Pf: (i) If (ii) Let and

14. Four Fundamental Subspaces Let for some It will be shown later that

15. Theorem5.1.2: Let. Then and pf: Let and _____(1) for some _____(2) ________(3) Also, if _____(4) Similarly,

16. Example: Let Clearly,

17. Theorem5.2.2: Let be a subspace of, then (i) (ii) If is a basis for and is a basis for,then is a basis for. pf: If The result follows Suppose. Let and

18. To show that is a basis for, It remains to show their independency. Let. Then Similarly, This completes the proof

19. Def: Let are two subspaces of. We say that is a direct sum of,denoted by, if each can be written uniquely as a sum, where Example: Let Then but

20. Theorem5.2.2: If is a subspace of, then pf: By Theorem5.2.2, To show uniqueness, Suppose where

21. Theorem5.2.4: If is a subspace of, then pf: Let If (Why?)

22. Remark: Let. i.e., Since and are bijections.

23. Let bijection

24. Cor5.2.5: Let and. Then either (i) or (ii) pf: or Note that

25. Example: Let. Find The basic idea is that the row space and the sol. of are invariant under row operation. Sol: (i) (Why?) (ii) (Why?) (iii) Similarly, and (iv) Clearly,

26. Example: Let (i) and (ii) The mapping is a bijection and (iv) What is the matrix representation for ?

27. Linear Product Space A tool to measure the orthogonality of two vectors in general vector space

28. Def: An inner product on a vector space is a function Satisfying the following conditions: (i) with equality iff (ii) (iii)

29. Example: (i) Let Then is an inner product of (ii) Let, Then is an inner product of (iii) Let and then is an inner product of (iv) Let, is a positive function and are distinct real numbers. Then is an inner product of

30. Def: Let be an inner product of a vector space and. we say The length or norm of is

31. Theorem5.3.1: (The Pythagorean Law) pf:

32. Example: Consider with inner product (i) (ii) (iii) (iv) (Pythagorean Law) or

33. Example: Consider with inner product It can be shown that (i) (ii) (iii) Thus are orthonormal set.

34. Example: Let and let Then not orthogonal to

35. Def: Let be two vectors in an inner product space. Then the scalar projection of onto is defined as The vector projection of onto is

36. Lemma: Let be the vector projection of onto. Then for some pf:

37. Theorem5.3.2: (Cauchy-Schwarz Inequality) Let be two vectors in an inner product space. Then Moreover, equality holds are linear dependent. pf: If If Equality holds i.e., equality holds iff are linear dependent.

38. Note: From Cauchy-Schwarz Inequality. This, we can define as the angle between the two vectors

39. Def: Let be a vector space A fun is said to be a norm if it satisfies with equality scalar

40. Theorem5.3.3: If is an inner product space, then is a norm on pf: trivial Def: The distance between is defined as

41. Example: Let. Then is a norm is a norm for any In particular, is the euclidean norm

42. Example: Let. Then

43. Example: Let Thus, However, (Why?)

44. Example: Let Then

45. Least square problem A typical example: Given Find the best line to fit the data. or or find such that is minimum Geometrical meaning :

46. Least square problem: Given then the equation may not have solutions The objective of least square problem is trying to find such that has minimum value i.e., find satisfying

47. Preview of the results: It will be shown that Moreover, If columns of are Linear independent.

48. Theorem5.4.1: H. Let be a subspace of C. (i) for all (ii) pf: where If Since the expression is unique, result (i) is then proved. (ii) follows directly from (i) by noting that

49. Question: How to find which solves Answer: Let From previous Theorem, we know that normal equation

50. Theorem5.4.2: Let and Then the normal equation. Has a unique sol. and is the unique least square sol. to pf: Clearly, is nonsingular (Why?) is the unique sol. To normal equation. is the unique sol. To the least square problem (Why?) ( has linear independent columns)

51. Note: The projection vector is the element of that is closet to in the least square sense. Thus, The matrix is called the Projection matrix (that project any vector of to )

52. Example: Suppose a spring obeys the Hook’s law and a series of data are taken (with measurement error) as How to determine ? sol: Note that is inconsistent The normal equation. is The least square sol.

53. Example: Given the data Find the best least square fit by a linear function. sol: Let the desired linear function be The problem be comes to find the least square sol. of Least square sol. The best linear least square fit is

54. Example: Find the best quadratic least square fit to the data sol: Let the desired quadratic function. be The problem becomes to find the least square sol. of least square sol. the best quadratic least square fit is

55. Orthonormal Set Simplify the least square sol. (avoid computing inverse) Numerical computational stability

56. Def: is said to be an orthogonal set in an inner product space if Moreover, if, then is said to be orthonormal

57. Example: is an orthogonal set but not orthonormal However, is orthonormal

58. Theorem5.5.1: Let be an orthogonal set of nonzero vectors in an inner product space. Then they are linear independent pf: Suppose is linear independent.

59. Example: is an orthonormal set of with inner product Note: Now you know the meaning what one says that

60. Theorem5.5.2: Let be an orthonormal basis for an inner product space. If then pf:

61. Cor: Let be an orthonormal basis for an inner product space. If and, then pf:

62. Cor: (Parseval’s Formula) If is an orthonormal basis for an innerproduct space and, then pf: direct from previous corollary

63. Example: and from an orthonormal basis for. If, then and

64. Example: Determine without computing antiderivatives. sol: and is an orthonormal set of

65. Def: is said to be an orthogonal matrix if the column vectors of form an orthonormal set in Example: The rotational matrix and the elementary reflection matrix are orthogonal matrix.

66. Properties of orthogonal matrix: at be orthogonal. Then The column vectors of form an orthonormal basis for Preserve inner product preserve norm preserve angle.

67. Note: Let the columns of form an orthonormal set of.Then and the least square sol to is This avoid computing matrix inverse.

68. Cor5.5.9: Let be a nonzero subspace of and is an orthonormal basis for. If then the projection of onto is pf:

69. Note: Let columns of be an orthonormal set The projection of onto is the sum of the projection of onto each.

70. Example: Let Find the vector in that is closet to Sol: Clearly is a basis for. Let Thus Hw: Try What is ?

71. Approximation of functions Example: Find the best least square approximation to on by a linear function. Sol: (i) Clearly, but is not orthonormal (ii) seek a function of the form By calculation is an orthonormal set of (iii) Thus the projection. is the best linear least square approximation to on

72. Approximation of trigonometric polynomials FACT: forms an orthonormal set in with respect to the inner product Problem: Given a periodic function, find a trigonometric polynomial of degree n which is a best least square approximation to.

73. Sol: It suffices to find the projection of onto the subspace The best approximation of has coefficients

74. Example: Consider with inner product of (i) Check are orthonormal (ii) Let Similarly (iii) (iv)

75. Cram-Schmidt Orthogonalization Process Question: Given a set of linear independent vectors, how to transform them into orthogonal ones while preserve spanning set.?

76. Given,Clearly Clearly Similarly, Clearly We have the next result

77. Theorem5.6.1: (The Cram-Schmidt process) H. (i) be a basis for an inner product space (ii) C. is an orthonormal basis

78. Example: Find an orthonormal basis for with inner product given by where Sol: Starting with a basis

79. QR-Decomposition Given Let _______(1) _________________(2) _______________________(3) Define Where has orthonormal columns and is upper-triangular

80. To solve with Then, the example can be solved By backsubstitution without finding Inverse (if is square)

81. Example: Solve By direct calculation, The solution can be obtained from