Lecture 12 Projection and Least Square Approximation Shang-Hua Teng

Line Fitting and Predication Input: Table of paired data values (x, y) –Some connection between x and y. –Example: height weight –Example: revenue stock price –Example: Yesterday’s temperature at Pittsburgh today’s temperature at Boston Output: a and b that best predicates y from x: y = ax + b

      Scatter Plot of Data Revenue Stock Price

        Regression Line y = ax+b Revenue Stock Price

        Predication with Regression Line y = ax+b Revenue Stock Price

When Life is Perfect y = ax+b Revenue Stock Price  x1x1 x7x7 y7y7 x2x2 y1y1 y3y3 x3x3 y2y2 How do we find a and b

When Life is Perfect y = ax+b Revenue Stock Price         x1x1 x7x7 y7y7 x2x2 y1y1 y3y3 x3x3 y2y2

How to Solve it? By Elimination What will happen?

Another Method: Try to Solve In general: if A x = b has a solution, then A T Ax = A T b has the same solution

        When Life is not Perfect No perfect Regression Line y = ax+b Revenue Stock Price

        When Life is not Perfect No perfect Regression Line y = ax+b Revenue Stock Price No solution!!!!!! What happen during elimination

Linear Algebra Magic In general: if A x = b has no solution, then A T Ax = A T b gives the best approximation

Least Squares No errors in x Errors in y Best Fit Find the line that minimize the norm of the y errors (sum of the squares)

        When Life is not Perfect Least Square Approximation Revenue Stock Price

In General: When Ax = b Does not Have Solution Residue error Least Square Approximation: Find the best

One Dimension

In General

Least Square Approximation In general: if A x = b has no solution, then Solving A T Ax = A T b produces the least square approximation

Polynomial Regression Minimize the residual between the data points and the curve -- least-squares regression Data Find values of a 0, a 1, a 2, … a m Linear Quadratic Cubic General Parabola

Polynomial Regression Residual Sum of squared residuals Linear Equations

Least Square Solution Normal Equations

Example x y x y

Example

Regression Equation y = x x x 3

Projection Projection onto an axis (a,b) x axis is a vector subspace

Projection onto an Arbitrary Line Passing through 0 (a,b)

Projection on to a Plane

Projection onto a Subspace Input: 1. Given a vector subspace V in R m 2.A vector b in R m … Desirable Output: –A vector in x in V that is closest to b –The projection x of b in V –A vector x in V such that (b-x) is orthogonal to V

How to Describe a Vector Subspace V in R m If dim(V) = n, then V has n basis vectors –a 1, a 2, …, a n –They are independent V = C(A) where A = [a 1, a 2, …, a n ]

Projection onto a Subspace Input: 1. Given n independent vectors a 1, a 2, …, a n in R m 2.A vector b in R m … Desirable Output: –A vector in x in C([a 1, a 2, …, a n ]) that is closest to b –The projection x of b in C([a 1, a 2, …, a n ]) –A vector x in V such that (b-x) is orthogonal to C([a 1, a 2, …, a n ])

Think about this Picture C(A T ) N(A) RnRn RmRm C(A) N(A T ) xnxn A x n = 0 xrxr b A x r = b A x= b dim r dim n- r dim m- r

Projection on to a Line b a  p

Projection Matrix: on to a Line b a  p What matrix P has the property p = Pb