Download presentation

Presentation is loading. Please wait.

Published byMerry Alexander Modified over 4 years ago

1
Scientific Computing Linear Least Squares

2
Interpolation vs Approximation Recall: Given a set of (x,y) data points, Interpolation is the process of finding a function (usually a polynomial) that passes through these data points.

3
Interpolation vs Approximation Given a set of (x,y) data points, Approximation is the process of finding a function (usually a line or a polynomial) that comes the “closest” to the data points. Data has “noise” – cannot find interpolating line.

4
General Least Squares Idea Given data and a class of functions F, the goal is to find the “best” function f in F that approximates the data. Consider the data to be two vectors of length n + 1. That is, x = [x 0 x 1 … x n ] t and y = [y 0 y 1 … y n ] t

5
General Least Squares Idea Definition: The error, or residual, of a given function with respect to this data is the vector r = y - f(x): That is r = [r 0 r 1 … r n ] t where r i = y i - f(x i ) We want to find a function f such that the error is made as small as possible. How do we measure the size of the error? With vector norms.

6
Vector Norms A vector norm is a quantity that measures how large a vector is (the magnitude of the vector). Our previous examples for vectors in R n : Manhattan Euclidean Chebyshev For Least Squares, the best norm to use will be the 2- norm.

7
Ordinary Least Squares Definition The least-squares best approximating function to a set of data, x, y from a class of functions, F, is the function f * in F that minimizes the 2-norm of the error. That is, if f * is the least squares best approximating function, then This is often called the ordinary least squares method of approximating data.

8
Linear Least Squares Linear Least Squares: We assume that the class of functions is the class of all possible lines. By the method in the last slide, we then want to find a linear function f(x) = ax + b that minimizes the 2- norm of the vector y-f(x), i.e. that minimizes:

9
Linear Least Squares Linear Least Squares: To minimize it is enough to minimize the term inside the square root. So, if f(x) = ax+b, we need to minimize over all possible values of a and b. From calculus, we know that the minimum will occur where the partial derivatives with respect to a and b are zero.

10
Linear Least Squares Linear Least Squares: These equations can be written as: These last two equations are called the Normal Equations for the best line fit to the data.

11
Linear Least Squares Linear Least Squares: Note that these are two equations in the unknowns a and b. Let Then, the solution is

12
Matrix Formulation of Linear Least Squares Linear Least Squares: Want to minimize Let Then, we want to find a vector c that minimizes the length squared of the error vector Ac-y (or y – Ac) That is, we want to minimize This is equivalent to minimizing the Euclidean distance from y to Ac.

13
Matrix Formulation of Linear Least Squares Linear Least Squares: Find a vector c to minimize the Euclidian distance from y to Ac. Equivalently, minimize ||y–Ac|| 2 or (y–Ac) t (y–Ac) If we take all partials of this expression (with respect to c 0, c 1 ) and set these equal to zero, we get

14
Matrix Formulation of Linear Least Squares The equation A t Ac = A t y is also called the Normal Equation for the linear best fit. The equations one gets from A t Ac = A t y are exactly the same equations we had before: The solution c=[a b] for A t Ac = A t y gives the constants for the line ax+b.

15
Matlab Example % Example of how to find the linear least squares fit to noisy data x = 1:.1:6; % x values y =.1*x + 1; % linear y-values ynoisy = y +.1*randn(size(x)); % add noise to y values plot(x, ynoisy,'.') % Plot the noisy data hold on % So we can plot more data later % Find d11, d12, d21, d22, e1, e2 D=[sum(x.^2), sum(x); sum(x), length(x)]; e1 = x*ynoisy'; e2 = sum(ynoisy); % Solve for a and b det = D(1,1)*D(2,2) - D(1,2)*D(2,1); a = (D(2,2)*e1 - D(1,2)*e2)/det; b = (D(1,1)*e2 - D(2,1)*e1)/det; % Create a vector of y-values for the linear best fit fit_y = a.*x + b; plot(x, fit_y,'-') % Plot the best fit line

16
Matlab Example A t Ac = A t y % Example of how to find the linear least squares fit to noisy data x = 1:.1:6; % x values y =.1*x + 1; % linear y-values ynoisy = y +.1*randn(size(x)); % add noise to y values plot(x, ynoisy,'.') % Plot the noisy data hold on % So we can more data later % Create matrix A A = zeros(length(x), 2); A(:,1) = x; A(:,2) = ones(length(x),1); D = A'*A; e = A'*ynoisy'; % Solve for constants a and b c= D \ e; fit_y = c(1).*x + c(2); plot(x, fit_y,'O')

Similar presentations

© 2020 SlidePlayer.com Inc.

All rights reserved.

To make this website work, we log user data and share it with processors. To use this website, you must agree to our Privacy Policy, including cookie policy.

Ads by Google