Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 1 Part 4 Chapter 15 General Least Squares and Non- Linear Regression

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Part Organization Chapter 14 –Brief Review of Statistics –Linear Regression (How to determine the best fit) –Linearization of Nonlinear Equations Chapter 15 –Polynomial Regression –Multiple Linear Regression Chapter 16 – Skip Chapter 17 – Interpolating Chapter 18 – Spline Interpolation 2

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 3 Chapter 15 – General Least Squares Some engineering data is poorly represented by a straight line. For these cases a curve is better suited to fit the data. –In chapter 14 we looked at techniques to linearize other models –This approach allowed us to use linear regression –An alternate approach is to use polynomial regression

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 4 Taylor’s Theorem Recall that any smooth function can be approximated by a polynomial Polynomial regression fits a polynomial to a set of data points

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 5 General Linear Least Squares Minimized by taking its partial derivative w.r.t. each of the coefficients and setting the resulting equation equal to zero

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. For Simple Linear Regression 6

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 7 Least-Squares Fit of a Straight Line To minimize Sr, we need to find the partial derivatives with respect to a0 and a1 and set them equal to 0

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Coefficients for a Simple Linear Regression 8

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. For 2 nd Order Polynomial Regression 9

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Example Fit this data to a second order polynomial

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. As before we need to minimize the sum of the squares of the residuals 11 Take the derivatives with respect to the coefficients (a) and set them equal to 0

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 3 equations and 3 unknowns 12

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. This is starting to get cumbersome We could solve for the coefficients using matrix algebra 13

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 14

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 15

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. This is starting to get cumbersome We could solve for the coefficients using matrix algebra…. But… we could also use the built-in MATLAB function polyfit 16

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 17

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 18 polyfit has the advantage that you can use it for higher order polynomials

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Or use left division 19

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 20 General Linear Least Squares Minimized by taking its partial derivative w.r.t. each of the coefficients and setting the resulting equation equal to zero

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. For 2 nd Order Polynomial Regression 21

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display a 0 * 1 + a 1 * 0 + a 2 * 0 a 0 * 1 + a 1 * 1 + a 2 * 1 a 0 * 1 + a 1 * 2 + a 2 * 4 a 0 * 1 + a 1 * 3 + a 2 * 9 a 0 * 1 + a 1 * 4 + a 2 * 16 a 0 * 1 + a 1 * 5 + a 2 * 25 =

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display a 0 * 1 + a 1 * 0 + a 2 * 0 a 0 * 1 + a 1 * 1 + a 2 * 1 a 0 * 1 + a 1 * 2 + a 2 * 4 a 0 * 1 + a 1 * 3 + a 2 * 9 a 0 * 1 + a 1 * 4 + a 2 * 16 a 0 * 1 + a 1 * 5 + a 2 * 25 = If you have an equal number of equations and unknowns the \ operator uses a modified Gaussian elimination strategy

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display a 0 * 1 + a 1 * 0 + a 2 * 0 a 0 * 1 + a 1 * 1 + a 2 * 1 a 0 * 1 + a 1 * 2 + a 2 * 4 a 0 * 1 + a 1 * 3 + a 2 * 9 a 0 * 1 + a 1 * 4 + a 2 * 16 a 0 * 1 + a 1 * 5 + a 2 * 25 = If you have more equations than unknowns the system is over-specified, and the \ operator utilizes QR factorization to find the best fit

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 25 The General Linear Least Squares Technique can be applied to more than just polynomials Minimized by taking its partial derivative w.r.t. each of the coefficients and setting the resulting equation equal to zero

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. What if you want to do a multiple linear regression? 26 Use a similar approach as that outlined in Section 15.2

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 27 This is pretty complicated, and it’s only good for 2 variables plus a constant!

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 28 This does the same thing, generalized for m dimensions

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Or… use left division 29

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Or… use left division a 0 * 1 + a 1 * 0 + a 2 * 0 a 0 * 1 + a 1 * 2 + a 2 * 1 a 0 * 1 + a 1 * a 2 * 2 a 0 * 1 + a 1 * 1 + a 2 * 3 a 0 * 1 + a 1 * 4 + a 2 * 6 a 0 * 1 + a 1 * 7 + a 2 * 2 = QR Factorization

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Using the Interactive Curve Fitting Tools MATLAB 7 includes interactive plotting tools. They include –basic curve fitting, –more complicated curve fitting –statistical tools

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Use the curve fitting tools… Create a graph Making sure that the figure window is the active window select –Tools-> Basic Fitting –The basic fitting window will open on top of the plot

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.

Plot generated using the Basic Fitting Window

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Residuals are the difference between the actual and calculated data points

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Basic Fitting Window

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Data Statistics Window You can also access the data statistics window from the figure menu bar. Select Tools->Data Statistics from the figure window. This window allows you to calculate statistical functions interactively, such as mean and standard deviation, based on the data in the figure, and allows you to save the results to the workspace.

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 38

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Save to your Current Directory 39

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Reload a figure by double clicking it’s name in the current directory 40

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Or in the command window 41