1. CISE301_Topic41 CISE301: Numerical Methods Topic 4: Least Squares Curve Fitting Lectures 18-19: KFUPM Read Chapter 17 of the textbook

2. CISE301_Topic42 Lecture 18 Introduction to Least Squares

3. CISE301_Topic43 Motivation Given a set of experimental data: x123 y5.15.96.3 The relationship between x and y may not be clear. Find a function f(x) that best fit the data 1 2 3

4. CISE301_Topic44 Motivation  In engineering, two types of applications are encountered: Trend analysis: Predicting values of dependent variable, may include extrapolation beyond data points or interpolation between data points. Hypothesis testing: Comparing existing mathematical model with measured data. 1. What is the best mathematical function f that represents the dataset? 2. What is the best criterion to assess the fitting of the function f to the data?

5. CISE301_Topic45 Curve Fitting Given a set of tabulated data, find a curve or a function that best represents the data. Given: 1. The tabulated data 2. The form of the function 3. The curve fitting criteria Find the unknown coefficients

6. CISE301_Topic46 Least Squares Regression Linear Regression  ‌ Fitting a straight line to a set of paired observations: (x 1, y 1 ), (x 2, y 2 ),…,(x n, y n ). y=a 0 +a 1 x+e a 1 -slope. a 0 -intercept. e-error, or residual, between the model and the observations.

7. CISE301_Topic47 Selection of the Functions

8. CISE301_Topic48 Decide on the Criterion Chapter 17 Chapter 18

9. CISE301_Topic49 Least Squares Regression xixi x1x1 x2x2 ….xnxn yiyi y1y1 y2y2 ynyn Given: The form of the function is assumed to be known but the coefficients are unknown. The difference is assumed to be the result of experimental error.

10. CISE301_Topic410 Determine the Unknowns

11. CISE301_Topic411 Determine the Unknowns

12. CISE301_Topic412 Determining the Unknowns

13. CISE301_Topic413 Normal Equations

14. CISE301_Topic414 Solving the Normal Equations

15. CISE301_Topic415 Example 1: Linear Regression x123 y5.15.96.3

16. CISE301_Topic416 Example 1: Linear Regression i123sum xixi 1236 yiyi 5.15.96.317.3 xi2xi2 14914 x i y i 5.111.818.935.8

17. CISE301_Topic417 Multiple Linear Regression Example: Given the following data: Determine a function of two variables: f(x,t) = a + b x + c t That best fits the data with the least sum of the square of errors. t0123 x0.10.40.2 y3212

18. CISE301_Topic418 Solution of Multiple Linear Regression Construct, the sum of the square of the error and derive the necessary conditions by equating the partial derivatives with respect to the unknown parameters to zero, then solve the equations. t0123 x0.10.40.2 y3212

19. CISE301_Topic419 Solution of Multiple Linear Regression

20. CISE301_Topic420 System of Equations

21. CISE301_Topic421 Example 2: Multiple Linear Regression i1234Sum titi 01236 xixi 0.10.40.2 0.9 yiyi 32128 xi2xi2 0.010.160.04 0.25 x i t i 00.4 0.61.4 x i y i 0.30.80.20.41.7 ti2ti2 014914 t i y i 022610

22. CISE301_Topic422 Example 2: System of Equations

23. CISE301_Topic423 Lecture 19 Nonlinear Least Squares Problems  Examples of Nonlinear Least Squares  Solution of Inconsistent Equations  Continuous Least Square Problems

24. Polynomial Regression  The least squares method can be extended to fit the data to a higher-order polynomial CISE301_Topic424

25. Equations for Quadratic Regression CISE301_Topic425

26. Normal Equations CISE301_Topic426

27. Example 3: Polynomial Regression Fit a second-order polynomial to the following data CISE301_Topic427 xixi 012345∑=15 yiyi 2.17.713.627.240.961.1∑=152.6 xi2xi2 01491625∑=55 xi3xi3 0182764125225 xi4xi4 011681256625∑=979 x i y i 07.727.281.6163.6305.5∑=585.6 x i 2 y i 07.754.4244.8654.41527.5∑=2488.8

28. Example 3: Equations and Solution CISE301_Topic428

29. CISE301_Topic429 How Do You Judge Functions?

30. Example showing that Quadratic is preferable than Linear Regression CISE301_Topic430 x y Quadratic Regression x y Linear Regression

31. CISE301_Topic431 Fitting with Nonlinear Functions xixi 0.240.650.951.241.732.012.232.52 yiyi 0.23-0.23-1.1-0.450.270.1-0.290.24

32. CISE301_Topic432 Fitting with Nonlinear Functions

33. CISE301_Topic433 Normal Equations

34. CISE301_Topic434 Example 4: Evaluating Sums xi0.240.650.951.241.732.012.232.52∑=11.57 yi0.23-0.23-1.1-0.450.270.1-0.290.24∑=-1.23 (ln xi) 2 2.0360.18560.00260.04630.30040.48740.64320.8543 ∑=4.556 ln(xi) cos(xi) -1.386-0.3429-0.02980.0699-0.0869-0.2969-0.4912-0.7514∑=-3.316 ln(xi) * e xi -1.814-0.8252-0.13260.74333.09185.21047.458511.487∑=25.219 yi * ln(xi) -0.3280.09910.0564-0.09680.14800.0698-0.23260.2218∑=-0.0625 cos(xi) 2 0.9430.63370.33840.10550.02510.18080.37510.6609∑=3.26307 cos(xi) * e xi 1.2351.52491.50411.1224-0.8942-3.1735-5.696-10.104∑=-14.481 yi*cos(xi) 0.223-0.1831-0.6399-0.1462-0.0428-0.04250.1776-0.1951∑=-0.8485 (e xi ) 2 1.6163.66936.685911.94131.81755.70186.488154.47∑=352.39 yi * e xi 0.2924-0.4406-2.844-1.5551.5230.7463-2.6972.9829∑=-1.9923

35. CISE301_Topic435 Example 4: Equations & Solution

36. CISE301_Topic436 Example 5 Given: xixi 123 yiyi 2.459 Difficult to Solve

37. CISE301_Topic437 Linearization Method

38. CISE301_Topic438 Example 5: Equations

39. CISE301_Topic439 Evaluating Sums and Solving xixi 123∑=6 yiyi 2.459 z i =ln(y i )0.8754691.6094382.197225∑=4.68213 xi2xi2 149∑=14 x i z i 0.8754693.2188766.591674∑=10.6860