1. © 1998, Geoff Kuenning Linear Regression Models What is a (good) model? Estimating model parameters Allocating variation Confidence intervals for regressions Verifying assumptions visually

2. Data Analysis Overview Experimental environment prototype real sys exec- driven sim trace- driven sim stochastic sim Workload parameters System Config parameters Factor levels Raw Data Samples...... Predictor values x, factor levels Samples of response... y1y2yny1y2yn Regression models response var = f (predictors)

3. © 1998, Geoff Kuenning What Is a (Good) Model? For correlated data, model predicts response given an input Model should be equation that fits data Standard definition of “fits” is least-squares –Minimize squared error –While keeping mean error zero –Minimizes variance of errors

4. © 1998, Geoff Kuenning Least-Squared Error If then error in estimate for x i is Minimize Sum of Squared Errors (SSE) Subject to the constraint. e i = y i - y i.

5. © 1998, Geoff Kuenning Estimating Model Parameters Best regression parameters are where Note error in eq. 14.1 book!

6. © 1998, Geoff Kuenning Parameter Estimation Example Execution time of a script for various loop counts: = 6.8, = 2.32,  xy = 88.54,  x 2 = 264 b 0 = 2.32  (0.29)(6.8) = 0.35

7. © 1998, Geoff Kuenning Graph of Parameter Estimation Example

8. © 1998, Geoff Kuenning Variants of Linear Regression Some non-linear relationships can be handled by transformations –For y = ae bx take logarithm of y, do regression on log(y) = b 0 +b 1 x, let b = b 1, –For y = a+b log(x), take log of x before fitting parameters, let b = b 1, a = b 0 –For y = ax b, take log of both x and y, let b = b 1,

9. © 1998, Geoff Kuenning Allocating Variation If no regression, best guess of y is Observed values of y differ from, giving rise to errors (variance) Regression gives better guess, but there are still errors We can evaluate quality of regression by allocating sources of errors

10. © 1998, Geoff Kuenning The Total Sum of Squares Without regression, squared error is

11. © 1998, Geoff Kuenning The Sum of Squares from Regression Recall that regression error is Error without regression is SST So regression explains SSR = SST - SSE Regression quality measured by coefficient of determination.

12. © 1998, Geoff Kuenning Evaluating Coefficient of Determination Compute

13. © 1998, Geoff Kuenning Example of Coefficient of Determination For previous regression example –  y = 11.60,  y 2 = 29.79,  xy = 88.54, –SSE = 29.79-(0.35)(11.60)-(0.29)(88.54) = 0.05 –SST = 29.79-26.9 = 2.89 –SSR = 2.89-.05 = 2.84 –R 2 = (2.89-0.05)/2.89 = 0.98

14. © 1998, Geoff Kuenning Standard Deviation of Errors Variance of errors is SSE divided by degrees of freedom –DOF is n  2 because we’ve calculated 2 regression parameters from the data –So variance (mean squared error, MSE) is SSE/(n  2) Standard deviation of errors is square root:

15. © 1998, Geoff Kuenning Checking Degrees of Freedom Degrees of freedom always equate: –SS0 has 1 (computed from ) –SST has n  1 (computed from data and, which uses up 1) –SSE has n  2 (needs 2 regression parameters) –So

16. © 1998, Geoff Kuenning Example of Standard Deviation of Errors For our regression example, SSE was 0.05, so MSE is 0.05/3 = 0.017 and s e = 0.13 Note high quality of our regression: –R 2 = 0.98 –s e = 0.13 –Why such a nice straight-line fit?

17. © 1998, Geoff Kuenning Confidence Intervals for Regressions Regression is done from a single population sample (size n) –Different sample might give different results –True model is y =  0 +  1 x –Parameters b 0 and b 1 are really means taken from a population sample

18. © 1998, Geoff Kuenning Calculating Intervals for Regression Parameters Standard deviations of parameters: Confidence intervals are where t has n - 2 degrees of freedom !

19. © 1998, Geoff Kuenning Example of Regression Confidence Intervals Recall s e = 0.13, n = 5,  x 2 = 264, = 6.8 So Using a 90% confidence level, t 0.95;3 = 2.353

20. © 1998, Geoff Kuenning Regression Confidence Example, cont’d Thus, b 0 interval is –Not significant at 90% And b 1 is –Significant at 90% (and would survive even 99.9% test) ! !

21. © 1998, Geoff Kuenning Confidence Intervals for Nonlinear Regressions For nonlinear fits using exponential transformations: –Confidence intervals apply to transformed parameters –Not valid to perform inverse transformation on intervals

22. © 1998, Geoff Kuenning Confidence Intervals for Predictions Previous confidence intervals are for parameters –How certain can we be that the parameters are correct? Purpose of regression is prediction –How accurate are the predictions? –Regression gives mean of predicted response, based on sample we took

23. © 1998, Geoff Kuenning Predicting m Samples Standard deviation for mean of future sample of m observations at x p is Note deviation drops as m  Variance minimal at x = Use t-quantiles with n–2 DOF for interval

24. © 1998, Geoff Kuenning Example of Confidence of Predictions Using previous equation, what is predicted time for a single run of 8 loops? Time = 0.35 + 0.29(8) = 2.67 Standard deviation of errors s e = 0.13 90% interval is then !

25. © 1998, Geoff Kuenning Verifying Assumptions Visually Regressions are based on assumptions: –Linear relationship between response y and predictor x Or nonlinear relationship used in fitting –Predictor x nonstochastic and error-free –Model errors statistically independent With distribution N(0,c) for constant c If assumptions violated, model misleading or invalid

26. © 1998, Geoff Kuenning Testing Linearity Scatter plot x vs. y to see basic curve type LinearPiecewise Linear OutlierNonlinear (Power)

27. © 1998, Geoff Kuenning Testing Independence of Errors Scatter-plot  i versus Should be no visible trend Example from our curve fit:

28. © 1998, Geoff Kuenning More on Testing Independence May be useful to plot error residuals versus experiment number –In previous example, this gives same plot except for x scaling No foolproof tests

29. © 1998, Geoff Kuenning Testing for Normal Errors Prepare quantile-quantile plot Example for our regression:

30. © 1998, Geoff Kuenning Testing for Constant Standard Deviation Tongue-twister: homoscedasticity Return to independence plot Look for trend in spread Example:

31. © 1998, Geoff Kuenning Linear Regression Can Be Misleading Regression throws away some information about the data –To allow more compact summarization Sometimes vital characteristics are thrown away –Often, looking at data plots can tell you whether you will have a problem

32. © 1998, Geoff Kuenning Example of Misleading Regression IIIIIIIV xyxyxyxy 108.04109.14107.4686.58 86.9588.1486.7785.76 137.58138.741312.7487.71 98.8198.7797.1188.84 118.33119.26117.8188.47 149.96148.10148.8487.04 67.2466.1366.0885.25 44.2643.1045.391912.50 1210.84129.13128.1585.56 74.8277.2676.4287.91 55.6854.7455.7386.89

33. © 1998, Geoff Kuenning What Does Regression Tell Us About These Data Sets? Exactly the same thing for each! N = 11 Mean of y = 7.5 Y = 3 +.5 X Standard error of regression is 0.118 All the sums of squares are the same Correlation coefficient =.82 R 2 =.67

34. © 1998, Geoff Kuenning Now Look at the Data Plots III IIIIV