1. Linear Regression Linear Regression

2. Copyright © 2005 Brooks/Cole, a division of Thomson Learning, Inc. Purpose Understand Linear Regression. Use R functions for Linear Regression (Ordinary Least Squares – OLS) Predict the dependent variables based on the model. Investigate different statistical parameter tests that measure the effectiveness of the model 2

3. Introduction The motivation for using Regression technique: – Forecast the value of a dependent variable (Y) from the value of independent variables (X 1, X 2,…X k.). – Analyze the specific relationships between the independent variables and the dependent variable.

4. House size House Cost Most lots sell for $25,000 Building a house costs about $75 per square foot. House cost = 25000 + 75(Size) The Model The model has a deterministic and a probabilistic components

5. House cost = 25000 + 75 (Size) House size House Cost Most lots sell for $25,000   However, house cost vary even among same size houses! Since cost behave unpredictably, we add a random component.

6. The first order linear model Y = dependent variable X = independent variable  0 = Y-intercept  1 = slope of the line  = error variable X Y 00 Run Rise   = Rise/Run  0 and  1 are unknown population parameters, therefore are estimated from the data.

7. Estimating the Coefficients The estimates are determined by – drawing a sample from the population of interest, – calculating sample statistics. – producing a straight line that cuts into the data.           Question: What should be considered a good line? X Y

8. The Least Squares (Regression) Line A good line is one that minimizes the sum of squared differences between the points and the line.

9. 3 3     4 1 1 4 (1,2) 2 2 (2,4) (3,1.5) Sum of squared differences =(2 - 1) 2 +(4 - 2) 2 +(1.5 - 3) 2 + (4,3.2) (3.2 - 4) 2 = 6.89 Sum of squared differences =(2 -2.5) 2 +(4 - 2.5) 2 +(1.5 - 2.5) 2 +(3.2 - 2.5) 2 = 3.99 2.5 Let us compare two lines The second line is horizontal The smaller the sum of squared differences the better the fit of the line to the data.

10. The Estimated Coefficients To calculate the estimates of the line coefficients, that minimize the differences between the data points and the line, use the formulas: The regression equation that estimates the equation of the first order linear model is:

11. Example – A car dealer wants to find the relationship between the odometer reading and the selling price of used cars. – A random sample of 100 cars is selected, and the data recorded. – Find the regression line. Independent variable X Dependent variable Y The Simple Linear Regression Line

12. Solution –Solving by hand: Calculate a number of statistics where n = 100.

13. Output

14. This is the slope of the line. For each additional mile on the odometer, the price decreases by an average of $0.0623 Interpreting the Linear Regression - Equation The intercept is b 0 = $17067. 0 No data Do not interpret the intercept as the “Price of cars that have not been driven” 17067

15. Error Variable: Required Conditions The error  is a critical part of the regression model. Four requirements involving the distribution of  must be satisfied. – The probability distribution of  is normal. – The mean of  is zero: E(  ) = 0. – The standard deviation of  is   for all values of X. – The set of errors associated with different values of Y are all independent.

16. The Normality of  From the first three assumptions we have: Y is normally distributed with mean E(Y) =  0 +  1 X, and a constant standard deviation   From the first three assumptions we have: Y is normally distributed with mean E(Y) =  0 +  1 X, and a constant standard deviation     0 +  1 X 1  0 +  1 X 2  0 +  1 X 3 E(Y|X 2 ) E(Y|X 3 ) X1X1 X2X2 X3X3  E(Y|X 1 )  The standard deviation remains constant, but the mean value changes with X

17. Assessing the Model The least squares method will produces a regression line whether or not there are linear relationship between X and Y. Consequently, it is important to assess how well the linear model fits the data. Several methods are used to assess the model. All are based on the sum of squares for errors, SSE.

18. Copyright © 2005 Brooks/Cole, a division of Thomson Learning, Inc. R code 1. Run the “runif” function in R which generates random deviates within the specified minimum and maximum range. x <- runif(100, 0, 10) This generates 100 random values for “x” in the range 0 to 10. 2. Create the dependent variable “y” with the “beta” values as 5 and 6 and the “sigma” = 1 (generated with the “rnorm” function, random generation for the normal distribution with mean =0 and SD= 1.) y <- 5 + 6*x + rnorm(100) 3. Plot it > plot(x,y) 4. Review the results in the graphics window 18

19. Copyright © 2005 Brooks/Cole, a division of Thomson Learning, Inc. 19 An OLS Model is generated with an R function call “lm”. d <- lm(y ~ x) Use the following command to display the structure of the object “d” created with the function call “lm” str(d) You can see the details of the fitted model. Get the compact results of the model with the following command: print(d) Visualize the model with the command: par(mfrow=c(2,2)) plot(d)

20. Copyright © 2005 Brooks/Cole, a division of Thomson Learning, Inc. 20 The explanation of the plots are as follows: Residuals vs. Fitted : you want to make sure that the errors are evenly distributed over the entire range of fitted values; if the errors are markedly larger (or biased either positively or negatively) in some range of the data, this is evidence that this model may not be entirely appropriate for the problem. Q-Q plot : tests whether or not the errors are in fact distributed approximately normally (as the model formulation assumes). If they are, the Q-Q plot will be along the x=y line. If they aren't, the model may still be adequate, but perhaps a more robust modeling method is suitable. Scale-Location: a similar idea to Residuals v. Fitted; you want to make sure that the variance (or its stand-in, "scale") is approximately constant over the range of fitted values. Residuals vs. Leverage: used for identifying potential outliers and "influential" points. Points that are far from the centroid of the scatter plot in the x direction (high leverage) are influential, in the sense that they may have disproportionate influence on the fit (that doesn't mean they are wrong, necessarily). Points that are far from the centroid in the y direction (large residuals) are potential outliers.

21. Copyright © 2005 Brooks/Cole, a division of Thomson Learning, Inc. Type following commands >ypred <- predict(d) > par(mfrow=c(1,1)) > plot(y,y, type="l", xlab="true y", ylab="predicted y") > points(y, ypred) Review the results in the graphics window 21