1. 1 1 Slide © 2003 South-Western/Thomson Learning™ Slides Prepared by JOHN S. LOUCKS St. Edward’s University

2. 2 2 Slide © 2003 South-Western/Thomson Learning™ Chapter 15 Multiple Regression n Multiple Regression Model n Least Squares Method n Multiple Coefficient of Determination n Model Assumptions n Testing for Significance n Using the Estimated Regression Equation for Estimation and Prediction for Estimation and Prediction n Qualitative Independent Variables n Residual Analysis

3. 3 3 Slide © 2003 South-Western/Thomson Learning™ The Multiple Regression Model n The Multiple Regression Model y =  0 +  1 x 1 +  2 x 2 +... +  p x p +  n The Multiple Regression Equation E( y ) =  0 +  1 x 1 +  2 x 2 +... +  p x p n The Estimated Multiple Regression Equation y = b 0 + b 1 x 1 + b 2 x 2 +... + b p x p ^

4. 4 4 Slide © 2003 South-Western/Thomson Learning™ The Least Squares Method n Least Squares Criterion n Computation of Coefficients Values The formulas for the regression coefficients b 0, b 1, b 2,... b p involve the use of matrix algebra. We will rely on computer software packages to perform the calculations. n A Note on Interpretation of Coefficients b i represents an estimate of the change in y corresponding to a one-unit change in x i when all other independent variables are held constant. ^

5. 5 5 Slide © 2003 South-Western/Thomson Learning™ The Multiple Coefficient of Determination n Relationship Among SST, SSR, SSE SST = SSR + SSE n Multiple Coefficient of Determination R 2 = SSR/SST n Adjusted Multiple Coefficient of Determination ^^

6. 6 6 Slide © 2003 South-Western/Thomson Learning™ Model Assumptions Assumptions About the Error Term  Assumptions About the Error Term  The error  is a random variable with mean of zero. The error  is a random variable with mean of zero. The variance of , denoted by  2, is the same for all values of the independent variables. The variance of , denoted by  2, is the same for all values of the independent variables. The values of  are independent. The values of  are independent. The error  is a normally distributed random variable reflecting the deviation between the y value and the expected value of y given by The error  is a normally distributed random variable reflecting the deviation between the y value and the expected value of y given by  0 +  1 x 1 +  2 x 2 +... +  p x p  0 +  1 x 1 +  2 x 2 +... +  p x p

7. 7 7 Slide © 2003 South-Western/Thomson Learning™ Testing for Significance: F Test n Hypotheses H 0 :  1 =  2 =... =  p = 0 H 0 :  1 =  2 =... =  p = 0 H a : One or more of the parameters H a : One or more of the parameters is not equal to zero. is not equal to zero. n Test Statistic F = MSR/MSE

8. 8 8 Slide © 2003 South-Western/Thomson Learning™ Testing for Significance: F Test n Rejection Rule Using test statistic: Reject H 0 if F > F  Using p-value: Reject H 0 if p -value <  where F  is based on an F distribution with p d.f. in the numerator and n - p - 1 d.f. in the denominator

9. 9 9 Slide © 2003 South-Western/Thomson Learning™ Testing for Significance: F Test n ANOVA Table (assuming p independent variables) Source of Sum of Degrees of Mean Source of Sum of Degrees of Mean Variation Squares Freedom Squares F Variation Squares Freedom Squares F Regression SSR p Error SSE n - p - 1 Total SST n - 1

10. 10 Slide © 2003 South-Western/Thomson Learning™ Testing for Significance: t Test n Hypotheses H 0 :  i = 0 H 0 :  i = 0 H a :  i = 0 H a :  i = 0 n Test Statistic

11. 11 Slide © 2003 South-Western/Thomson Learning™ Testing for Significance: t Test n Rejection Rule Using test statistic: Reject H 0 if t t  Using p-value: Reject H 0 if p -value <  where t  is based on a t distribution with n - p - 1 degrees of freedom n - p - 1 degrees of freedom

12. 12 Slide © 2003 South-Western/Thomson Learning™ Testing for Significance: Multicollinearity n The term multicollinearity refers to the correlation among the independent variables. n When the independent variables are highly correlated (say, | r | >.7), it is not possible to determine the separate effect of any particular independent variable on the dependent variable. n If the estimated regression equation is to be used only for predictive purposes, multicollinearity is usually not a serious problem. n Every attempt should be made to avoid including independent variables that are highly correlated.

13. 13 Slide © 2003 South-Western/Thomson Learning™ Using the Estimated Regression Equation for Estimation and Prediction n The procedures for estimating the mean value of y and predicting an individual value of y in multiple regression are similar to those in simple regression. n We substitute the given values of x 1, x 2,..., x p into the estimated regression equation and use the corresponding value of y as the point estimate. n The formulas required to develop interval estimates for the mean value of y and for an individual value of y are beyond the scope of the text. n Software packages for multiple regression will often provide these interval estimates. ^

14. 14 Slide © 2003 South-Western/Thomson Learning™ Example: Programmer Salary Survey A software firm collected data for a sample of 20 computer programmers. A suggestion was made that regression analysis could be used to determine if salary was related to the years of experience and the score on the firm’s programmer aptitude test. The years of experience, score on the aptitude test, and corresponding annual salary ($1000s) for a sample of 20 programmers is shown on the next slide.

15. 15 Slide © 2003 South-Western/Thomson Learning™ Example: Programmer Salary Survey Exper. Score Salary Exper. Score Salary Exper. Score Salary Exper. Score Salary 4782498838 71004327326.6 18623.7107536.2 58234.358131.6 88635.867429 10843888734 07522.247930.1 18023.169433.9 6833037028.2 6913338930

16. 16 Slide © 2003 South-Western/Thomson Learning™ Example: Programmer Salary Survey n Multiple Regression Model Suppose we believe that salary ( y ) is related to the years of experience ( x 1 ) and the score on the programmer aptitude test ( x 2 ) by the following regression model: y =  0 +  1 x 1 +  2 x 2 +  where y = annual salary ($000) y = annual salary ($000) x 1 = years of experience x 1 = years of experience x 2 = score on programmer aptitude test x 2 = score on programmer aptitude test

17. 17 Slide © 2003 South-Western/Thomson Learning™ Example: Programmer Salary Survey n Multiple Regression Equation Using the assumption E (  ) = 0, we obtain E( y ) =  0 +  1 x 1 +  2 x 2 n Estimated Regression Equation b 0, b 1, b 2 are the least squares estimates of  0,  1,  2 b 0, b 1, b 2 are the least squares estimates of  0,  1,  2Thus y = b 0 + b 1 x 1 + b 2 x 2 ^

18. 18 Slide © 2003 South-Western/Thomson Learning™ Example: Programmer Salary Survey Solving for the Estimates of  0,  1,  2 Solving for the Estimates of  0,  1,  2 ComputerPackage for Solving MultipleRegressionProblemsComputerPackage MultipleRegressionProblems b 0 = b 1 = b 1 = b 2 = b 2 = R 2 = etc. b 0 = b 1 = b 1 = b 2 = b 2 = R 2 = etc. Input Data Least Squares Output x 1 x 2 y 4 78 24 4 78 24 7 100 43 7 100 43...... 3 89 30 3 89 30 x 1 x 2 y 4 78 24 4 78 24 7 100 43 7 100 43...... 3 89 30 3 89 30

19. 19 Slide © 2003 South-Western/Thomson Learning™ Using Excel’s Regression Tool to Develop the Estimated Multiple Regression Equation n Formula Worksheet (showing data entered) Note: Rows 10-21 are not shown.

20. 20 Slide © 2003 South-Western/Thomson Learning™ n Performing the Multiple Regression Analysis Step 1 Select the Tools pull-down menu Step 2 Choose the Data Analysis option Step 3 Choose Regression from the list of Analysis Tools … continued Using Excel’s Regression Tool to Develop the Estimated Multiple Regression Equation

21. 21 Slide © 2003 South-Western/Thomson Learning™ n Performing the Multiple Regression Analysis Step 4 When the Regression dialog box appears: Enter D1:D21 in the Input Y Range box Enter D1:D21 in the Input Y Range box Enter B1:C21 in the Input X Range box Enter B1:C21 in the Input X Range box Select Labels Select Labels Select Confidence Level Select Confidence Level Enter 95 in the Confidence Level box Enter 95 in the Confidence Level box Select Output Range and enter A24 in the Select Output Range and enter A24 in the Output Range box Click OK Click OK Using Excel’s Regression Tool to Develop the Estimated Multiple Regression Equation

22. 22 Slide © 2003 South-Western/Thomson Learning™ n Value Worksheet (Regression Statistics) Using Excel’s Regression Tool to Develop the Estimated Multiple Regression Equation

23. 23 Slide © 2003 South-Western/Thomson Learning™ n Value Worksheet (ANOVA Output) Using Excel’s Regression Tool to Develop the Estimated Multiple Regression Equation The Significance F value in cell F35 is the p -value used to test for overall significance.

24. 24 Slide © 2003 South-Western/Thomson Learning™ n Value Worksheet (Regression Equation Output) Using Excel’s Regression Tool to Develop the Estimated Multiple Regression Equation Note: Columns F-I are not shown. The P-value in cell E41 is used to test for the individual significance of Experience.

25. 25 Slide © 2003 South-Western/Thomson Learning™ n Value Worksheet (Regression Equation Output) Using Excel’s Regression Tool to Develop the Estimated Multiple Regression Equation Note: Columns F-I are not shown. The P-value in cell E42 is used to test for the individual significance of Test Score.

26. 26 Slide © 2003 South-Western/Thomson Learning™ n Estimated Regression Equation SALARY = 3.174 + 1.404(EXPER) + 0.2509(SCORE) SALARY = 3.174 + 1.404(EXPER) + 0.2509(SCORE) Note: Predicted salary will be in thousands of dollars Using Excel’s Regression Tool to Develop the Estimated Multiple Regression Equation

27. 27 Slide © 2003 South-Western/Thomson Learning™ n Value Worksheet (Regression Equation Output) Using Excel’s Regression Tool to Develop the Estimated Multiple Regression Equation Note: Columns C-E are hidden.

28. 28 Slide © 2003 South-Western/Thomson Learning™ Example: Programmer Salary Survey n F Test Hypotheses H 0 :  1 =  2 = 0 Hypotheses H 0 :  1 =  2 = 0 H a : One or both of the parameters H a : One or both of the parameters is not equal to zero. is not equal to zero. Rejection Rule Rejection Rule For  =.05 and d.f. = 2, 17: F.05 = 3.59 For  =.05 and d.f. = 2, 17: F.05 = 3.59 Reject H 0 if F > 3.59. Test Statistic Test Statistic F = MSR/MSE = 250.16/5.85 = 42.76 Conclusion Conclusion We can reject H 0. We can reject H 0.

29. 29 Slide © 2003 South-Western/Thomson Learning™ Example: Programmer Salary Survey n t Test for Significance of Individual Parameters Hypotheses H 0 :  i = 0 Hypotheses H 0 :  i = 0 H a :  i = 0 H a :  i = 0 Rejection Rule Rejection Rule For  =.05 and d.f. = 17, t.025 = 2.11 For  =.05 and d.f. = 17, t.025 = 2.11 Reject H 0 if t > 2.11 Reject H 0 if t > 2.11 Test Statistics Test Statistics Conclusions Conclusions Reject H 0 :  1 = 0 Reject H 0 :  2 = 0 Reject H 0 :  1 = 0 Reject H 0 :  2 = 0

30. 30 Slide © 2003 South-Western/Thomson Learning™ Qualitative Independent Variables n In many situations we must work with qualitative independent variables such as gender (male, female), method of payment (cash, check, credit card), etc. n For example, x 2 might represent gender where x 2 = 0 indicates male and x 2 = 1 indicates female. n In this case, x 2 is called a dummy or indicator variable. n If a qualitative variable has k levels, k - 1 dummy variables are required, with each dummy variable being coded as 0 or 1. n For example, a variable with levels A, B, and C would be represented by x 1 and x 2 values of (0, 0), (1, 0), and (0,1), respectively.

31. 31 Slide © 2003 South-Western/Thomson Learning™ Example: Programmer Salary Survey (B) As an extension of the problem involving the computer programmer salary survey, suppose that management also believes that the annual salary is related to whether or not the individual has a graduate degree in computer science or information systems. The years of experience, the score on the programmer aptitude test, whether or not the individual has a relevant graduate degree, and the annual salary ($000) for each of the sampled 20 programmers are shown on the next slide.

32. 32 Slide © 2003 South-Western/Thomson Learning™ Example: Programmer Salary Survey (B) Exp. Score Degr. Salary Exp. Score Degr. Salary Exp. Score Degr. Salary Exp. Score Degr. Salary 478No24988Yes38 7100Yes43273No26.6 186No23.71075Yes36.2 582Yes34.3581No31.6 886Yes35.8674No29 1084Yes38887Yes34 075No22.2479No30.1 180 No 23.1694Yes33.9 683No30370No28.2 691Yes33389No30

33. 33 Slide © 2003 South-Western/Thomson Learning™ Example: Programmer Salary Survey (B) n Multiple Regression Equation E( y ) =  0 +  1 x 1 +  2 x 2 +  3 x 3 n Estimated Regression Equation y = b 0 + b 1 x 1 + b 2 x 2 + b 3 x 3 y = b 0 + b 1 x 1 + b 2 x 2 + b 3 x 3where y = annual salary ($000) y = annual salary ($000) x 1 = years of experience x 1 = years of experience x 2 = score on programmer aptitude test x 3 = 0 if individual does not have a grad. degree 1 if individual does have a grad. degree 1 if individual does have a grad. degree Note: x 3 is referred to as a dummy variable. ^

34. 34 Slide © 2003 South-Western/Thomson Learning™ Using Excel’s Regression Tool to Develop the Estimated Multiple Regression Equation n Formula Worksheet (showing data) Note: Rows 9-21 are not shown.

35. 35 Slide © 2003 South-Western/Thomson Learning™ Using Excel’s Regression Tool to Develop the Estimated Multiple Regression Equation n Value Worksheet (Regression Statistics)

36. 36 Slide © 2003 South-Western/Thomson Learning™ Using Excel’s Regression Tool to Develop the Estimated Multiple Regression Equation n Value Worksheet (ANOVA Output)

37. 37 Slide © 2003 South-Western/Thomson Learning™ Using Excel’s Regression Tool to Develop the Estimated Multiple Regression Equation n Value Worksheet (Regression Equation Output) Note: Columns F-I are not shown.

38. 38 Slide © 2003 South-Western/Thomson Learning™ Using Excel’s Regression Tool to Develop the Estimated Multiple Regression Equation n Value Worksheet (Regression Equation Output) Note: Columns C-E are hidden.

39. 39 Slide © 2003 South-Western/Thomson Learning™ Example: Programmer Salary Survey (B) n Interpreting the Parameters b 1 = 1.15 b 1 = 1.15 Salary is expected to increase by $1,150 for each additional year of experience (when all other independent variables are held constant)

40. 40 Slide © 2003 South-Western/Thomson Learning™ Example: Programmer Salary Survey (B) n Interpreting the Parameters b 2 = 0.197 b 2 = 0.197 Salary is expected to increase by $197 for each additional point scored on the programmer aptitude test (when all other independent variables are held constant)

41. 41 Slide © 2003 South-Western/Thomson Learning™ Example: Programmer Salary Survey (B) n Interpreting the Parameters b 3 = 2.28 b 3 = 2.28 Salary is expected to be $2,280 higher for an individual with a graduate degree than one without a graduate degree (when all other independent variables are held constant)

42. 42 Slide © 2003 South-Western/Thomson Learning™ Residual Analysis n For simple linear regression the residual plot against and the residual plot against x provide the same information. and the residual plot against x provide the same information. n In multiple regression analysis it is preferable to use the residual plot against to determine if the model assumptions are satisfied.

43. 43 Slide © 2003 South-Western/Thomson Learning™ Residual Analysis n Standardized residuals are frequently used in residual plots for purposes of: Identifying outliers (typically, standardized residuals +2) Identifying outliers (typically, standardized residuals +2) Providing insight about the assumption that the error term  has a normal distribution Providing insight about the assumption that the error term  has a normal distribution n The computation of the standardized residuals in multiple regression analysis is too complex to be done by hand n Excel’s Regression tool can be used

44. 44 Slide © 2003 South-Western/Thomson Learning™ Using Excel to Construct a Standardized Residual Plot n Value Worksheet (Residual Output) Note: Rows 37-51 are not shown.

45. 45 Slide © 2003 South-Western/Thomson Learning™ Using Excel to Construct a Standardized Residual Plot Outlier

46. 46 Slide © 2003 South-Western/Thomson Learning™ End of Chapter 15