1. 1 1 Slide Multiple Regression n Multiple Regression Model n Least Squares Method n Multiple Coefficient of Determination n Model Assumptions n Testing for Significance n Using the Estimated Regression Equation for Estimation and Prediction for Estimation and Prediction n Qualitative Independent Variables n Residual Analysis

2. 2 2 Slide n In this chapter we continue our study of regression analysis by considering situations involving two or more independent variables. Multiple Regression n This subject area, called multiple regression analysis, enables us to consider more factors and thus obtain better estimates than are possible with simple linear regression.

3. 3 3 Slide The equation that describes how the dependent variable y is related to the independent variables x 1, x 2,... x p and an error term is: The equation that describes how the dependent variable y is related to the independent variables x 1, x 2,... x p and an error term is: Multiple Regression Model y =  0 +  1 x 1 +  2 x 2 +... +  p x p +  where:  0,  1,  2,...,  p are the parameters,  is a random variable called the error term, p is the number of independent variables n Multiple Regression Model

4. 4 4 Slide The equation that describes how the mean value of y is related to x 1, x 2,... x p is called the multiple regression equation. Multiple Regression Equation E ( y ) =  0 +  1 x 1 +  2 x 2 +... +  p x p

5. 5 5 Slide A simple random sample is used to compute sample statistics b 0, b 1, b 2,..., b p that are used as the point estimators of the parameters  0,  1,  2,...,  p. Estimated Multiple Regression Equation ^ y = b 0 + b 1 x 1 + b 2 x 2 +... + b p x p The estimated multiple regression equation is:

6. 6 6 Slide Estimation Process Multiple Regression Model E ( y ) =  0 +  1 x 1 +  2 x 2 +...+  p x p +  Multiple Regression Equation E ( y ) =  0 +  1 x 1 +  2 x 2 +...+  p x p Unknown parameters are  0,  1,  2,...,  p Sample Data: x 1 x 2... x p y.... Estimated Multiple Regression Equation Sample statistics are b 0, b 1, b 2,..., b p b 0, b 1, b 2,..., b p b 0, b 1, b 2,..., b p provide estimates of  0,  1,  2,...,  p

7. 7 7 Slide Least Squares Method n Least Squares Criterion where: y i = observed value of the dependent variable for the i th observation variable for the i th observation  = estimated value of the dependent variable for the i th observation variable for the i th observation

8. 8 8 Slide The years of experience, score on the aptitude The years of experience, score on the aptitude test, and corresponding annual salary ($1000s) for a sample of 20 programmers is shown on the next slide. n Example: Programmer Salary Survey Multiple Regression Model A software firm collected data for a sample A software firm collected data for a sample of 20 computer programmers. A suggestion was made that regression analysis could be used to determine if salary was related to the years of experience and the score on the firm’s programmer aptitude test.

9. 9 9 Slide 471581001669210568463378100868286847580839188737581748779947089 244323.734.335.83822.223.13033 3826.636.231.6293430.133.928.230 Exper.ScoreScoreExper.SalarySalary Multiple Regression Model

10. 10 Slide Suppose we believe that salary ( y ) is Suppose we believe that salary ( y ) is related to the years of experience ( x 1 ) and the score on the programmer aptitude test ( x 2 ) by the following regression model: Multiple Regression Model where y = annual salary ($1000) y = annual salary ($1000) x 1 = years of experience x 1 = years of experience x 2 = score on programmer aptitude test x 2 = score on programmer aptitude test y =  0 +  1 x 1 +  2 x 2 + 

11. 11 Slide Solving for the Estimates of  0,  1,  2 Input Data Least Squares Output x 1 x 2 y 4 78 24 4 78 24 7 100 43 7 100 43...... 3 89 30 3 89 30 ComputerPackage for Solving MultipleRegressionProblems b 0 = b 0 = b 1 = b 2 = R 2 = etc. Download ProgramerSalary.xlsx

12. 12 Slide n Excel Worksheet (showing partial data) Note: Rows 10-21 are not shown. Solving for the Estimates of  0,  1,  2

13. 13 Slide Using Excel’s Regression Tool n Performing the Regression Analysis Step 3 Choose Regression from the list of Analysis Tools Analysis Tools Step 2 In the Analysis group, click Data Analysis Step 1 Click the Data tab on the Ribbon

14. 14 Slide n Excel’s Regression Dialog Box Solving for the Estimates of  0,  1,  2

15. 15 Slide n Excel’s Regression Equation Output Note: Columns F-I are not shown. Solving for the Estimates of  0,  1,  2

16. 16 Slide Estimated Regression Equation SALARY = 3.174 + 1.404(EXPER) + 0.251(SCORE) Note: Predicted salary will be in thousands of dollars.

17. 17 Slide Interpreting the Coefficients In multiple regression analysis, we interpret each In multiple regression analysis, we interpret each regression coefficient as follows: regression coefficient as follows: b i represents an estimate of the change in y b i represents an estimate of the change in y corresponding to a 1-unit increase in x i when all corresponding to a 1-unit increase in x i when all other independent variables are held constant. other independent variables are held constant.

18. 18 Slide Salary is expected to increase by $1,404 for Salary is expected to increase by $1,404 for each additional year of experience (when the variable score on programmer attitude test is held constant). b 1 = 1. 404 Interpreting the Coefficients

19. 19 Slide Salary is expected to increase by $251 for each Salary is expected to increase by $251 for each additional point scored on the programmer aptitude additional point scored on the programmer aptitude test (when the variable years of experience is held constant). b 2 = 0.251 Interpreting the Coefficients

20. 20 Slide Multiple Coefficient of Determination n Relationship Among SST, SSR, SSE where: SST = total sum of squares SST = total sum of squares SSR = sum of squares due to regression SSR = sum of squares due to regression SSE = sum of squares due to error SSE = sum of squares due to error SST = SSR + SSE

21. 21 Slide n Excel’s ANOVA Output Multiple Coefficient of Determination SSR SST

22. 22 Slide Multiple Coefficient of Determination R 2 = 500.3285/599.7855 =.83418 R 2 = SSR/SST

23. 23 Slide Adjusted Multiple Coefficient of Determination n Adding independent variables, even ones that are not statistically significant, causes the prediction errors to become smaller, thus reducing the sum of squares due to error, SSE. n Because SSR = SST – SSE, when SSE becomes smaller, SSR becomes larger, causing R 2 = SSR/SST to increase. n The adjusted multiple coefficient of determination compensates for the number of independent variables in the model.

24. 24 Slide Adjusted Multiple Coefficient of Determination where: n is the number of observations n is the number of observations p is the number of independent variables p is the number of independent variables After adjusting the number of independent variables, we see 81.46% of variability in salary can be explained by the regression.

25. 25 Slide n Excel’s Regression Statistics Adjusted Multiple Coefficient of Determination

26. 26 Slide The variance of , denoted by  2, is the same for all The variance of , denoted by  2, is the same for all values of the independent variables. values of the independent variables. The variance of , denoted by  2, is the same for all The variance of , denoted by  2, is the same for all values of the independent variables. values of the independent variables. The error  is a normally distributed random variable The error  is a normally distributed random variable reflecting the deviation between the y value and the reflecting the deviation between the y value and the expected value of y given by  0 +  1 x 1 +  2 x 2 +.. +  p x p. expected value of y given by  0 +  1 x 1 +  2 x 2 +.. +  p x p. The error  is a normally distributed random variable The error  is a normally distributed random variable reflecting the deviation between the y value and the reflecting the deviation between the y value and the expected value of y given by  0 +  1 x 1 +  2 x 2 +.. +  p x p. expected value of y given by  0 +  1 x 1 +  2 x 2 +.. +  p x p. Assumptions About the Error Term  The error  is a random variable with mean of zero. The error  is a random variable with mean of zero. The values of  are independent. The values of  are independent.

27. 27 Slide In simple linear regression, the F and t tests provide In simple linear regression, the F and t tests provide the same conclusion. the same conclusion. In simple linear regression, the F and t tests provide In simple linear regression, the F and t tests provide the same conclusion. the same conclusion. Testing for Significance In multiple regression, the F and t tests have different In multiple regression, the F and t tests have different purposes. purposes. In multiple regression, the F and t tests have different In multiple regression, the F and t tests have different purposes. purposes.

28. 28 Slide Testing for Significance: F Test The F test is referred to as the test for overall The F test is referred to as the test for overall significance. significance. The F test is referred to as the test for overall The F test is referred to as the test for overall significance. significance. The F test is used to determine whether a significant The F test is used to determine whether a significant relationship exists between the dependent variable relationship exists between the dependent variable and the set of all the independent variables. and the set of all the independent variables. The F test is used to determine whether a significant The F test is used to determine whether a significant relationship exists between the dependent variable relationship exists between the dependent variable and the set of all the independent variables. and the set of all the independent variables.

29. 29 Slide A separate t test is conducted for each of the A separate t test is conducted for each of the independent variables in the model. independent variables in the model. A separate t test is conducted for each of the A separate t test is conducted for each of the independent variables in the model. independent variables in the model. If the F test shows an overall significance, the t test is If the F test shows an overall significance, the t test is used to determine whether each of the individual used to determine whether each of the individual independent variables is significant. independent variables is significant. If the F test shows an overall significance, the t test is If the F test shows an overall significance, the t test is used to determine whether each of the individual used to determine whether each of the individual independent variables is significant. independent variables is significant. Testing for Significance: t Test We refer to each of these t tests as a test for individual We refer to each of these t tests as a test for individual significance. significance. We refer to each of these t tests as a test for individual We refer to each of these t tests as a test for individual significance. significance.

30. 30 Slide Testing for Significance: F Test Hypotheses Rejection Rule Test Statistics H 0 :  1 =  2 =... =  p = 0 H 0 :  1 =  2 =... =  p = 0 H a : One or more of the parameters H a : One or more of the parameters is not equal to zero. is not equal to zero. F = MSR/MSE Reject H 0 if p -value F   where F  is based on an F distribution with p d.f. in the numerator and n - p - 1 d.f. in the denominator. p is the number of independent variables

31. 31 Slide F Test for Overall Significance Hypotheses H 0 :  1 =  2 = 0 H 0 :  1 =  2 = 0 H a : One or both of the parameters H a : One or both of the parameters is not equal to zero. is not equal to zero. Rejection Rule For  =.05 and d.f. = 2, 17; F.05 = 3.59 p -value approach: Reject H 0 if p -value <.05 Critical value approach: Reject H 0 F > 3.59

32. 32 Slide n Excel’s ANOVA Output F Test for Overall Significance p -value used to test for overall significance overall significance p -value = FDIST(42.76013,2,17)=2.32774E-07 numerator’s DF denominator’s DF

33. 33 Slide F Test for Overall Significance Test Statistics F = MSR/MSE = 250.16/5.85 = 42.76 = 250.16/5.85 = 42.76 Conclusion p -value <.05, so we can reject H 0. (Also, F = 42.76 > 3.59) The regression includes at least one significant independent variable.

34. 34 Slide Testing for Significance: t Test Hypotheses Rejection Rule Test Statistics where t  is based on a t distribution with n - p - 1 degrees of freedom. p is the number of independent variables. p-value approach: Reject H 0 if p -value <  Critical value approach: Reject H 0 if t t 

35. 35 Slide t Test for Significance of Individual Parameters Hypotheses Rejection Rule For  =.05 and d.f. = 17, t.025 = 2.11 d.f. = n-p-1 = 20-2-1 = 17 p-value approach: Reject H 0 if p -value <  Critical value approach: Reject H 0 if t t  Critical values are t.025 = TINV(0.025*2,17)=2.11 and - t.025 =-2.11

36. 36 Slide n Excel’s Regression Equation Output Note: Columns F-I are not shown. t Test for Significance of Individual Parameters t statistic and p -value used to test for the individual significance of “Experience” p -value for Experience = TDIST(7.07,17,2)=1.9E-06

37. 37 Slide n Excel’s Regression Equation Output Note: Columns F-I are not shown. t Test for Significance of Individual Parameters t statistic and p -value used to test for the individual significance of “Test Score” p -value for Test Score = TDIST(3.2433,17,2)=0.00478

38. 38 Slide t Test for Significance of Individual Parameters Test Statistics Conclusions Reject both H 0 :  1 = 0 and H 0 :  2 = 0. Both independent variables are significant. p -values p-value 1= 1.9E-06 p-value 2= 0.00478 critical values t.025 = 2.11 -t.025 = -2.11

39. 39 Slide In many situations we must work with qualitative In many situations we must work with qualitative independent variables such as gender (male, female), independent variables such as gender (male, female), method of payment (cash, check, credit card), etc. method of payment (cash, check, credit card), etc. In many situations we must work with qualitative In many situations we must work with qualitative independent variables such as gender (male, female), independent variables such as gender (male, female), method of payment (cash, check, credit card), etc. method of payment (cash, check, credit card), etc. For example, x 2 might represent gender where x 2 = 0 For example, x 2 might represent gender where x 2 = 0 indicates male and x 2 = 1 indicates female. indicates male and x 2 = 1 indicates female. For example, x 2 might represent gender where x 2 = 0 For example, x 2 might represent gender where x 2 = 0 indicates male and x 2 = 1 indicates female. indicates male and x 2 = 1 indicates female. Qualitative Independent Variables In this case, x 2 is called a dummy or indicator variable. In this case, x 2 is called a dummy or indicator variable. For a qualitative independent variable with k For a qualitative independent variable with k categories, we create k-1 dummy variables. For a qualitative independent variable with k For a qualitative independent variable with k categories, we create k-1 dummy variables.

40. 40 Slide As an extension of the problem involving the As an extension of the problem involving the computer programmer salary survey, suppose that management also believes that the annual salary is related to whether the individual has a graduate degree in computer science or information systems. The years of experience, the score on the programmer The years of experience, the score on the programmer aptitude test, whether the individual has a relevant graduate degree, and the annual salary ($1000) for each of the sampled 20 programmers are shown on the next slide. Qualitative Independent Variables n Example: Programmer Salary Survey

41. 41 Slide 471581001669210568463378100868286847580839188737581748779947089 244323.734.335.83822.223.13033 3826.636.231.6293430.133.928.230 Exper.ScoreScoreExper.SalarySalaryDegr. No NoYes YesYesYes Yes Degr. Yes Yes No NoYes Yes Yes Qualitative Independent Variables

42. 42 Slide Estimated Regression Equation y = b 0 + b 1 x 1 + b 2 x 2 + b 3 x 3 ^where: y = annual salary ($1000) y = annual salary ($1000) x 1 = years of experience x 1 = years of experience x 2 = score on programmer aptitude test x 2 = score on programmer aptitude test x 3 = 0 if individual does not have a graduate degree x 3 = 0 if individual does not have a graduate degree 1 if individual does have a graduate degree 1 if individual does have a graduate degree x 3 is a dummy variable Use ProgramerSalary.xlsx (Qualitative IV-2Cat tab)

43. 43 Slide n Excel Formula Worksheet (showing data) Note: Rows 10-21 are not shown. Estimated Regression Equation

44. 44 Slide Estimated Regression Equation significant not significant

45. 45 Slide n Excel’s ANOVA Output Categorical Independent Variables ABCDEF 32 33ANOVA 34dfSSMSFSignificance F 35Regression3507.8960169.298729.47879.41675E-07 36Residual1691.88955.7431 37Total19599.7855 R 2 = 507.896/599.7855 =.8468

46. 46 Slide n Excel’s Regression Statistics Categorical Independent Variables Previously, R Square =.8342 Previously, Adjusted R Square =.815 (essentially no improvement)

47. 47 Slide For example, a variable indicating level of education could be represented by x 1 and x 2 values as follows: For example, a variable indicating level of education could be represented by x 1 and x 2 values as follows: More Complex Qualitative Variables Highest Degree x 1 x 2 Bachelor’s00 Master’s10 Ph.D.01 Use ProgramerSalary.xlsx (Qualitative IV-3Cat tab)

48. 48 Slide More Complex Qualitative Variables Use ProgramerSalary.xlsx (Qualitative IV-3Cat tab) significant not significant

49. 49 Slide Residual Analysis n For simple linear regression the residual plot against and the residual plot against x provide the same information. and the residual plot against x provide the same information. n In multiple regression analysis it is preferable to use the residual plot against to determine if the model assumptions are satisfied.

50. 50 Slide Standardized Residual Plot Against n Standardized residuals are frequently used in residual plots for purposes of: Identifying outliers (typically, standardized residuals +2) Identifying outliers (typically, standardized residuals +2) Providing insight about the assumption that the error term  has a normal distribution Providing insight about the assumption that the error term  has a normal distribution n The computation of the standardized residuals in multiple regression analysis is too complex to be done by hand n Excel’s Regression tool can be used

51. 51 Slide n Excel Value Worksheet Note: Rows 37-51 are not shown. Standardized Residual Plot Against standardized residual > 2  outlier Remember to check the standardized residual box in Regression tool dialog box

52. 52 Slide Standardized Residual Plot Against n Excel’s Standardized Residual Plot Outlier