12a - 1 © 2000 Prentice-Hall, Inc. Statistics Multiple Regression and Model Building Chapter 12 part I
12a - 2 © 2000 Prentice-Hall, Inc. Learning Objectives 1.Explain the Linear Multiple Regression Model 2.Explain Residual Analysis 3.Test Overall Significance 4.Explain Multicollinearity 5.Interpret Linear Multiple Regression Computer Output
12a - 3 © 2000 Prentice-Hall, Inc. Types of Regression Models
12a - 4 © 2000 Prentice-Hall, Inc. Regression Modeling Steps 1.Hypothesize Deterministic Component 2.Estimate Unknown Model Parameters 3.Specify Probability Distribution of Random Error Term Estimate Standard Deviation of Error Estimate Standard Deviation of Error 4.Evaluate Model 5.Use Model for Prediction & Estimation
12a - 5 © 2000 Prentice-Hall, Inc. Regression Modeling Steps 1.Hypothesize Deterministic Component 2.Estimate Unknown Model Parameters 3.Specify Probability Distribution of Random Error Term Estimate Standard Deviation of Error Estimate Standard Deviation of Error 4.Evaluate Model 5.Use Model for Prediction & Estimation Expanded in Multiple Regression
12a - 6 © 2000 Prentice-Hall, Inc. Linear Multiple Regression Model Hypothesizing the Deterministic Component Expanded in Multiple Regression
12a - 7 © 2000 Prentice-Hall, Inc. Regression Modeling Steps 1.Hypothesize Deterministic Component 2.Estimate Unknown Model Parameters 3.Specify Probability Distribution of Random Error Term Estimate Standard Deviation of Error Estimate Standard Deviation of Error 4.Evaluate Model 5.Use Model for Prediction & Estimation
12a - 8 © 2000 Prentice-Hall, Inc. Linear Multiple Regression Model 1.Relationship between 1 dependent & 2 or more independent variables is a linear function Dependent (response) variable Independent (explanatory) variables Population slopes Population Y-intercept Random error
12a - 9 © 2000 Prentice-Hall, Inc. Population Multiple Regression Model Bivariate model
12a - 10 © 2000 Prentice-Hall, Inc. Sample Multiple Regression Model Bivariate model
12a - 11 © 2000 Prentice-Hall, Inc. Parameter Estimation Expanded in Multiple Regression
12a - 12 © 2000 Prentice-Hall, Inc. Regression Modeling Steps 1.Hypothesize Deterministic Component 2.Estimate Unknown Model Parameters 3.Specify Probability Distribution of Random Error Term Estimate Standard Deviation of Error Estimate Standard Deviation of Error 4.Evaluate Model 5.Use Model for Prediction & Estimation
12a - 13 © 2000 Prentice-Hall, Inc. Multiple Linear Regression Equations Too complicated by hand! Ouch!
12a - 14 © 2000 Prentice-Hall, Inc. Interpretation of Estimated Coefficients
12a - 15 © 2000 Prentice-Hall, Inc. Interpretation of Estimated Coefficients 1.Slope ( k ) Estimated Y Changes by k for Each 1 Unit Increase in X k Holding All Other Variables Constant Estimated Y Changes by k for Each 1 Unit Increase in X k Holding All Other Variables Constant Example: If 1 = 2, then Sales (Y) Is Expected to Increase by 2 for Each 1 Unit Increase in Advertising (X 1 ) Given the Number of Sales Rep’s (X 2 ) Example: If 1 = 2, then Sales (Y) Is Expected to Increase by 2 for Each 1 Unit Increase in Advertising (X 1 ) Given the Number of Sales Rep’s (X 2 ) ^ ^ ^
12a - 16 © 2000 Prentice-Hall, Inc. Interpretation of Estimated Coefficients 1.Slope ( k ) Estimated Y Changes by k for Each 1 Unit Increase in X k Holding All Other Variables Constant Estimated Y Changes by k for Each 1 Unit Increase in X k Holding All Other Variables Constant Example: If 1 = 2, then Sales (Y) Is Expected to Increase by 2 for Each 1 Unit Increase in Advertising (X 1 ) Given the Number of Sales Rep’s (X 2 ) Example: If 1 = 2, then Sales (Y) Is Expected to Increase by 2 for Each 1 Unit Increase in Advertising (X 1 ) Given the Number of Sales Rep’s (X 2 ) 2.Y-Intercept ( 0 ) Average Value of Y When X k = 0 Average Value of Y When X k = 0 ^ ^ ^ ^
12a - 17 © 2000 Prentice-Hall, Inc. Parameter Estimation Example You work in advertising for the New York Times. You want to find the effect of ad size (sq. in.) & newspaper circulation (000) on the number of ad responses (00). You’ve collected the following data: RespSizeCirc
12a - 18 © 2000 Prentice-Hall, Inc. Parameter Estimation Computer Output Parameter Estimates Parameter Estimates Parameter Standard T for H0: Parameter Standard T for H0: Variable DF Estimate Error Param=0 Prob>|T| INTERCEP ADSIZE CIRC PP 22 00 11 ^ ^ ^ ^
12a - 19 © 2000 Prentice-Hall, Inc. Interpretation of Coefficients Solution
12a - 20 © 2000 Prentice-Hall, Inc. Interpretation of Coefficients Solution 1.Slope ( 1 ) # Responses to Ad Is Expected to Increase by.2049 (20.49) for Each 1 Sq. In. Increase in Ad Size Holding Circulation Constant # Responses to Ad Is Expected to Increase by.2049 (20.49) for Each 1 Sq. In. Increase in Ad Size Holding Circulation Constant ^
12a - 21 © 2000 Prentice-Hall, Inc. Interpretation of Coefficients Solution 1.Slope ( 1 ) # Responses to Ad Is Expected to Increase by.2049 (20.49) for Each 1 Sq. In. Increase in Ad Size Holding Circulation Constant # Responses to Ad Is Expected to Increase by.2049 (20.49) for Each 1 Sq. In. Increase in Ad Size Holding Circulation Constant 2.Slope ( 2 ) # Responses to Ad Is Expected to Increase by.2805 (28.05) for Each 1 Unit (1,000) Increase in Circulation Holding Ad Size Constant # Responses to Ad Is Expected to Increase by.2805 (28.05) for Each 1 Unit (1,000) Increase in Circulation Holding Ad Size Constant ^ ^
12a - 22 © 2000 Prentice-Hall, Inc. Evaluating the Model Expanded in Multiple Regression
12a - 23 © 2000 Prentice-Hall, Inc. Regression Modeling Steps 1.Hypothesize Deterministic Component 2.Estimate Unknown Model Parameters 3.Specify Probability Distribution of Random Error Term Estimate Standard Deviation of Error Estimate Standard Deviation of Error 4.Evaluate Model 5.Use Model for Prediction & Estimation
12a - 24 © 2000 Prentice-Hall, Inc. Evaluating Multiple Regression Model Steps 1.Examine Variation Measures 2.Do Residual Analysis 3.Test Parameter Significance Overall Model Overall Model Individual Coefficients Individual Coefficients 4.Test for Multicollinearity
12a - 25 © 2000 Prentice-Hall, Inc. Evaluating Multiple Regression Model Steps 1.Examine Variation Measures 2.Do Residual Analysis 3.Test Parameter Significance Overall Model Overall Model Individual Coefficients Individual Coefficients 4.Test for Multicollinearity New! New! Expanded! New!
12a - 26 © 2000 Prentice-Hall, Inc. Variation Measures
12a - 27 © 2000 Prentice-Hall, Inc. Evaluating Multiple Regression Model Steps 1.Examine Variation Measures 2.Do Residual Analysis 3.Test Parameter Significance Overall Model Overall Model Individual Coefficients Individual Coefficients 4.Test for Multicollinearity New! New! Expanded! New!
12a - 28 © 2000 Prentice-Hall, Inc. Coefficient of Multiple Determination 1.Proportion of Variation in Y ‘Explained’ by All X Variables Taken Together R 2 = Explained Variation = SSR Total Variation SS yy 2.Never Decreases When New X Variable Is Added to Model Only Y Values Determine SS yy Only Y Values Determine SS yy Disadvantage When Comparing Models Disadvantage When Comparing Models
12a - 29 © 2000 Prentice-Hall, Inc. Residual Analysis
12a - 30 © 2000 Prentice-Hall, Inc. Evaluating Multiple Regression Model Steps 1.Examine Variation Measures 2.Do Residual Analysis 3.Test Parameter Significance Overall Model Overall Model Individual Coefficients Individual Coefficients 4.Test for Multicollinearity New! New! Expanded! New!
12a - 31 © 2000 Prentice-Hall, Inc. Residual Analysis 1.Graphical Analysis of Residuals Plot Estimated Errors vs. X i Values Plot Estimated Errors vs. X i Values Difference Between Actual Y i & Predicted Y i Difference Between Actual Y i & Predicted Y i Estimated Errors Are Called Residuals Estimated Errors Are Called Residuals Plot Histogram or Stem-&-Leaf of Residuals Plot Histogram or Stem-&-Leaf of Residuals 2.Purposes Examine Functional Form (Linear vs. Non-Linear Model) Examine Functional Form (Linear vs. Non-Linear Model) Evaluate Violations of Assumptions Evaluate Violations of Assumptions
12a - 32 © 2000 Prentice-Hall, Inc. Linear Regression Assumptions 1.Mean of Probability Distribution of Error Is 0 2.Probability Distribution of Error Has Constant Variance 3.Probability Distribution of Error is Normal 4.Errors Are Independent
12a - 33 © 2000 Prentice-Hall, Inc. Residual Plot for Functional Form Add X 2 Term Correct Specification ^ ^
12a - 34 © 2000 Prentice-Hall, Inc. Residual Plot for Equal Variance Unequal Variance Correct Specification Fan-shaped. Standardized residuals used typically.
12a - 35 © 2000 Prentice-Hall, Inc. Residual Plot for Independence Not Independent Correct Specification Plots reflect sequence data were collected.
12a - 36 © 2000 Prentice-Hall, Inc. Residual Analysis Computer Output Dep Var Predict Student Dep Var Predict Student Obs SALES Value Residual Residual | |** | | *| | | | | | **| | | |*** | Plot of standardized (student) residuals
12a - 37 © 2000 Prentice-Hall, Inc. Testing Parameters
12a - 38 © 2000 Prentice-Hall, Inc. Evaluating Multiple Regression Model Steps 1.Examine Variation Measures 2.Do Residual Analysis 3.Test Parameter Significance Overall Model Overall Model Individual Coefficients Individual Coefficients 4.Test for Multicollinearity New! New! Expanded! New!
12a - 39 © 2000 Prentice-Hall, Inc. Testing Overall Significance 1.Shows If There Is a Linear Relationship Between All X Variables Together & Y 2.Uses F Test Statistic 3.Hypotheses H 0 : 1 = 2 =... = k = 0 H 0 : 1 = 2 =... = k = 0 No Linear Relationship No Linear Relationship H a : At Least One Coefficient Is Not 0 H a : At Least One Coefficient Is Not 0 At Least One X Variable Affects Y At Least One X Variable Affects Y
12a - 40 © 2000 Prentice-Hall, Inc. Testing Overall Significance Computer Output Analysis of Variance Sum of Mean Sum of Mean Source DF Squares Square F Value Prob>F Model Error C Total kn - k -1 n - 1P-Value MS(Model) MS(Error)
12a - 41 © 2000 Prentice-Hall, Inc. Multicollinearity
12a - 42 © 2000 Prentice-Hall, Inc. Evaluating Multiple Regression Model Steps 1.Examine Variation Measures 2.Do Residual Analysis 3.Test Parameter Significance Overall Model Overall Model Individual Coefficients Individual Coefficients 4.Test for Multicollinearity New! New! Expanded! New!
12a - 43 © 2000 Prentice-Hall, Inc. Multicollinearity 1.High Correlation Between X Variables 2.Coefficients Measure Combined Effect 3.Leads to Unstable Coefficients Depending on X Variables in Model 4.Always Exists -- Matter of Degree 5.Example: Using Both Age & Height as Explanatory Variables in Same Model
12a - 44 © 2000 Prentice-Hall, Inc. Detecting Multicollinearity 1.Examine Correlation Matrix Correlations Between Pairs of X Variables Are More than With Y Variable Correlations Between Pairs of X Variables Are More than With Y Variable 2.Examine Variance Inflation Factor (VIF) If VIF j > 5, Multicollinearity Exists If VIF j > 5, Multicollinearity Exists 3.Few Remedies Obtain New Sample Data Obtain New Sample Data Eliminate One Correlated X Variable Eliminate One Correlated X Variable
12a - 45 © 2000 Prentice-Hall, Inc. Correlation Matrix Computer Output Correlation Analysis Pearson Corr Coeff /Prob>|R| under HO:Rho=0/ N=6 RESPONSE ADSIZE CIRC RESPONSE ADSIZE CIRC RESPONSE ADSIZE CIRC r Y1 r Y2 All 1’s r 12
12a - 46 © 2000 Prentice-Hall, Inc. Variance Inflation Factors Computer Output Parameter Standard T for H0: Parameter Standard T for H0: Variable DF Estimate Error Param=0 Prob>|T| INTERCEP ADSIZE CIRC Variance Variance Variable DF Inflation INTERCEP ADSIZE CIRC VIF 1 5
12a - 47 © 2000 Prentice-Hall, Inc. Regression Cautions
12a - 48 © 2000 Prentice-Hall, Inc. Regression Cautions 1.Violated Assumptions 2.Relevancy of Historical Data 3.Level of Significance 4.Extrapolation 5.Cause & Effect
12a - 49 © 2000 Prentice-Hall, Inc. Y Interpolation X ExtrapolationExtrapolation Relevant Range Extrapolation
12a - 50 © 2000 Prentice-Hall, Inc. Cause & Effect Liquor Consumption # Teachers
12a - 51 © 2000 Prentice-Hall, Inc. Conclusion 1.Explained the Linear Multiple Regression Model 2.Explained Residual Analysis 3.Tested Overall Significance 4.Explained Multicollinearity 5.Interpreted Linear Multiple Regression Computer Output
End of Chapter Any blank slides that follow are blank intentionally.