Presentation on theme: "4/14/2015330 lecture 81 STATS 330: Lecture 8. 4/14/2015330 lecture 82 Collinearity Aims of today’s lecture: Explain the idea of collinearity and its connection."— Presentation transcript:
4/14/ lecture 81 STATS 330: Lecture 8
4/14/ lecture 82 Collinearity Aims of today’s lecture: Explain the idea of collinearity and its connection with estimating regression coefficients To discuss added variable plots, a graphical method for deciding if a variable should be added to a regression
4/14/ lecture 83 Variance of regression coefficients We saw in Lecture 6 how the standard errors of the regression coefficients depend on the error variance 2 : the bigger 2, the bigger the standard errors. We also suggested that the standard error depends on the arrangement of the x’s. In today’s lecture, we explore this idea a bit further.
4/14/ lecture 84 Example Suppose we have a regression relationship of the form Y=1 + 2x –w + between a response variable Y and two explanatory variables x and w. Consider two data sets, A and B, each following the model above.
4/14/ lecture 85 Data sets A & B: x,w data
4/14/ lecture 86
4/14/ lecture 87 Conclusion: The greater the correlation, the more variable the plane. In fact, for the coefficient of x
4/14/ lecture 88 Generalization If we have k explanatory variables, then the variance of the jth estimated coefficient is where R j 2 is the R 2 if we regress variable j on the other explanatory variables
4/14/ lecture 89 Best case If x j is orthogonal to (uncorrelated with) the other explanatory variables, then R j 2 is zero and the variance is the smallest possible i.e.
4/14/ lecture 810 Variance inflation factor The factor represents the increase in variance caused by correlation between the explanatory variables and is called the variance inflation factor (VIF)
4/14/ lecture 811 Calculating the VIF: theory To calculate the VIF for the jth explanatory variable, use the relationship using the residuals from regressing the jth explanatory variable on the other explanatory variables
4/14/ lecture 812 Calculating the VIF: example For the petrol data, calculate the VIF for t.vp (tank vapour pressure) > attach(vapour.df) > tvp.reg <- lm(t.vp~t.temp + p.temp + p.vp,data=vapour.df) > var(t.vp)/var(residuals(tvp.reg))  Correlation increases variance by a factor of 66
4/14/ lecture 813 Calculating the VIF: quick method A useful mathematical relationship: Suppose we calculate the inverse of the correlation matrix of the explanatory variables. Then the VIF’s are the diagonal elements. > X<-vapour.df[,-5] # delete 5 th column # (hc, the response) > VIF <- diag(solve(cor(X))) > VIF t.temp p.temp t.vp p.vp
4/14/ lecture 814 Pairs plot
4/14/ lecture 815 Collinearity If one or more variables in a regression have big VIF’s, the regression is said to be collinear Caused by one or more variables being almost linear combinations of the others Sometimes indicated by high correlations between the independent variables Results in imprecise estimation of regression coefficients Standard errors are high, so t-statistics are small, variables are often non-significant ( data is insufficient to detect a difference)
4/14/ lecture 816 Non-significance If a variable has a non-significant t, then either The variable is not related to the response, or The variable is related to the response, but it is not required in the regression because it is strongly related to a third variable that is in the regression, so we don’t need both. First case: small t-value, small VIF, small correlation with response Second case, small t-value, big VIF, big correlation with response
4/14/ lecture 817 Remedy The usual remedy is to drop one or more variables from the model. This “breaks” the linear relationship between the variables This leads to the problem of “subset selection”, which subset to choose. See Lectures 14 and 15
4/14/ lecture 818 Example: Cement data Measurements on batches of cement Response variable: Heat (heat emitted) Explanatory variables X1: amount of tricalcium aluminate (%) X2: amount of tricalcium silicate (%) X3: amount of tetracalcium aluminaoferrite (%) X4: amount of dicalcium silicate (%)
4/14/ lecture 819 Example: Cement data Heat X1X2X3X
4/14/ lecture 820 Example: Cement data Estimate Std. Error t value Pr(>|t|) (Intercept) X X X X Residual standard error: on 8 degrees of freedom Multiple R-Squared: , Adjusted R-squared: F-statistic: on 4 and 8 DF, p-value: 4.756e-07 > round(cor(cement.df),2) Heat X1 X2 X3 X4 Heat X X X X Big correlation Large p- values Big R- squared
4/14/ lecture 821 Cement data > diag(solve(cor(cement.df[, -1]))) X1 X2 X3 X cement.df$X1 + cement.df$X2 + cement.df$X3 + cement.df$X4  Omit Heat Very large!
4/14/ lecture 822 Drop X4 > diag(solve(cor(cement.df[, -c(1,5)]))) X1 X2 X Estimate Std. Error t value Pr(>|t|) (Intercept) e-07 *** X e-05 *** X e-07 *** X Residual standard error: on 9 degrees of freedom Multiple R-Squared: , Adjusted R-squared: F-statistic: on 3 and 9 DF, p-value: 3.367e-08 VIF’s now small X1, X2 now signif
4/14/ lecture 823 Added variable plots (AVP’s) To see if a variable, say x, is needed in a regression: Step 1: Calculate the residuals from regressing the response on all the explanatory variables except x Step 2: calculate the residuals from regressing x on the other explanatory variables Step 3: Plot the first set of residuals versus the second set NB: Also called partial regression plots in some books
4/14/ lecture 824 Rationale The first set of residuals represents the variation in y not explained by the other explanatory variables The second set of residuals represents the part of x not explained by the other explanatory variables If there is a relationship between the two sets, there is a relationship between x and the response that is not accounted for by the other explanatory variables Thus, if we see a relationship in the plot, x is needed in the regression!!!
4/14/ lecture 825 Example: the petrol data Let’s do an AVP for tank vapour pressure, t.vp. > rest.reg<- lm(hc~t.temp + p.temp + p.vp,data=vapour.df) > y.res<-residuals(rest.reg) > tvp.reg<-lm(t.vp~t.temp + p.temp + p.vp,data=vapour.df) > tvp.res<-residuals(tvp.reg) > plot(tvp.res,y.res, xlab = "Tank vapour pressure", ylab="Hydrocarbon emission", main = “AVP for Tank vapour pressure")
4/14/ lecture 826 Hint of a relationship: so variable required?
4/14/ lecture 827 Short cut in R There is a function added.variable.plots in R to draw the plots automatically. This is one of the functions in the R330 package which must be installed before the function can be used. > vapour.lm<-lm(hc~.,data=vapour.df) > par(mfrow=c(2,2)) # 2 x 2 array of plots > added.variable.plots(vapour.lm) Note this useful trick!
4/14/ lecture 828 Not significant in the regression (Lect 7)
4/14/ lecture 829 Some curious facts about AVP’s Assuming a constant term in both regressions, a least squares line fitted though the AVP goes through the origin. The slope of this line is the fitted regression coefficient for the variable in the original regression The residuals from this line are the same as the residuals from the original regression