Chapter 11 Variable Selection Procedures Lecture 19 Chapter 11 Variable Selection Procedures The Milk Production Data:1 response and 6 predictor variables Regression Analysis: CurrentMilk versus Previous, Fat, ... The regression equation is CurrentMilk = 51.0 + 0.698 Previous + 0.798 Fat - 6.42 Protein - 0.0317 Days + 0.527 Lactation - 10.3 I79 Predictor Coef SE Coef T P Constant 51.030 7.303 6.99 0.000 Previous 0.69846 0.05203 13.42 0.000 Fat 0.7979 0.9524 0.84 0.403 Protein -6.416 1.609 -3.99 0.000 Days -0.03165 0.01548 -2.04 0.042 Lactatio 0.5268 0.5488 0.96 0.338 I79 -10.298 2.847 -3.62 0.000 S = 10.67 R-Sq = 66.4% R-Sq(adj) = 65.3% 4/29/2019 ST3131, Lecture 19

Results from the comparison of the above two models: Regression Analysis: CurrentMilk versus Previous, Protein, Days, I79 The regression equation is CurrentMilk = 52.7 + 0.704 Previous - 5.62 Protein - 0.0318 Days - 10.8 I79 Predictor Coef SE Coef T P Constant 52.748 6.983 7.55 0.000 Previous 0.70353 0.05099 13.80 0.000 Protein -5.619 1.468 -3.83 0.000 Days -0.03176 0.01531 -2.07 0.039 I79 -10.778 2.778 -3.88 0.000 S = 10.66 R-Sq = 66.1% R-Sq(adj) = 65.4% Results from the comparison of the above two models: 1). Model Complexity: model 1 (Full) is than model 2 (Reduced) 2). Goodness of Fit: model 1 is than model 2 4/29/2019 ST3131, Lecture 19

3). Parameter Estimates: 51.030 .69846 .7979 -6.416 -.03165 .5268 -10.298 52.748 .70353 -5.619 -.03176 -10.778 4). SE of Parameter Estimates: 7.303 .05203 .9524 1.609 .01548 .5488 2.847 6.983 .05099 1.468 .01531 2.778 4/29/2019 ST3131, Lecture 19

Questions Formulation of the Problem Full Model: all q predictor variables of interest are included Reduced Model: some of predictor variables are deleted Two Possible Cases for Deleted Predictor Variables: Case 1 All deleted coefficients are zero Case 2 Some of deleted coefficients are not zero Questions What are the effects of including variables in an equation when they should be properly left out since they are zero ( in Case 1)? What are the effects of leaving out variables when they should be included since they are not zero (in Case 2)? 4/29/2019 ST3131, Lecture 19

Consequences of Variables Deletion 1). The Reduced Model is less complicated. 2). The variances of the estimates of the retained parameters (and the fitted values) are smaller. 3). The estimates of the retained parameters (and the fitted values) are biased when some or all deleted coefficients are not zero (case 2). The general PURPOSE of variable selection is to reach a final reduced model via introducing significant variables and/or deleting some insignificant variables so that the precision (measured by MSE or other criteria) of the estimates of the retained parameters, the fitted values, or the prediction values, is improved. 4/29/2019 ST3131, Lecture 19

Mean Squared Error (MSE) The expectation of the squared difference between an estimator and its true/underlying value is called the MSE of the estimator: MSE=Variance+Bias^2 MSE is a measure of the precision of an estimator apart from its true value. The parameter estimates and the fitted values in the reduced model have smaller variances but larger bias. A tradeoff between the variance and squared bias allows the MSE to be minimized. Minimization of the MSE results in the “best” reduced model according to the MSE criterion. The parameter estimates and the fitted values in the Full model are unbiased but their variances are largest and hence in general may not be the best model unless all coefficients are significant from 0. 4/29/2019 ST3131, Lecture 19

General Criteria for Variable Selections: To judge the adequacy of various fitted equations, we need some criteria. We introduce some important and useful criteria here. 1). Residual Mean Square/ Noise Variance Estimator RMS = =SSE /(n-p-1) SSE = : represents Goodness of Fits/Bias decreases with increasing p 1/(n-p-1) :represents Model Complexity/Variance increases with increasing p For some 0<p<=q, RMS is minimized. 4/29/2019 ST3131, Lecture 19

2). Coefficient of Determination R R =1-SSE /SST=1-(n-p-1) RMS /SST R represents only the Goodness of Fits/Bias increases with increasing p 3) Adjusted Coefficient of Determination R R =1-(n-1) RMS /SST R is just a linear function of RMS and hence takes both the Goodness of Fit (Bias) and the Model Complexity (Variance) into account. That is why in Chapter 3, we suggest to use the adjusted coefficient of determination as a criterion to select different model instead of the coefficient of determination itself. 4/29/2019 ST3131, Lecture 19

Which can be estimated by the Mallows Cp statistic Cp=SSE / +(2p+2-n) To predict the future response, we want to make the prediction MSE be minimized: Jp= (y -y ) = MSE Which can be estimated by the Mallows Cp statistic Cp=SSE / +(2p+2-n) SSE : represents Goodness of Fits/Bias , decreases with increasing p (2p+2-n) :represents Model Complexity/Variance, increases with increasing p E(Cp)=p, the best reduced model is selected with smallest |Cp-p| over 0<p<=q 4/29/2019 ST3131, Lecture 19

Best Subsets Regression: CurrentMilk versus Previous, Fat, ... Response is CurrentM Vars R-Sq R-Sq(adj) C-p S s t n s o 9 1 55.4 55.2 59.5 12.132 X 1 21.9 21.5 251.0 16.061 X 1 12.6 12.2 304.2 16.991 X 1 7.8 7.4 331.4 17.448 X 1 1.5 1.0 367.4 18.035 X 2 62.7 62.3 20.3 11.134 X X 2 61.4 61.0 27.4 11.319 X X 2 57.2 56.8 51.3 11.918 X X 2 55.8 55.3 59.7 12.119 X X 2 55.4 55.0 61.5 12.163 X X 3 65.3 64.8 7.0 10.755 X X X 3 63.5 63.0 17.3 11.032 X X X 3 63.5 62.9 17.7 11.042 X X X 3 62.7 62.1 22.1 11.158 X X X 3 62.7 62.1 22.3 11.162 X X X 4 66.1 65.4 4.7 10.665 X X X X 4 65.5 64.8 7.8 10.750 X X X X 4 65.5 64.7 8.3 10.765 X X X X 4 64.0 63.3 16.5 10.986 X X X X 4 63.6 62.8 18.9 11.051 X X X X 5 66.3 65.4 5.7 10.665 X X X X X 5 66.2 65.4 5.9 10.671 X X X X X 5 65.7 64.8 9.2 10.761 X X X X X 5 64.1 63.2 18.1 11.003 X X X X X 5 63.6 62.7 20.9 11.078 X X X X X 6 66.4 65.3 7.0 10.674 X X X X X X 4/29/2019 ST3131, Lecture 19

Model Building for Various Uses of Regression Equations Regression equations has many uses, including Process Description, Estimation, Prediction, and Control. For different uses, the focus of model building may be different. 1). Process Description When a regression equation is used to describe a given process, there are two conflicting requirements: (a). To account for as much of the variation as possible (thus should as many of variables as possible) (b). To make the model as simple as possible (thus should as many of variables as possible) Model Building: choose the smallest number of predictor variables that accounts for the most substantial part of the variation in the response variable. 4/29/2019 ST3131, Lecture 19

2). Estimation and Prediction When an regression equation is constructed for prediction at a future observation, the purpose is to make the prediction as accurate as possible. Model Building: select the variables so that the MSE of prediction is minimized. 4/29/2019 ST3131, Lecture 19

3). Control The purpose for constructing an regression equation may be to determine the magnitude by which the value of a predictor variable must be altered to obtain a specified value of the response (target) variable. For this control purpose, it is desired that the coefficients of the variables in the equation be measured accurately. Model Building: select variables so that the standard errors of the regression coefficients are small. 4/29/2019 ST3131, Lecture 19

Remarks: 1). A regression equation may be constructed for several uses of regression equation since these uses can overlap with each other. 2). The use of the equation will in general determine the criterion that is to be optimized in its formulation. 3). A model which is best for one criterion may not be best for another. 4). Even for a same criterion, the best model may not be unique. 5).When there are several models that are adequate and could be used in forming an equation, we should list them out and make some further examination. 4/29/2019 ST3131, Lecture 19