1. Chapter 11 Variable Selection Procedures
Lecture 19
Chapter 11 Variable Selection Procedures
The Milk Production Data:1 response and 6 predictor variables
Regression Analysis: CurrentMilk versus Previous, Fat, ...
The regression equation is
CurrentMilk = Previous Fat Protein Days
Lactation I79
Predictor Coef SE Coef T P
Constant
Previous
Fat
Protein
Days
Lactatio I
S = R-Sq = 66.4% R-Sq(adj) = 65.3%
4/29/2019
ST3131, Lecture 19

2. Results from the comparison of the above two models:
Regression Analysis: CurrentMilk versus Previous, Protein, Days, I79
The regression equation is
CurrentMilk = Previous Protein Days I79
Predictor Coef SE Coef T P
Constant
Previous
Protein
Days I
S = R-Sq = 66.1% R-Sq(adj) = 65.4%
Results from the comparison of the above two models:
1). Model Complexity: model 1 (Full) is than model 2 (Reduced)
2). Goodness of Fit: model 1 is than model 2
4/29/2019
ST3131, Lecture 19

3. 3). Parameter Estimates:
51.030
.69846
.7979
-6.416
.5268
52.748
.70353
-5.619
4). SE of Parameter Estimates:
7.303
.05203
.9524
1.609
.01548
.5488
2.847
6.983
.05099
1.468
.01531
2.778
4/29/2019
ST3131, Lecture 19

4. Questions Formulation of the Problem
Full Model: all q predictor variables of interest are included
Reduced Model: some of predictor variables are deleted
Two Possible Cases for Deleted Predictor Variables:
Case 1 All deleted coefficients are zero
Case 2 Some of deleted coefficients are not zero
Questions
What are the effects of including variables in an equation when they should be properly left out since they are zero ( in Case 1)?
What are the effects of leaving out variables when they should be included since they are not zero (in Case 2)?
4/29/2019
ST3131, Lecture 19

5. Consequences of Variables Deletion
1). The Reduced Model is less complicated.
2). The variances of the estimates of the retained parameters (and the fitted values) are smaller.
3). The estimates of the retained parameters (and the fitted values) are biased when some or all deleted coefficients are not zero (case 2).
The general PURPOSE of variable selection is to reach a final reduced model via introducing significant variables and/or deleting some insignificant variables so that the precision (measured by MSE or other criteria) of the estimates of the retained parameters, the fitted values, or the prediction values, is improved.
4/29/2019
ST3131, Lecture 19

6. Mean Squared Error (MSE) The expectation of the squared difference between an estimator and its true/underlying value is called the MSE of the estimator:
MSE=Variance+Bias^2
MSE is a measure of the precision of an estimator apart from its true value.
The parameter estimates and the fitted values in the reduced model have smaller variances but larger bias. A tradeoff between the variance and squared bias allows the MSE to be minimized. Minimization of the MSE results in the “best” reduced model according to the MSE criterion.
The parameter estimates and the fitted values in the Full model are unbiased but their variances are largest and hence in general may not be the best model unless all coefficients are significant from 0.
4/29/2019
ST3131, Lecture 19

7. General Criteria for Variable Selections:
To judge the adequacy of various fitted equations, we need some criteria. We introduce some important and useful criteria here.
1). Residual Mean Square/ Noise Variance Estimator
RMS = =SSE /(n-p-1)
SSE = : represents Goodness of Fits/Bias
decreases with increasing p
1/(n-p-1) :represents Model Complexity/Variance
increases with increasing p
For some 0<p<=q, RMS is minimized.
4/29/2019
ST3131, Lecture 19

8. 2). Coefficient of Determination R
R =1-SSE /SST=1-(n-p-1) RMS /SST
R represents only the Goodness of Fits/Bias
increases with increasing p
3) Adjusted Coefficient of Determination R
R =1-(n-1) RMS /SST
R is just a linear function of RMS and hence takes both the Goodness of Fit (Bias) and the Model Complexity (Variance) into account. That is why in Chapter 3, we suggest to use the adjusted coefficient of determination as a criterion to select different model instead of the coefficient of determination itself.
4/29/2019
ST3131, Lecture 19

9. Which can be estimated by the Mallows Cp statistic Cp=SSE / +(2p+2-n)
To predict the future response, we want to make the prediction MSE be minimized:
Jp= (y -y ) = MSE
Which can be estimated by the Mallows Cp statistic
Cp=SSE / (2p+2-n)
SSE : represents Goodness of Fits/Bias , decreases with increasing p
(2p+2-n) :represents Model Complexity/Variance, increases with increasing p
E(Cp)=p, the best reduced model is selected with smallest |Cp-p| over 0<p<=q
4/29/2019
ST3131, Lecture 19

10. Best Subsets Regression: CurrentMilk versus Previous, Fat, ...
Response is CurrentM
Vars R-Sq R-Sq(adj) C-p S s t n s o 9 X X X X X
X X
X X
X X
X X
X X
X X X
X X X
X X X
X X X
X X X
X X X X
X X X X
X X X X
X X X X
X X X X
X X X X X
X X X X X
X X X X X
X X X X X
X X X X X
X X X X X X
4/29/2019
ST3131, Lecture 19

11. Model Building for Various Uses of Regression Equations
Regression equations has many uses, including Process Description, Estimation, Prediction, and Control. For different uses, the focus of model building may be different.
1). Process Description
When a regression equation is used to describe a given process, there are two conflicting requirements:
(a). To account for as much of the variation as possible
(thus should as many of variables as possible)
(b). To make the model as simple as possible
(thus should as many of variables as possible)
Model Building: choose the smallest number of predictor variables that accounts for the most substantial part of the variation in the response variable.
4/29/2019
ST3131, Lecture 19

12. 2). Estimation and Prediction
When an regression equation is constructed for prediction at a future observation, the purpose is to make the prediction as accurate as possible.
Model Building: select the variables so that the MSE of prediction is minimized.
4/29/2019
ST3131, Lecture 19

13. 3). Control
The purpose for constructing an regression equation may be to determine the magnitude by which the value of a predictor variable must be altered to obtain a specified value of the response (target) variable.
For this control purpose, it is desired that the coefficients of the variables in the equation be measured accurately.
Model Building: select variables so that the standard errors of the regression coefficients are small.
4/29/2019
ST3131, Lecture 19

14. Remarks:
1). A regression equation may be constructed for several uses of regression equation since these uses can overlap with each other.
2). The use of the equation will in general determine the criterion that is to be optimized in its formulation.
3). A model which is best for one criterion may not be best for another.
4). Even for a same criterion, the best model may not be unique.
5).When there are several models that are adequate and could be used in forming an equation, we should list them out and make some further examination.
4/29/2019
ST3131, Lecture 19