1. 6.1.4 AIC, Model Selection, and the Correct Model oAny model is a simplification of reality oIf a model has relatively little bias, it tends to provide accurate estimates of the quantities of interest oBest model is often the simplest (less parameters)- model parsimony Akaike Information Criterion (AIC)- alternative to significance tests to estimate quantities of interest oCriterion for choosing between competing statistical models oAIC judges a model by how close its fitted values tend to be to the true values oThe AIC selects the model that minimizes: AIC = -2(maximized log likelihood – # parameters in the model) oThis penalizes a model for having too many parameters oServes the purpose of model comparison only; does not provide diagnostic about the fit of the model to the data

2. AIC = -2(maximized log likelihood – # parameters in the model) In SAS: AIC = -2LogL + 2p Crab Example : Table 6.2 (p. 215): The best models have smallest AIC’s oBest models have main effects, COLOR and WIDTH (AIC = 197.5) PROC LOGISTIC (Backward Elimination) : proc logistic descending ; class color spine / param = ref ; model y = width weight color spine / selection = backward lackfit ; Backward Elimination Procedure Step 0. The following effects were entered: Intercept width weight color spine Step 1. Effect spine is removed Step 2. Effect weight is removed Model Fit Statistics Criterion Intercept Only Intercept and Covariates AIC 227.759201.202 -2 Log L 225.759185.202 Analysis of Maximum Likelihood Estimates Parameter DFDFEstimate Standard Error Wald Chi-SquarePr > ChiSq Intercept 1-12.35082.628722.0749<.0001 width 10.49720.101723.8872<.0001 In our case, AIC is equal in all steps: 227.759 = -2LogL + 2p = 225.759 + 2(1), where p = 1

3. oRather than using selection techniques, such as stepwise, which look at significance levels of each parameter, use theory and common sense to build a model (Add and remove parameters that make sense) oA time ordering among variables may suggest causal relationships Example : (table 6.3, p. 217) In a British study, 1036 men and women (married and divorced) were asked whether they’ve had premarital and/or extramarital sex. We want to determine whether G = gender, P = premarital sex, and E = extramarital sex are factors in whether a person is M= married or divorced. Simple Model : G → P → E → M Any of these is an explanatory variable when a variable listed to its right is the response Complex Model (Triangular) : (Fig. 3.1, p. 218) 1 st stage : predicts G has a direct effect on P 2 nd stage : predicts P and G have direct effects on E 3rd stage : predicts E has direct effect on M ; P has direct and indirect effects on M; G has indirect effects through P and E 6.1.5 Using Causal Hypotheses to Guide Model Building

4. Table 6.4 : Goodness of Fit Tests for Model Selection 1 st Stage : predicts Gender has a direct effect on Premarital Sex The estimated odds of premarital sex for females is.27 times that for males. PMS YesNoTotal Female 100576 676 Male 141219 360 Total2417951036 data causal2 ; input gender $ PMS TOTALPMS ; datalines ; F 100 676 M 141 360 ; Model (Response P, no Actual Explanatory) PROC GENMOD DATA = CAUSAL2 DESCENDING ; CLASS GENDER ; MODEL PMS/TOTALPMS = / DIST = BIN LINK = LOGIT; Model (Response P, Actual Explanatory G) PROC GENMOD DATA = CAUSAL2 DESCENDING ; CLASS GENDER ; MODEL PMS/TOTALPMS = GENDER / DIST = BIN LINK = LOGIT TYPE3 RESIDUALS OBSTATS ; Criteria For Assessing Goodness Of Fit CriterionDFValueValue/DF Deviance 175.2594 Pearson Chi-Square 178.1753 Log Likelihood -561.9568 Criteria For Assessing Goodness Of Fit CriterionDFValueValue/DF Deviance 10.000 Pearson Chi-Square 10.000 Log Likelihood -524.3271

5. The L-R statistic -2(L 0 – L 1 ) test whether certain model parameters are zero by comparing the log likelihood L 1 for the fitted model M 1 with L 0 for the simpler model M 0 (formula p. 187) For the example, we will use the fact -2(L 0 – L 1 ) = G 2 (M 0 ) - G 2 (M 1 ) using SAS output. 1 st Stage : G 2 = G 2 (M 0 ) - G 2 (M 1 ) = 75.2594 – 0.0000 = 75.2594 -2(L 0 – L 1 ) = -2(-561.9568 – (-524.3271) = 75.2594 Df = 1 – 0 = 1, so χ 2 p-value <.001 and there is evidence of a gender effect on pre marital sex suggesting having G as an explanatory variable is a better model. 2 nd Stage : predicts Gender and Premarital Sex have direct effects on Extramarital Sex Goodness of Fit as a Likelihood-Ratio Test PMSEMS GENDER YesNo Total FemaleYes 2179 100 No 40536 576 MaleYes 39102 141 No 21198 219 data causal3 ; input gender $ PMS $ EMS TOTALEMS ; datalines ; F Y 21 100 F N 40 576 M Y 39 141 M N 21 219 ;

6. Model (Response E, no Actual Explanatory) PROC GENMOD DATA = CAUSAL3 DESCENDING ; CLASS GENDER PMS ; MODEL EMS/TOTALEMS = / DIST = BIN LINK = LOGIT TYPE3 RESIDUALS OBSTATS ; Model (Response E, P Actual Explanatory) PROC GENMOD DATA = CAUSAL3 ; CLASS GENDER PMS ; MODEL EMS/TOTALEMS = PMS / DIST = BIN LINK = LOGIT TYPE3 RESIDUALS OBSTATS ; Model (Response E, G+P Actual Explanatory) PROC GENMOD DATA = CAUSAL3 DESCENDING ; CLASS GENDER PMS ; MODEL EMS/TOTALEMS = GENDER PMS / DIST = BIN LINK = LOGIT TYPE3 RESIDUALS OBSTATS ; Criteria For Assessing Goodness Of Fit Criterion DFDFValueValue/DF Deviance 348.924416.3081 Pearson Chi-Square 356.773918.9246 Log Likelihood -373.4687 Criteria For Assessing Goodness Of Fit CriterionDFValueValue/DF Deviance 22.90801.4540 Pearson Chi-Square 22.95421.4771 Log Likelihood -350.4605 Model (Response E, no Actual Explanatory)Model (Response E, P Actual Explanatory) Criteria For Assessing Goodness Of Fit CriterionDFValueValue/DF Deviance 10.0008 Pearson Chi-Square 10.0008 Log Likelihood -349.0069 Model (Response E, G+P Actual Explanatory Model E = 1 vs. E = P G 2 (M 0 ) - G 2 (M 1 ) = 48.9244 – 2.9080 = 46.016 -2(L 0 – L 1 ) = -2(-373.4687–(-350.4605) = 46.016 df = 3-2= 1, so χ 2 p-value <.001, so there is evidence of a P effect on E Model E = P vs. E = G+P G 2 = G 2 (M 0 ) - G 2 (M 1 ) = 2.9080 -.0008 = 2.9 df = 2-1 = 1, so χ 2 p-value >.10 so only weak evidence occurs that G had a direct effect as well as indirect effect on E. So E = P is a sufficient model.

7. 3 rd stage : predicts Extramarital Sex has direct effect on Marriage ; Premarital Sex has direct and indirect effects on Marriage; Gender has indirect effects through PMS and EMS PMSEMS Divorced GENDER YesNo FemaleYes 174 21 No 5425 79 NoYes 364 40 No 214322 536 MaleYes 2811 39 No 6042 102 NoYes 174 21 No 68130 198 data causal ; input gender $ PMS $ EMS $ DIVORCED TOTAL ; datalines ; F Y Y 17 21 F Y N 54 79 F N Y 36 40 F N N 214 536 M Y Y 28 39 M Y N 60 102 M N Y 17 21 M N N 68 198 ; Model M = E + P vs. M = E*P G 2 = G 2 (M 0 ) - G 2 (M 1 ) = 18.1596 – 5.2455 = 12.91, with df = 5-4= 1 so χ 2 p-value <.10 so the interaction EMS*PMS is a better model to predict Divorce Model M = E*P vs. M = E*P + G G 2 = G 2 (M 0 ) - G 2 (M 1 ) = 5.2455 -.6978 = 4.5477, with df = 4-3= 1 so χ 2.025 < p-value <.05 so adding G to interaction EMS*PMS fits slightly better. Conclusion for Causal Relationships Good alternative for model building by using common sense to hypothesize relationships

8. 6.1.6 New Model-Building Strategies for Data Mining oData mining is the analysis of huge data sets, in order to find previously unsuspected relationships which are of interest or value oModel Building is challenging oThere are alternatives to traditional statistical methods, such as automated algorithms that ignore concepts such as sampling error and modeling oSignificance tests are usually irrelevant, since nearly any variable has significant effect if n is sufficiently large oFor large n, inference is less relevant than summary measures of predictive power