Midterm Review Goodness of Fit and Predictive Accuracy October 28, 2008

Assessing the fit of the model Summary measures of goodness of fit: Pearson chi-square statistic Deviance Akaike criterion Hosmer-Lemeshow statistic (can be used both when data can and cannot be grouped by covariate combinations). Comparing observed values to expected (predicted) values on the individual and grouped level not useful in this setting. Valid only if number of observations large compared to number of covariate patterns.

Assessing the fit of the model Summary goodness of fit measures give an overall indication of the fit of the model. Small value does not rule out possibility of substantial and, thus, interesting deviation from fit for a few subjects, BUT large value is clear indication of substantial problem with model.

Hosmer-Lemeshow goodness of fit test H0: Model fits data well. Ha: Model does not fit data well. To calculate test statistic: Order the fitted values. Group the fitted values into g classes of roughly equal size. Calculate the observed and expected number in each class. Perform a Χ2 goodness of fit test.

Example (H&L, page 150)

Example (H&L, page 150) How are the data grouped in this case? Deciles of estimated probabilities. What is the observed frequency in the drug free group for the fifth decile? 16 How is this value obtained? As the sum of the observed outcomes for the 58 subjects in this group. What is the corresponding estimated expected frequency? 12.7 How is this value obtained? As the sum of the 58 estimated probabilities for these subjects. When fitted logistic model is the correct model, the distribution of the H-L statistic is well-approximated by the chi-square distribution with 8 degrees of freedom. The H-L statistic = 4.39 with p-value 0.820. What can we conclude about the fit of the model? The model seems to fit the data well.

Predictive accuracy What do we mean by predictive accuracy? What are some commonly used measures of predictive accuracy? Sensitivity Specificity Area under the ROC curve (AUC) Why might one prefer to look at ROC curve rather than sensitivity/specificity? Sensitivity and specificity rely on a single cutpoint to classify a test result as positive. ROC curves plot sensitivity versus 1-specificity for all possible cutpoints.

Example: predictive accuracy Dr. C uses logistic regression to model the relationship between esophageal cancer risk and tobacco use, adjusting for known confounders. Dr. C calculates the area under the ROC curve (AUC) for the final model using the complete data. Dr. C speaks to a colleague about her analysis, who suggests that (in the future) she report AUC based on a different sample than was used to fit the model. Do you agree or disagree with her colleague’s suggestion? Why? Agree. We would expect AUC calculated based on the sample used to fit the model to be too optimistic.

Example: predictive accuracy Worried as to the effect this might have had on her results, Dr. C randomly divides the data into two samples of equal size. She re-fits her model using the first half of the data. She then calculates the AUC for both the first half of the data (the estimation sample) and the second half of the data (the validation sample) and obtains the results on the following slide. Can you tell which plot was obtained using the estimation data and which plot was obtained using the validation data? Why or why not? We would expect the AUC to be smaller for the validation sample than for the estimation sample in expectation. This is not guaranteed, however, for an individual dataset. Therefore, we cannot be sure which plot corresponds to the estimation sample and which corresponds to the validation sample.

Example: predictive accuracy

Example: predictive accuracy I tell you that the plot on the right was obtained using the validation sample. Comment on the predictive accuracy of Dr. C’s final model. With AUC = 0.7787, we conclude that the ability of the model to discriminate between subjects who experience the outcome (esophageal cancer) and subjects who do not experience the outcome is acceptable [see general rule on page 162 of H&L].