Presentation on theme: "November 13, 2013 Collect model fits for 4 problems Return reports"— Presentation transcript:
1November 13, 2013 Collect model fits for 4 problems Return reports VIFsLaunch chapter 11
2Grocery Data Assignment X3 (holiday) and X1 (cases shipped) do the job and X2 adds nothing; normality and variance constant okay; no need for quadratic terms or interactions; no need at all to square X3 (two level factor)
3Some notes State conclusion (final model) up front Report model with fit of X1 and X3 rather than from a fit with X1, X2, X3 and just drop X2Check assumptions for X1, X3 model not model with X2 as wellBox-Cox suggests no transformation even though l = 2 is “best”Interaction bit
4Notes on writing aspects Avoid imperative form of verbs (Fit the multivariate. Run the model. Be good.(You) Verb …Don’t use contractions. It’s bad form. They’re considered informal.Spell check does not catch wrong words (e.g., blow instead of below, not instead of note)Writing skills are important (benefits considerable)
5Variance Inflation Factors VIFs (not BFFs)Variance Inflation Factors(1-R2)-1 where R2 is the R-squared from regressing Xi on the other Xi’sAvailable in JMP if you know where to lookBody Fat examples
6x3 appears to be both a “dud” for predicting y and not very collinear with either x1 or x2
7In computing VIFs, need to regress x3 on x1 and x2, and compute 1/(1-RSquare) which here is about What gives?I thought x3 was pretty much unrelated to x1 and x2?????
14Interesting body fat example Looks like very little co-linearityHowever, massive multi-collinearity!This is why it can be challenging at times!!!Why we don’t throw extra dud variables into the model
15Steps in the analysisMultivariate to get acquainted with data(Analyze distribution all variables)Looking for a decent model—parsimoniousLinear, interactions, quadraticStepwise if many variablesPRESS vs. root mean square errorAdded variable plots according to tasteCheck assumptions (lots of plots)Check for outliers, influential observations (hats and Cook’s Di)
16Chapter 11: Remedial Measures We’ll cover in some detail:11.1 Weighted Least SquaresRidge regressionRegression trees11.5 Bootstrapping
1711.1 Weighted Least Squares Suppose that the constant variance assumption does not hold. Each residual has a different variance but keep the zero covariances:Least squares is out—what should we do?
18Use Maximum Likelihood for inspiration! Now define the i-th weight to be:Then the likelihood is:
19Taking logarithms, we get: Log Likelihood is a constant plus:Criterion is same as least squares, except each squared residual is weighted by wi -- hence the weighted least squares criterion.The coefficient vector bw that minimizes Qw is the vector of weighted least squares estimates
22Usual Case: Variances are Unknown Need to estimate each variance!Recall:Give a statistic that can estimate si2:Give a statistic that can estimate si:MSE……root MSE….. Press?????
23Estimating a Standard Deviation Function Step 1: Do ordinary least squares; obtain residualsStep 2: Regress the absolute values of the residuals against Y or whatever predictor(s) seem to be associated with changes in the variances of the residuals.Step 3: Use the predicted absolute residual for case i, |ei| as the estimated variance of ei, call it siStep 4: Then wi = (1/si)2^^^^
24Subset x and y for table 11.1Fit y on x and save residuals, compute absolute value of residualsRegress these residuals on xThe predicted values are estimated stan. Dev.’sWeights are reciprocal of stan. Dev. SquaredUse these weights with WLS on original y and x variables to get y-hat = x
27Notes on WLS EstimatesWLS estimates are minimum variance, unbiased.If you use Ordinary Least Squares (OLS) when variance is not constant, estimates are still unbiased, just not minimum variance.If you have replicates at each unique X category, you can just use the sample standard deviation of the responses at each category to determine the weight for any response in the category.R2 has no clear cut meaning here.Must use the standard deviation function value (instead of s) for confidence intervals for prediction
2811.2 Ridge RegressionBiased regression to reduce the effect of multicollinearity.Shrinkage estimation: Reduce the variance of the parameters by shrinking them (a bit) in absolute magnitude. This will introduce some bias, but may reduce the MSE overall.Recall: MSE = bias squared plus variance:Not in original pdf of slides
30How to Shrink? Penalized least squares! Start with standardized regression model:Add a “penalty” proportional to the total size of the parameters (proportionality or biasing constant is c):Not in pdf of slides
31Matrix Ridge SolutionStart with small c and increase (iteratively) until the coefficients stabilize. Plot is called “ridge trace”Not in pdf of slidesHere, use c about equal to .02
35Data mining and Predictive Modeling Predictive modeling mainly involves the application of:RegressionLogistic regressionRegression treesClassification treesNeural networksto very large data sets.The difference, technically, is because we have so much data, we can rely on the use of validation techniques---the use of training, validation and test sets to assess our models. There is much less concern about:Statistical significance (everything is significant!)Outliers/influence (a few outliers have no effect)Meaning of coefficients (models may have thousands of predictors)Distributional assumptions, independence, etc.Recall the quiz problem with 387K observations…even dopey ZIP Code and AgeBuilt were significant
36Data mining and Predictive Modeling We will talk about some of the statistical techniques used in predictive modeling, once the data have been gathered, cleaned, organized. But data gathering usually involves merging disparate data from different sources, data warehouses, etc., and usually represents at least 80% of the work.General Rule: Your organization’s data warehouse will not have the information you need to build your predictive model. (Paraphrased, Usama Fayyad, VP data, Yahoo)
37Regression TreesIdea: Can we cut up the predictor space into rectangles such that the response is roughly constant in each rectangle, but the mean changes from rectangle to rectangle?We’ll just use the sample average (Y) in each rectangle as our predictor!Simple, easy-to-calculate, assumption-free, nonparametric regression method. Note there is no “equation.” The predictive model takes the form of a decision tree._
38See file ch11ta08steroidSplitTreeCalc.jmp Overall Average of y is Steroid DataSee file ch11ta08steroidSplitTreeCalc.jmpOverallAverage of y is17.64; SSE is1284.8distinct agesnum in group
39Example: Steroid Data Predictive Model _ Fit ^ ^Example: What is Y at Age = 9.5?
40How do we find the regions (i.e., grow the tree)?For one predictor X, it’s easy.Step 1: To find the first split point Xs, make a grid of possible split points along the X axis. Each possible split point divides the X axis into two regions R21 and R22. Now compute SSE for the two-region regression tree:Do this for every grid point X. The point that leads to the minimum SSE is the split point.Steps 2: If you now have r regions, determine the best split point for each of the r regions as you did in step 1; choose the one that leads to the lowest SSE for the r + 1 regions.Steps 3: Repeat Step 2 until SSE levels off (more later on stopping)
41Illustrate first split with Steroid Data See file ch11ta08steroidSplitTreeCalc.jmpOverallAverage of y is17.64; SSE is1284.8distinct agesnum in group
42In the JMP file, aforementioned… Point out calculations needed to determine optimal first splitEasy but a bit tediousBinary vs. multiple splitsRun it in JMP, be sure to set min # in splitsFit conventional model as well…Don’t forget the titanic data…
43Growing the Steroid Level Tree Split Split 2Split Split 4
44When do we stop growing?If you let the growth process go on forever, you’ll eventually have n regions each with just one observation. The mean of each region is the value of the observation, and R2 = 100%. (You fitted n means (parameters) and so you have n – n = 0 degrees of freedom for error). Where to stop??We do this by data-splitting and cross-validation. After each split, use your model (tree) to predict each observation in a hold-out sample and compute MSPR or R2 (holdout) . As we saw with OLS regression, MSPR will start to increase (R2 for holdout will decrease) when we overfit.We can rely on this because we have very large sample sizes.Comment assumes no replicates
45What about multiple predictors? For two or more predictors, no problem.For each region, we have to determine the best predictor to split on AND the best split point for that predictor. So if we have p – 1 predictors, and at stage r we have r regions, there are r(p – 1) possible split points.Example: Three splits for two predictors
47Using JMP for Regression Trees Analyze >> Modeling >> PartitionExclude at least 1/3 for validation sample using: Rows >> Row Selection >> Select Randomly; then Rows >> ExcludeJMP will automatically give the predicted R2 value (1 – SSE/SSTO for the validation set)You need to manually call for a split (doesn’t fit the tree automatically)
48Split buttonNote: R2 for hold-out sampleAs you grow the tree this value will peak and begin to decline!Clicking the red triangle gives options: select “split history” to see a plot of predicted R2 vs. number of splits
49Classification Trees Regression tree equivalent of logistic regression Response is binary 0-1; average response in each region is now p, not YFor each possible split point, instead of SSE, we compute the G2 statistic for the resulting 2 by r contingency table.Split goes to the smallest value. (Can also use the negative of the log(p-value), where the p-value is adjusted in a Bonferroni-like manner. This is called the “Logworth” statistic. Again, you want a small value.)
53Understanding ROC and Lift Charts Assessing ability to classify a case (predict) correctly in logistic regression, classification trees, or neural networks (with binary responses) as a function of the cutoff value chosen.ROC Curve: Plot true positive rate [P(Y = 1|Y=1)] vs false positive rate [(P(Y = 1|Y=0)].Example 1: Classify top 40% (of predicted probabilities) as 1; bottom 60% as 0. Same as cutoff = .45, here.Pred prob:Data:Classification:Top 40% Bottom 60%^^
55Top 40%--------- Bottom 60%---------------- Example 2: Classify top 40% (of predicted probabilities) as 1; bottom 60% as 0. Same as cutoff = .45, here.Pred prob:Data:Classification:Top 40% Bottom 60%True positive rate: P(Yhat = 1|Y=1) = 3/3 = 1.0 (Y axis value)False positive rate: P(Yhat = 1|Y=0) = 1/7 = (X axis value)
5611.5 Bootstrapping in Regression A method that uses computer simulation, rather than theory and analytical results, to obtain sampling distributions of statistics. From these we can estimate the precision of an estimator.
57Background: Simulated Intervals Suppose:Our objective is to get a confidence interval for the slope in a simple linear regression setting.We know the distribution of Y at each X value.How can we use computer simulation (Minitab) to get a confidence interval for the slope?
58Background: Simulated Intervals Easy:Obtain a random Y value for each of the n X pointsCompute the regression.Store b1Do the above, say 1,000,000 times. Do a histogram of the b1 values, use the .025 and .975 percentiles!
59Simulated Intervals: Example Toluca company data: Assume E(Y) = X and s = 50That is: e ~ N( X , 50).Exec:let k1 = k1 + 1random 25 c3;normal 0 50.let c4 = *X + c3Regress c4 1 'X';Coefficients c5.let c6(k1) = c5(2)
60What if we don’t know the distribution of errors? Answer: Use the empirical distribution:Fit the model (assume its true)Then in the simulation, for each run, obtain a random sample n residuals (with replacement) from the n observed residuals.Compute the new Y values, run the regression, and store the bootstrap slope value, b1*This is the basic approach to the fixed-X sampling bootstrap
61To obtain confidence interval: Could use percentiles as previous.Better approach is the reflection method:d1 = b1 – b1*(a/2)d2 = b1*(1 - a/2) – b1b1 – d2 < b1 < b1 + d1
62Random-X Sampling Version When error variances are not constant or predictor variables cannot be regarded as fixed constants, random X sampling is used:For each bootstrap sample, we sample a (Y, X) pair with replacement from the data set.In effect we sample rows of the data set with replacement.
63Fixed X Example—Toluca Data Assume that the base regression has been run. We have stored the residuals in column c3, predicted values in c4.let k1 = k1 + 1sample 25 c3 c5;replace.let c6 = c4 + c5Regress c6 1 'X';Coefficients c7.let c8(k1) = c7(2)
64Neural NetworksThe i-th observation is modeled as a nonlinear function of m derived predictors, H0, …, Hm-1.
65Neural Networks OK, so what is gY and how are the predictors derived? gY is usually a logistic function and the Hj are a nonlinear function of a linear combination of the predictors XHere Xi is the i-th row of the X matrix
66Neural NetworksPut these together and you get the neural network model:A common choice for all of the nonlinear function is again the logistic:
67Neural NetworksThe gj functions are sometimes called the “activation” functions:The original idea was that when a linear combination of the predictors got large enough, a brain synapse would “fire” or “activate.” So this was an attempt to model a “step” input function.
68The gj functions are sometimes called the “activation” functions: Neural NetworksThe gj functions are sometimes called the “activation” functions:The original idea was that when a linear combination of the predictors got large enough, a brain synapse would “fire” or “activate.” So this was an attempt to model a “step” input function.Same as 461??
69Neural NetworksUsing the logistic for the gY and gj functions leads to the single-hidden-layer, feedforward neural network. Sometimes called the single layer perceptron.
70Network Representation Useful to view as network and compare to multiple regression:
71Parameter Estimation: Penalized Least Squares Recall that we found if too many parameters are fit in OLS, our ability to predict hold-out data can deteriorate. So we looked at adjusted R2, AIC, BIC, Mallows Cp, which all have built-in penalties for having too many parameters.Dropping some predictors is like setting the corresponding parameter estimate to zero, which “shrinks” the size of the regression coefficient vector:Another way to do this would be to leave all of the predictors in, but require that there be penalty on the estimation method for Size(b).
72Parameter Estimation: Penalized Least Squares This leads to the “penalized least squares” method. Choose the parameter estimates to minimize:Where the overfit penalty:is the sum of squares of the estimates.
73Example Using JMP (SAS) Software We’ll consider the Ischemic Heart Disease data set in Appendix C.9. Response is log(total cost subscriber claims), and the predictors considered are:Note: X1 is variable 5, X2 is variable 6, X3 is variable 9, and X4 is variable 8. The first 400 observations are used to fit (train) the model, and the last 388 are held out for validation
79A Sampling Application Frequently, have an idea about the variability in y off of an x-variableForestry application:What is average age of trees in a standDiameter of tree is “easy”Age of tree via ?????Diameter (first year grad students); age of trees in sample (second year grad students); estimation of average age in forest (3rd year);Summary report and invoice for the Professor/supervisor
80Sample tended to have “smaller trees in it. Fit corrects for this.) 1132 in theforestAverage diameter is 10.3118.3 off of fit(raw average ageis ;Average diameter inSample is 9.44…i.e.,Sample tended to have “smaller trees in it. Fit corrects for this.)Note, got using estimated standard deviationsRun the fit on the lohr data