1. Multiple Regression models Estimation Goodness of fit tests
Week 4
Multiple Regression models
Estimation
Goodness of fit tests

2. Announcements Midterm on May 5th in Week 6
I sent an to online students to remind them about registering for the exam!
I reserved computer lab 658 on the 6th floor from 7:30-9:00pm to review SAS procedures for regression analysis.

3. Outline Last Week: This Week: Next 2 Weeks: Straight line regression
Multiple regression analysis with several predictors
Significance test on predictors
Goodness of fit test
Next 2 Weeks:
Residual Analysis (model diagnostics)
Computing predictions
Non linear models
Model selection techniques

4. Airline Index Price vs Jet Fuel Spot prices

5. Another graph to visualize the relationship
Corr(Airline index, Fuel Price) =
Regression line: Airline Index = * fuel

6. More general regression model
Consider one Y variable and k independent variables Xi, e.g. X1, X2, X3.
Data on n tuples (yi, xi1,xi2,xi3).
Scatter plots show linear association between Y and the X-variables
The observations on y can be assumed to satisfy the following model
Data
error
Prediction

7. Assumptions on the regression model
The relationship between the Y-variable and the X-variables is linear
The error terms ei (measured by the residuals)
have zero mean (E(ei)=0)
have the same standard deviation e for each fixed x
are approximately normally distributed - Typically true for large samples!
are independent (true if sample is S.R.S.)
Such assumptions are necessary to derive the inferential methods for testing and prediction (to be seen later)!
WARNING: if the sample size is small (n<50) and errors are not normal, you can’t use regression methods!

8. Parameter estimates
Suppose we have a random sample of n observations on Y and on k independent X-variables.
How do we estimate the values of the coefficients ’s in the regression model of Y versus the X-variables?

9. Regression Parameter estimates
The parameter estimates are those values for ’s that minimize the sum of the square errors:
Thus the parameter estimates are those values for ’s that will make the model residuals as small as possible!
The fitted model to compute predictions for Y is

10. Using Linear Algebra for model estimation (section 12.9)
Let be the parameter vector for the regression model in p variables.
The parameter estimates for each beta can be efficiently found using linear algebra as
where X is the (k x k) data matrix for the X-variables and Y is the data vector for the response variable.
Hard to compute by hand – better use a computer!
XT is the transpose of matrix X
X-1 denotes the inverse of matrix X

11. EXAMPLE: CPU usage
A study was conducted to examine what factors affect the CPU usage. A set of 38 processes written in a programming language was considered. For each program, data were collected on the
Y = CPU usage (time) in seconds of time,
X1= the number of lines (linet) in thousands generated by the process execution.
X2 = number of programs (step) forming the process
X3 = number of mounted computer devices (device).
Problem: Estimate the regression model of Y on X1,X2 and X3

12. I) Exploratory data step: Are the associations between Y and the x-variables linear?
Draw the scatter plot for each pair (Y, Xi)
CPU time
CPU time
Lines executed in process
Number of programs
CPU time
Do the plots show
linearity?
Mounted devices

13. The REG Procedure Parameter Estimates
PROC REG - SAS OUTPUT
The REG Procedure Parameter Estimates
Parameter Standard
Variable Label DF Estimate Error t Value Pr > |t|
Intercept
Linet <.0001
step <.0001
device
The fitted regression model is

14. Fitted model The fitted regression model is
The ’s estimated values measure the changes in Y for changes in X’s.
For instance, for each increase of 1000 lines executed by the process (keeping the other variables fixed), the CPU usage time will increase of seconds.
Fixing the other variables, what happens on the CPU time if I add another device?

15. Interpretation of model parameters
In multiple regression
The coefficient value i of Xi measures the predicted change in Y for any unit increase in Xi while the other independent variables stay constant.
For instance: measures the changes in Y for a unit increase of the variable X2 if the other x-variables X1 and X3 are fixed.

16. SAS procedure for regression analysis: PROC REG
PROC REG <DATA=dataset-name>; MODEL yvar=xvar1 xvar2 xvar3 xvar4; PLOT yvar*xvar1; RUN; MODEL: defines the variables in the model PLOT: defines plots of yvar vs x-variables defined in the MODEL statement. It can create multiple plots. Use the syntax: PLOT yvar *(xvar1 xvar2);

17. SAS Procedures for correlation values
The CORR option in PROC REG computes the correlation matrix for all the variables listed in MODEL.
PROC REG / CORR;
MODEL yvar=xvar1 xvar2 xvar3;
RUN;
Alternatively use PROC CORR to compute correlations:
PROC CORR;
VAR yvar xvar1 xvar2;
 list of variables. Correlation values are computed for each pair of variables in the list

18. SAS procedure for scatterplot: PROC GPLOT
PROC GPLOT; PLOT yvar*xvar; RUN; Creates a scatter plot of yvar versus xvar – Both variables must be defined in the SAS dataset. PLOT yvar*xvar=zvar; Creates a scatter plot of yvar versus xvar where points are labeled according to the z-var classifier.

19. SAS Code for the CPU usage data
Data cpu;
infile “cpudat.txt";
input time line step device;
linet=line/1000;
label time="CPU time in seconds" line="lines in program execution" step="number of computer programs" device="mounted devices" linet="lines in program (thousand)";
/*Exploratory data analysis */
/* computes correlation values between all variables in
dataset */
proc corr data=cpu;
run;
/* creates scatterplots between time vs linet, time vs step and time vs device, respectively */
proc gplot data=cpu;
plot time*(linet step device);

20. /* Regression analysis: fits model to predict time using linet, step and device*/
proc reg data=cpu;
model time=linet step device;
plot time*linet /nostat ;
run; quit;
If you want to fit a model with no intercept use the following model statement:
model time=linet step device / noint;

21. Are the estimated values accurate?
Residual Standard Deviation (pg. 632)
Testing effects of individual variables (pg )

22. How do we measure the accuracy of the estimated parameter values
How do we measure the accuracy of the estimated parameter values? (section 11.2)
For a simple linear regression with one X, the standard deviation of the parameter estimates are defined as:
They are both functions of the error variance regarded as a
sort of standard deviation (spread) of the points around the line!
The error variance is estimated by the residual standard deviation se (a.k.a. root mean square error )
Residuals!

23. How do we interpret residual standard deviation?
Used as a coarse approximation of the prediction error in predicting Y-values from regression model
Probable error in new predictions is
+/- 2 se
se also used in the formula of standard errors of parameter estimates:
They can be computed from the data and measure the noise in the parameter estimates

24. For general regression models with k x-variables
For k predictors (or explanatory variables), the standard errors of the parameter estimates have a complicated form. But they still depend on the error standard deviation
The residual standard deviation or root mean square error is defined as
k+1 = number of parameters ’s
This measures the precision of our predictions!

25. The REG Procedure
Analysis of Variance
Sum of Mean
Source DF Squares Square F Value Pr > F
Model <.0001
Error
Corrected Total
Root MSE R-Square
Dependent Mean Adj R-Sq
Coeff Var
The root mean square error for the CPU usage regression model is computed above. That gives an estimate of the error standard deviation
se=

26. Inference about regression parameters!
Regression estimates are affected by random error
The concepts of hypothesis testing and confidence intervals apply!
The t-distribution is used to construct significance tests and confidence intervals for the “true” parameters of the population.
Tests are used to select those x-variables that have a significant effect on Y

27. Tests on the slope for straight line regression
Consider the simple straight line case.
We often test the null hypothesis that the slope is equal to zero which is equivalent to “X has no effect on Y”!
Or in statistical terms :
X has a negative effect
Ho: vs Ha: X has a significant effect
X has a positive effect
The test is computed using
the t-statistic
With t-distribution with n-2 degrees of freedom!

28. Tests on the parameters in multiple regression
Assumptions on the data:
1. e1, e2, … en are independent of each other.
2. The ei are normally distributed with mean zero and have common variance s.
Significance Test on parameter tests hypothesis:
“Xj has no effect on Y” or in statistical terms
Xj has a negative effect on Y
Xj has a significant effect on Y
Xj has a positive effect on Y
The test is given by the t-statistic
with t-distribution with n-(k+1) degrees of freedom
Computed by SAS

29. Tests in SAS
The test p-values for regression coefficients are computed by PROC REG
SAS will produce the two-sided p-value.
If your alternative hypothesis is one-sided (either > or < ), then find the one-sided p-value dividing by 2 the p-value computed by SAS
one-sided p-value = (two-sided p-value)/2

30. The REG Procedure Parameter Estimates
SAS Output
The REG Procedure Parameter Estimates
Parameter Standard
Variable Label DF Estimate Error t Value Pr > |t|
Intercept
Linet <.0001
step <.0001
device
T-statistic value
P-value
T-tests on individual parameter values show that all the x-variables in the model are significant at 5% level (p-values <0.05).
We conclude that there is a significant association between Y and each x-variable.

31. Test on the intercept 0
The REG Procedure Parameter Estimates
Parameter Standard
Variable Label DF Estimate Error t Value Pr > |t|
Intercept
Linet <.0001
step <.0001
device
The test on the intercept says that the null hypothesis of 0=0 should be accepted. The test p-value is
This means that the model should have no intercept !
This is not recommended though – unless you know that Y=0 when all the x-variables are equal to zero.

32. What do we do if a model parameter is not significant?
If the t-test on a parameter bj shows that the parameter value is not significantly different from zero, we should:
Refit the regression model without the x-variable corresponding to bj.
Check if remaining variables are significant

33. Next Model diagnostics: Using the model for predictions
Evaluate the goodness of fit for the observed data
Analyze if model assumptions are satisfied
Using the model for predictions
Computing predictions and evaluating predictions accuracy

34. How is a Linear Regression Analysis done? A Protocol

35. Steps of Regression Analysis
Examine the scatterplot of the data.
Does the relationship look linear?
Are there points in locations they shouldn’t be?
Do we need a transformation?
Assuming a linear function looks appropriate, estimate the regression parameters.
Least Squares Estimates (done)
Examine the residuals for systematic inadequacies in the linear model as fit to the data. (next week)
Is there evidence that a more complicated relationship (say a polynomial) should be examined? (Residual analysis).
Are there data points which do not seem to follow the proposed relationship? (influential values or outliers).
Test whether there really is a statistically significant linear relationship.
Does model fit the data adequately?
Goodness of fit test (F-test for Variances) (next week)
Using the model for predictions (next week)

36. Residual analysis (Sections 11.5 and 13.4)
What are the residuals?

37. Standard residuals and standardized/studentized residuals
Standard residuals measure the difference between the actual y-values and the predicted values using the regression model:
For analysis we often use the studentized residuals to identify outliers, i.e. points that do not appear to be consistent with the rest of the data.
A studentized residual is computed as the i-th residual divided by its standard error.

38. Residual analysis
Residual analysis may display problems in the regression analysis and shows if there is some important variation in Y that is not explained by the regression model.
The studentized residuals follow an approximate t-distribution
(or N(0,1) in large datasets).
Detecting outliers:
Observations with | Student ei|>2 (i.e. larger than 2 or smaller than –2) may be outliers.
Use scatter plots of
Studentized residuals vs predicted values.
Studentized residuals vs x-values

39. Residual Plots
The analysis of the residuals is useful to detect possible problems and anomalies in the regression
Residual plots are scatter plots of the regression residuals against the X variables or the predicted values.
If model assumptions hold, points should be randomly scattered inside a band centered around the horizontal line at zero (the mean of the residuals).

40. Checking Assumptions
How well does the model fit?
Do predicted values seem to be placed in the middle of observed values?
Do data satisfy regression assumptions? (Problems seen in plot of X vs. Y will be reflected in residual plot.) y
Constant variance?
Systematic patterns suggest lack of independence or more complex model?
Poorly fit observations? x

41. “Good case” X “Bad cases”
Points are randomly scattered around the zero line
Residual X
“Bad cases”
Non linear relationship
Non constant variance

42. How do I know I have a problem?
Residuals are the computed as “observed y – predicted y”
Homoscedasticity (constant variance)
1) Plot predicted values versus residuals.
What is the pattern of the spread in the
residuals as the predicted values increase?
Acceptable x x x
Spread constant.
Spread increases.
Spread decreases then increases. x x x
Problems
Problems x x x x x x x x x x x x x x x x x x x x x

43. Residual Analysis
Residual analysis may display problems in the regression analysis –
It shows if there is some important variation in Y that is not explained by the regression model.
Plot residuals vs predicted values: To check linearity assumptions & constant variance for the errors;
plot residual.*predicted.; (PROC REG)
plot student.*predicted.;
Plot residuals vs each x-variable: To check linearity assumptions for Y and the x-variable;
plot residual.*xvar; (PROC REG)
plot student.*xvar;
Draw normal probability plot of residuals: To check normality assumption for the error terms; if points lie close to a line, the errors can be assumed to be approximately normal. Otherwise the assumption of normality is not satisfied.
plot npp.*residual.; (PROC REG)
plot npp.*student.;
Note the “.”

44. Residual plots using studentized residuals
Look for non-random patterns as they may be signs that model assumptions are not satisfied by the data
Plot residuals vs predicted values
To check linearity assumptions & constant variance for the errors:
PLOT student.*predicted.;
Plot residuals vs each x-variable;
To check linearity assumptions for Y and the x-variable;
PLOT student.*linet;

45. Normal probability plot of residuals
Draw normal probability plot of residuals;
To check normality assumption for the error terms; if points lie close to a line, the errors can be assumed to be approximately normal. Otherwise the assumption of normality is not satisfied.
SAS: Plot npp.*student.; (PROC REG)
Points look fairly linear.
Assumption of normality is satisfied!

46. SAS Code for the CPU usage data
Data cpu;
infile “cpudat.txt";
input time line step device;
linet=line/1000;
label time="CPU time in seconds" line="lines in program execution" step="number of computer programs" device="mounted devices" linet="lines in program (thousand)";
/*Exploratory data analysis */
proc corr data=cpu;
run;
proc gplot data=cpu;
plot time*(linet step device);

47. model time=linet step device / noint;
/* Regression analysis*/
proc reg data=cpu;
model time=linet step device;*/noint;
plot (residual. student.)*predicted./nostat;
plot student.*(linet step device)/nostat hplots=2 vplots=3;
plot npp.*student./nostat ;
run; quit;
If you want to fit a model with no intercept use the following model statement:
model time=linet step device / noint;