Quantitative Methods Residual Analysis Multiple Linear Regression C.W. Jackson/B. K. Gordor

Residual Analysis and Introduction to Multiple Linear Regression Outline Regression Assumptions Diagnostic Checks on Residuals Linearity Independence Constant Variance Normal Probability Plot Introduction to Multiple Linear Regression

Residuals in Regression

Linear Regression Assumptions Four principal assumptions which justify the use of linear regression models for purposes of inference or prediction: Linearity of the relationship between dependent and independent variables: Statistical independence of the errors in particular, no correlation between consecutive errors in the case of time series data) Homoscedasticity (constant variance) of the errors Normality of the error distribution.

Violation of Assumptions If any of these assumptions are violated, i.e. there are nonlinear relationships between dependent and independent variable(s) or the errors exhibit correlation (or non-independence) heteroscedasticity, (i.e. Opposite of homoscedacity or constant variance) non-normality Then statistical procedures such as t-test, p-values, forecasts, confidence intervals, are inefficient, biased or misleading.

Diagnosis Violations of linearity Serious: fitting a linear model to data which are nonlinearly related, leads to errors in predictions (especially when you extrapolate beyond the range of the sample data). How to diagnose Plot of residuals versus independent variables Plot of residuals versus predicted values No violation if Points are symmetrically distributed around horizontal line. No systematic patterns in plots How to remedy violation Consider applying a nonlinear transformation to the dependent and/or independent variables.

Characteristics of a well-behaved residual vs. fits plot The residuals scatter randomly around the 0 line. This suggests that the assumption that the relationship is linear is reasonable. The residuals roughly form a horizontal band around the 0 line. This suggests that the variances of the error terms are equal. No one residual stands out from the basic random pattern of residuals. This suggests that there are no unusual observations (i.e. outliers).

More Diagnosis Violations of Independence Potentially serious (especially in time series regression models) How to diagnose Plot of residuals versus independent variables Assess the residual values for bias (i.e. Not predominantly positive or negative) No violation if The residuals are randomly and symmetrically distributed around zero No correlation between consecutive errors Regression model does not systematically underpredicts or overpredicts.

More Diagnosis Violations of homoscedasticity Violation makes it difficult to estimate the true standard deviation of the forecast errors, resulting in confidence intervals that are too wide or too narrow. How to diagnose Plot of residuals versus predicted values or plots of residuals versus independent variables Violation if Residuals spread out in funnel shape in direction of horizontal axis How to remedy violation Consider applying a log transformation to the dependent variable.

Plot exhibits non constant variance (i.e. no homoscedasticity)

More Diagnosis Violations of normality Calculation of confidence intervals and various significance tests for coefficients are all based on the assumptions of normally distributed errors. How to diagnose Normal probability plot or Normal quantile plot of the residuals. These are plots of the fractiles of error distribution versus the fractiles of a normal distribution having the same mean and variance. No violation if the points on such a plot should fall close to the diagonal reference line. Violation if A bow-shaped pattern of deviations from the diagonal indicates that the residuals have excessive skewness. An S-shaped pattern of deviations indicates that the residuals have excessive kurtosis. How to remedy violation Consider a nonlinear transformation of variables

Example of a bad-looking normal quantile plot

Example of a good-looking normal quantile plot

Introduction to Multiple Linear Regression

Example: Anscombe Data Output from R