Multiple Regression Analysis: Estimation y = b0 + b1x1 + b2x2 + . . . bkxk + u Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Motivation for Multiple Regression The model with two Independent Variables Since exper/avginc is correlative with edu/expend, it is useful to explicityly put it in the equation for the unbiasedness of the estimation on b1. Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

does not exist in 3.4 any more. Why? How to interpret the parameters in equation 3.4? Ceteris Paribus effect does not exist in 3.4 any more. Why? Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

cons inc consmax Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

The key assumption about how u is related to x1 and x2 is as equation 3.5. What it means? Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

The Model with K Independent Variables Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Parallels with Simple Regression b0 is still the intercept b1 to bk all called slope parameters u is still the error term (or disturbance) Still need to make a zero conditional mean assumption, so now assume that E(u|x1,x2, …,xk) = 0 Still minimizing the sum of squared residuals, so have k+1 first order conditions Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Mechanics and Interpretation of Ordinary Least Squares Obtaining the OLS Estimates. The method of OLS chooses the estimates to minimize the sum of squared residuals, that is to make 3.10 as Small as possible. Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

OLS First Order Condition SRF SSR OLS First Order Condition Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Interpreting the OLS Regression Equation Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Example 3.1 Determinants of College GPA Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Example 3.2 Hourly Wage Equation Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

On the Meaning of “Holding Other Factors Fixed” in Multiple Regression The power of multiple regression analysis is that it provides this ceteris paribus interpretation even though the data have not been collected in a ceteris paribus fashion. Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Changing More than One Independent Variable Simutaneously Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

OLS Fitted Values and Residuals Normally, Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

A “Partialling Out” Interpretation of Multiple Regression Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Proof: Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Comparison of Simple and Multiple Regression Estimates SRF MRF Two cases when the parameter estimates of SRF and MRF are equal: Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Example 3.3 Participation in 401 (K) Pension Plans. Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Goodness-of-Fit Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Example 3.4 Determinants of College GPA Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Example 3.5 Explaining Arrest Records Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

The Expected Value of the OLS Estimators The Assumptions for Multiple Regression Assumption MLR.1: the population model is linear in parameters as y = b0 + b1x1 + b2x2+…+ bkxk+ u Assumption MLR.2: we can use a random sample of size n, {(xi1, xi2 ,…, xik, yi): i=1, 2, …, n}, from the population model. Thus we can write the sample model yi = b0 + b1xi1 + b2xi2+…+ bkxik+ ui Assumption MLR.3: E(u| x1, x2 ,…, xk) = 0 Model Misspecification: cons = b0 + b1inc + b2inc2+ u Omitting an important factor; Measurement Error. Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Assumption MLR.4: No Perfect Collinearity In the sample (and therefore in the population), none of the independent variables is constant, and there are no exact linear relationships among the independents variables. Assumption MLR.4 does allow the independent variables to be correlated; they just cannot be perfectly correlated. One variable is a constant multiple of another, the same variable measured in different units. One independent variable can be expressed as an exact linear function of the two or more of the other independent variable. The sample size, n, is too small in relation to the number of parameters being estimated. Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Proof: Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Including Irrelevant Variables in a Regression Model What happens if we include variables in our specification that don’t belong? There is no effect on our parameter estimate, and OLS remains unbiased, but have undesirable effects on the variances of the OLS estimators. What if we exclude a variable from our specification that does belong? OLS will usually be biased Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Omitted Variable Bias Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Omitted Variable Bias (cont) Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Omitted Variable Bias (cont) Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Omitted Variable Bias (cont) Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Omitted Variable Bias Summary Two cases where bias is equal to zero b2 = 0, that is x2 doesn’t really belong in model x1 and x2 are uncorrelated in the sample. Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Summary of Direction of Bias Corr(x1, x2) > 0 Corr(x1, x2) < 0 b2 > 0 Positive bias Negative bias b2 < 0 Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

The More General Case Technically, can only sign the bias for the more general case if all of the included x’s are uncorrelated Typically, then, we work through the bias assuming the x’s are uncorrelated, as a useful guide even if this assumption is not strictly true Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Example: Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Variance of the OLS Estimators Now we know that the sampling distribution of our estimate is centered around the true parameter Want to think about how spread out this distribution is Much easier to think about this variance under an additional assumption, so Assume Var(u|x1, x2,…, xk) = s2 (Homoskedasticity) Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Variance of OLS (cont) Let x stand for (x1, x2,…xk) Assumption MLR.5: Var(u|x) = s2 also implies that Var(y| x) = s2 The 4 assumptions for unbiasedness, plus this homoskedasticity assumption are known as the Gauss-Markov assumptions Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Variance of OLS (cont) Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Proof: Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Components of OLS Variances The error variance: a larger s2 implies a larger variance for the OLS estimators The total sample variation: a larger SSTj implies a smaller variance for the estimators Linear relationships among the independent variables: a larger Rj2 implies a larger variance for the estimators Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Misspecified Models Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Misspecified Models (cont) While the variance of the estimator is smaller for the misspecified model, unless b2 = 0 the misspecified model is biased As the sample size grows, the variance of each estimator shrinks to zero, making the variance difference less important Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Estimating the Error Variance We don’t know what the error variance, s2, is, because we don’t observe the errors, ui What we observe are the residuals, ûi We can use the residuals to form an estimate of the error variance Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

Error Variance Estimate (cont) df = n – (k + 1), or df = n – k – 1 df (i.e. degrees of freedom) is the (number of observations) – (number of estimated parameters) Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.

The Gauss-Markov Theorem Given our 5 Gauss-Markov Assumptions it can be shown that OLS is “BLUE” Best Linear Unbiased Estimator Thus, if the assumptions hold, use OLS Copyright © 2007 Thomson Asia Pte. Ltd. All rights reserved.