1. Lecture 4
This week’s reading: Ch. 1
Today:
Ch. 1: The Simple Regression Model
Interpretation of regression results
Goodness of fit

2. DERIVING LINEAR REGRESSION COEFFICIENTSY b2 b1 X1 Xn X
We chose the parameters of the fitted line so as to minimize the sum of the squares of the residuals. As a result, we derived the expressions for b1 and b2.
© Christopher Dougherty 1999–2006

3. INTERPRETATION OF A REGRESSION EQUATION
The scatter diagram shows hourly earnings in 2002 plotted against years of schooling, defined as highest grade completed, for a sample of 540 respondents from the National Longitudinal Survey of Youth.
© Christopher Dougherty 1999–2006

4. INTERPRETATION OF A REGRESSION EQUATION
Highest grade completed means just that for elementary and high school. Grades 13, 14, and 15 mean completion of one, two and three years of college.
© Christopher Dougherty 1999–2006

5. INTERPRETATION OF A REGRESSION EQUATION
Grade 16 means completion of four-year college. Higher grades indicate years of postgraduate education.
© Christopher Dougherty 1999–2006

6. INTERPRETATION OF A REGRESSION EQUATION
. reg EARNINGS S
Source | SS df MS Number of obs =
F( 1, 538) =
Model | Prob > F =
Residual | R-squared =
Adj R-squared =
Total | Root MSE =
EARNINGS | Coef. Std. Err t P>|t| [95% Conf. Interval]
S |
_cons |
This is the output from a regression of earnings on years of schooling, using Stata.
© Christopher Dougherty 1999–2006

7. INTERPRETATION OF A REGRESSION EQUATION
. reg EARNINGS S
Source | SS df MS Number of obs =
F( 1, 538) =
Model | Prob > F =
Residual | R-squared =
Adj R-squared =
Total | Root MSE =
EARNINGS | Coef. Std. Err t P>|t| [95% Conf. Interval]
S |
_cons |
For the time being, we will be concerned only with the estimates of the parameters. The variables in the regression are listed in the first column and the second column gives the estimates of their coefficients.
© Christopher Dougherty 1999–2006

8. INTERPRETATION OF A REGRESSION EQUATION
. reg EARNINGS S
Source | SS df MS Number of obs =
F( 1, 538) =
Model | Prob > F =
Residual | R-squared =
Adj R-squared =
Total | Root MSE =
EARNINGS | Coef. Std. Err t P>|t| [95% Conf. Interval]
S |
_cons |
In this case there is only one variable, S, and its coefficient is _cons, in Stata, refers to the constant. The estimate of the intercept is
© Christopher Dougherty 1999–2006

9. INTERPRETATION OF A REGRESSION EQUATION^
Here is the scatter diagram again, with the regression line shown.
© Christopher Dougherty 1999–2006

10. INTERPRETATION OF A REGRESSION EQUATION^
What do the coefficients actually mean?
© Christopher Dougherty 1999–2006

11. INTERPRETATION OF A REGRESSION EQUATION^
To answer this question, you must refer to the units in which the variables are measured.
© Christopher Dougherty 1999–2006

12. INTERPRETATION OF A REGRESSION EQUATION^
S is measured in years (strictly speaking, grades completed), EARNINGS in dollars per hour. So the slope coefficient implies that hourly earnings increase by $2.46 for each extra year of schooling.
© Christopher Dougherty 1999–2006

13. INTERPRETATION OF A REGRESSION EQUATION^
We will look at a geometrical representation of this interpretation. To do this, we will enlarge the marked section of the scatter diagram.
© Christopher Dougherty 1999–2006

14. INTERPRETATION OF A REGRESSION EQUATION
$15.53
$13.07
$2.46
One year
The regression line indicates that completing 12th grade instead of 11th grade would increase earnings by $2.46, from $13.07 to $15.53, as a general tendency.
© Christopher Dougherty 1999–2006

15. INTERPRETATION OF A REGRESSION EQUATION^
You should ask yourself whether this is a plausible figure. If it is implausible, this could be a sign that your model is misspecified in some way.
© Christopher Dougherty 1999–2006

16. INTERPRETATION OF A REGRESSION EQUATION^
For low levels of education it might be plausible. But for high levels it would seem to be an underestimate.
© Christopher Dougherty 1999–2006

17. INTERPRETATION OF A REGRESSION EQUATION^
What about the constant term? (Try to answer this question yourself before continuing with this sequence.)
© Christopher Dougherty 1999–2006

18. INTERPRETATION OF A REGRESSION EQUATION^
Literally, the constant indicates that an individual with no years of education would have to pay $13.93 per hour to be allowed to work.
© Christopher Dougherty 1999–2006

19. INTERPRETATION OF A REGRESSION EQUATION^
This does not make any sense at all. In former times craftsmen might require an initial payment when taking on an apprentice, and might pay the apprentice little or nothing for quite a while, but an interpretation of negative payment is impossible to sustain.
© Christopher Dougherty 1999–2006

20. INTERPRETATION OF A REGRESSION EQUATION^
A safe solution to the problem is to limit the interpretation to the range of the sample data, and to refuse to extrapolate on the ground that we have no evidence outside the data range.
© Christopher Dougherty 1999–2006

21. INTERPRETATION OF A REGRESSION EQUATION^
With this explanation, the only function of the constant term is to enable you to draw the regression line at the correct height on the scatter diagram. It has no meaning of its own.
© Christopher Dougherty 1999–2006

22. INTERPRETATION OF A REGRESSION EQUATION^
Another solution is to explore the possibility that the true relationship is nonlinear and that we are approximating it with a linear regression. We will soon extend the regression technique to fit nonlinear models.
© Christopher Dougherty 1999–2006

23. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
This sequence explains measures of goodness of fit in regression analysis. It is convenient to start by demonstrating three useful results. The first is that the mean value of the residuals must be zero.
© Christopher Dougherty 1999–2006

24. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
The residual in any observation is given by the difference between the actual and fitted values of Y for that observation.
© Christopher Dougherty 1999–2006

25. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
First substitute for the fitted value.
© Christopher Dougherty 1999–2006

26. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
Now sum over all the observations.
© Christopher Dougherty 1999–2006

27. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
Dividing through by n, we obtain the sample mean of the residuals in terms of the sample means of X and Y and the regression coefficients.
© Christopher Dougherty 1999–2006

28. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
If we substitute for b1, the expression collapses to zero.
© Christopher Dougherty 1999–2006

29. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
Next we will demonstrate that the mean of the fitted values of Y is equal to the mean of the actual values of Y.
© Christopher Dougherty 1999–2006

30. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
Again, we start with the definition of a residual.
© Christopher Dougherty 1999–2006

31. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
Sum over all the observations.
© Christopher Dougherty 1999–2006

32. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
Divide through by n. The terms in the equation are the means of the residuals, actual values of Y, and fitted values of Y, respectively.
© Christopher Dougherty 1999–2006

33. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
We have just shown that the mean of the residuals is zero. Hence the mean of the fitted values is equal to the mean of the actual values.
© Christopher Dougherty 1999–2006

34. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
Next we will demonstrate that the sum of the products of the values of X and the residuals is zero.
© Christopher Dougherty 1999–2006

35. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
We start by replacing the residual with its expression in terms of Y and X.
© Christopher Dougherty 1999–2006

36. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
We expand the expression.
© Christopher Dougherty 1999–2006

37. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
The expression is equal to zero. One way of demonstrating this would be to substitute for b1 and b2 and show that all the terms cancel out.
© Christopher Dougherty 1999–2006

38. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
A neater way is to recall the first order condition for b2 when deriving the regression coefficients. You can see that it is exactly what we need.
© Christopher Dougherty 1999–2006

39. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
Finally we will demonstrate that the sum of the products of the fitted values of Y and the residuals is zero.
© Christopher Dougherty 1999–2006

40. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
We start by substituting for the fitted value of Y.
© Christopher Dougherty 1999–2006

41. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
We expand and rearrange.
© Christopher Dougherty 1999–2006

42. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Four useful results:
The expression is equal to zero, given the first and third useful results.
© Christopher Dougherty 1999–2006

43. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
We now come to the discussion of goodness of fit. One measure of the variation in Y is the sum of its squared deviations around its sample mean, often described as the Total Sum of Squares, TSS.
© Christopher Dougherty 1999–2006

44. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
We will decompose TSS using the fact that the actual value of Y in any observations is equal to the sum of its fitted value and the residual.
© Christopher Dougherty 1999–2006

45. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
We substitute for Yi.
© Christopher Dougherty 1999–2006

46. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
From the useful results, the mean of the fitted values of Y is equal to the mean of the actual values. Also, the mean of the residuals is zero.
© Christopher Dougherty 1999–2006

47. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Hence we can simplify the expression as shown.
© Christopher Dougherty 1999–2006

48. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
We expand the squared terms on the right side of the equation.
© Christopher Dougherty 1999–2006

49. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
We expand the third term on the right side of the equation.
© Christopher Dougherty 1999–2006

50. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
The last two terms are both zero, given the first and fourth useful results.
© Christopher Dougherty 1999–2006

51. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Thus we have shown that TSS, the total sum of squares of Y can be decomposed into ESS, the ‘explained’ sum of squares, and RSS, the residual (‘unexplained’) sum of squares.
© Christopher Dougherty 1999–2006

52. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
The words explained and unexplained were put in quotation marks because the explanation may in fact be false. Y might really depend on some other variable Z, and X might be acting as a proxy for Z. It would be safer to use the expression apparently explained instead of explained.
© Christopher Dougherty 1999–2006

53. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
The main criterion of goodness of fit, formally described as the coefficient of determination, but usually referred to as R2, is defined to be the ratio of ESS to TSS, that is, the proportion of the variance of Y explained by the regression equation.
© Christopher Dougherty 1999–2006

54. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Obviously we would like to locate the regression line so as to make the goodness of fit as high as possible, according to this criterion. Does this objective clash with our use of the least squares principle to determine b1 and b2?
© Christopher Dougherty 1999–2006

55. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Fortunately, there is no clash. To see this, rewrite the expression for R2 in term of RSS as shown.
© Christopher Dougherty 1999–2006

56. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
The OLS regression coefficients are chosen in such a way as to minimize the sum of the squares of the residuals. Thus it automatically follows that they maximize R2.
© Christopher Dougherty 1999–2006

57. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Another natural criterion of goodness of fit is the correlation between the actual and fitted values of Y. We will demonstrate that this is maximized by using the least squares principle to determine the regression coefficients
© Christopher Dougherty 1999–2006

58. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
We will start with the numerator and substitute for the actual value of Y, and its mean, in the first factor.
© Christopher Dougherty 1999–2006

59. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
We rearrange a little.
© Christopher Dougherty 1999–2006

60. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
We expand the expression The last two terms are both zero (fourth and first useful results).
© Christopher Dougherty 1999–2006

61. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Thus the numerator simplifies to the sum of the squared deviations of the fitted values.
© Christopher Dougherty 1999–2006

62. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
We have the same expression in the denominator, under a square root. Cancelling, we are left with the square root in the numerator.
© Christopher Dougherty 1999–2006

63. © Christopher Dougherty 1999–2006
GOODNESS OF FIT
Thus the correlation coefficient is the square root of R2. It follows that it is maximized by the use of the least squares principle to determine the regression coefficients.
© Christopher Dougherty 1999–2006