DERIVING LINEAR REGRESSION COEFFICIENTS True model Y X This sequence shows how the regression coefficients for a simple regression model are derived, using the least squares criterion (OLS, for ordinary least squares) 1

DERIVING LINEAR REGRESSION COEFFICIENTS True model Y X We will start with a numerical example with just three observations: (1,3), (2,5), and (3,6). 2

DERIVING LINEAR REGRESSION COEFFICIENTS True model Y Fitted model b2 b1 X Writing the fitted regression as Y = b1 + b2X, we will determine the values of b1 and b2 that minimize RSS, the sum of the squares of the residuals. ^ 3

DERIVING LINEAR REGRESSION COEFFICIENTS True model Y Fitted model b2 b1 X Given our choice of b1 and b2, the residuals are as shown. 4

DERIVING LINEAR REGRESSION COEFFICIENTS The sum of the squares of the residuals is thus as shown above. 5

DERIVING LINEAR REGRESSION COEFFICIENTS The quadratics have been expanded. 6

DERIVING LINEAR REGRESSION COEFFICIENTS Like terms have been added together. 7

DERIVING LINEAR REGRESSION COEFFICIENTS For a minimum, the partial derivatives of RSS with respect to b1 and b2 should be zero. (We should also check a second-order condition.) 8

DERIVING LINEAR REGRESSION COEFFICIENTS The first-order conditions give us two equations in two unknowns. 9

DERIVING LINEAR REGRESSION COEFFICIENTS Solving them, we find that RSS is minimized when b1 and b2 are equal to 1.67 and 1.50, respectively. 10

DERIVING LINEAR REGRESSION COEFFICIENTS True model Y Fitted model b2 b1 X Here is the scatter diagram again. 11

DERIVING LINEAR REGRESSION COEFFICIENTS True model Y Fitted model b2 b1 X The fitted line and the fitted values of Y are as shown. 12

DERIVING LINEAR REGRESSION COEFFICIENTS Before we move on to the general case, it is as well to make a small but important mathematical point. 13

DERIVING LINEAR REGRESSION COEFFICIENTS When we establish the expression for RSS, we do so as a function of b1 and b2. At this stage, b1 and b2 are not specific values. Our task is to determine the particular values that minimize RSS. 14

DERIVING LINEAR REGRESSION COEFFICIENTS We should give these values special names, to differentiate them from the rest. 15

DERIVING LINEAR REGRESSION COEFFICIENTS Obvious names would be b1OLS and b2OLS, OLS standing for Ordinary Least Squares and meaning that these are the values that minimize RSS. We have re-written the first-order conditions and their solution accordingly. 16

DERIVING LINEAR REGRESSION COEFFICIENTS True model Y X1 Xn X Now we will proceed to the general case with n observations. 17

DERIVING LINEAR REGRESSION COEFFICIENTS True model Y Fitted model b2 b1 X1 Xn X Given our choice of b1 and b2, we will obtain a fitted line as shown. 18

DERIVING LINEAR REGRESSION COEFFICIENTS True model Y Fitted model b2 b1 X1 Xn X The residual for the first observation is defined. 19

DERIVING LINEAR REGRESSION COEFFICIENTS True model Y Fitted model b2 b1 X1 Xn X Similarly we define the residuals for the remaining observations. That for the last one is marked. 20

DERIVING LINEAR REGRESSION COEFFICIENTS RSS, the sum of the squares of the residuals, is defined for the general case. The data for the numerical example are shown for comparison.. 21

DERIVING LINEAR REGRESSION COEFFICIENTS The quadratics are expanded. 22

DERIVING LINEAR REGRESSION COEFFICIENTS Like terms are added together. 23

DERIVING LINEAR REGRESSION COEFFICIENTS } Note that in this equation the observations on X and Y are just data that determine the coefficients in the expression for RSS. 24

DERIVING LINEAR REGRESSION COEFFICIENTS } The choice variables in the expression are b1 and b2. This may seem a bit strange because in elementary calculus courses b1 and b2 are usually constants and X and Y are variables. 25

DERIVING LINEAR REGRESSION COEFFICIENTS } However, if you have any doubts, compare what we are doing in the general case with what we did in the numerical example. 26

DERIVING LINEAR REGRESSION COEFFICIENTS } The first derivative with respect to b1. 27

DERIVING LINEAR REGRESSION COEFFICIENTS } With some simple manipulation we obtain a tidy expression for b1 . 28

DERIVING LINEAR REGRESSION COEFFICIENTS } The first derivative with respect to b2. 29

DERIVING LINEAR REGRESSION COEFFICIENTS Divide through by 2. 30

DERIVING LINEAR REGRESSION COEFFICIENTS We now substitute for b1 using the expression obtained for it and we thus obtain an equation that contains b2 only. 31

DERIVING LINEAR REGRESSION COEFFICIENTS The definition of the sample mean has been used. 32

DERIVING LINEAR REGRESSION COEFFICIENTS The last two terms have been disentangled. 33

DERIVING LINEAR REGRESSION COEFFICIENTS Terms not involving b2 have been transferred to the right side. 34

DERIVING LINEAR REGRESSION COEFFICIENTS To create space, the equation is shifted to the top of the slide. 35

DERIVING LINEAR REGRESSION COEFFICIENTS Hence we obtain an expression for b2. 36

DERIVING LINEAR REGRESSION COEFFICIENTS In practice, we shall use an alternative expression. We will demonstrate that it is equivalent. 37

DERIVING LINEAR REGRESSION COEFFICIENTS Expanding the numerator, we obtain the terms shown. 38

DERIVING LINEAR REGRESSION COEFFICIENTS In the second term the mean value of Y is a common factor. In the third, the mean value of X is a common factor. The last term is the same for all i. 39

DERIVING LINEAR REGRESSION COEFFICIENTS We use the definitions of the sample means to simplify the expression. 40

DERIVING LINEAR REGRESSION COEFFICIENTS Hence we have shown that the numerators of the two expressions are the same. 41

DERIVING LINEAR REGRESSION COEFFICIENTS The denominator is mathematically a special case of the numerator, replacing Y by X. Hence the expressions are quivalent. 42

DERIVING LINEAR REGRESSION COEFFICIENTS True model Y Fitted model b2 b1 X1 Xn X The scatter diagram is shown again. We will summarize what we have done. We hypothesized that the true model is as shown, we obtained some data, and we fitted a line. 43

DERIVING LINEAR REGRESSION COEFFICIENTS True model Y Fitted model b2 b1 X1 Xn X We chose the parameters of the fitted line so as to minimize the sum of the squares of the residuals. As a result, we derived the expressions for b1 and b2. 44

DERIVING LINEAR REGRESSION COEFFICIENTS True model Y Fitted model b2 b1 X1 Xn X Again, we should make the mathematical point discussed in the context of the numerical example. These are the particular values of b1 and b2 that minimize RSS, and we should differentiate them from the rest by giving them special names, for example b1OLS and b2OLS. 45

DERIVING LINEAR REGRESSION COEFFICIENTS True model Y Fitted model b2 b1 X1 Xn X However, for the next few chapters, we shall mostly be concerned with the OLS estimators, and so the superscript 'OLS' is not really necessary. It will be dropped, to simplify the notation. 46

DERIVING LINEAR REGRESSION COEFFICIENTS True model Fitted model Typically, an intercept should be included in the regression specification. Occasionally, however, one may have reason to fit the regression without an intercept. In the case of a simple regression model, the true and fitted models become as shown. 47

DERIVING LINEAR REGRESSION COEFFICIENTS True model Fitted model We will derive the expression for b2 from first principles using the least squares criterion. The residual in observation i is ei = Yi – b2Xi. 48

DERIVING LINEAR REGRESSION COEFFICIENTS True model Fitted model With this, we obtain the expression for the sum of the squares of the residuals. 49

DERIVING LINEAR REGRESSION COEFFICIENTS True model Fitted model We differentiate with respect to b2. The OLS estimator is the value that makes this slope equal to zero (the first-order condition for a minimum). Note that we have differentiated properly between the general b2 and the specific b2OLS. 50

DERIVING LINEAR REGRESSION COEFFICIENTS True model Fitted model Hence, we obtain the OLS estimator of b2 for this model. 51

DERIVING LINEAR REGRESSION COEFFICIENTS True model Fitted model The second derivative is positive, confirming that we have found a minimum. 52

Copyright Christopher Dougherty 2012. These slideshows may be downloaded by anyone, anywhere for personal use. Subject to respect for copyright and, where appropriate, attribution, they may be used as a resource for teaching an econometrics course. There is no need to refer to the author. The content of this slideshow comes from Section 1.3 of C. Dougherty, Introduction to Econometrics, fourth edition 2011, Oxford University Press. Additional (free) resources for both students and instructors may be downloaded from the OUP Online Resource Centre http://www.oup.com/uk/orc/bin/9780199567089/. Individuals studying econometrics on their own who feel that they might benefit from participation in a formal course should consider the London School of Economics summer school course EC212 Introduction to Econometrics http://www2.lse.ac.uk/study/summerSchools/summerSchool/Home.aspx or the University of London International Programmes distance learning course EC2020 Elements of Econometrics www.londoninternational.ac.uk/lse. 2012.10.28