Topic4 Ordinary Least Squares

Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random variable  with E(  ) = 0 That is, E(Y) =   +   X Or, Y =   +   X + 

O X Y..... Observed points

O X Y Actual Line.. Y=  1 +  2 x...

O X Y Actual Line. Y=  1 +  2 x....

O X Y Actual Line. Y=  1 +  2 x....

O X Y Actual Line Y=  1 +  2 x.....

O X Y Actual Line Y=  1 +  2 x.....

O X Y. Actual Line Y=  1 +  2 x....

O X Y. Actual Line Y=  1 +  2 x... Y= b 1 + b 2 x Fitted Line. BC is an error of Estimation AC is an effect of the random factor C B. A.

The Ordinary Least Squares (OLS) estimates are obtained by minimising the sum of the squares of each of these errors. The OLS estimates are obtained from the values of X and the actual Y values (Y A ) as follows:

Error of estimation (e)  Y A –Y E | where Y E is the estimated value of Y.  e 2  Y A –Y E ] 2  e 2  Y A –(b 1 + b 2 X)] 2  e 2 /  b 1  Y A –(b 1 + b 2 X)] (-1) =0  e 2 /  b 2  Y A –(b 1 + b 2 X)] (-X) = 0

 Y –(b 1 + b 2 X)] (-1) = 0 -NY MEAN + N b 1 + b 2 NX MEAN = 0 b 1 = Y MEAN – b 2 X MEAN ….. (1)

 e 2 /  b 2  Y –(b 1 + b 2 X)] (-X) = 0  Y –(b 1 + b 2 X)] (-X) = 0 b 1  X –b 2  X 2 =  XY ………..(2) b 1 = Y MEAN - b 2 X MEAN ….. (1)

These estimates are given below (with the superscripts for Y dropped).   ^ 1 = (∑Y)(ΣX 2 ) – (∑X)(∑XY) N∑ X 2 - (∑X) 2  ^ 2 = N∑YX – (∑X)(∑Y) N∑ X 2 - (∑X) 2

Alternatively,  ^ 1 = Y MEAN -  ^ 2 X MEAN  ^ 2 = Covariance(X,Y) Variance(X)

(a)  e i  (Y i – Y i E ) = 0 and (b)  X 2i e i   X 2i (Y i – Y i E ) = 0 where Y i E is the estimated value of Y i. X 2i is the same as X i from before Proof:  (Y i – Y i E )=  Y i –  ^ 1 -  ^ 2 X 2i ) =  Y i –   ^ 1 -  ^ 2 X 2i = nY MEAN – n  ^ 1 - n  ^ 2 X MEAN = n(Y MEAN –  ^ 1 -  ^ 2 X MEAN ) = 0 [ since  ^ 1 = Y MEAN -  ^ 2 X MEAN ] Two Important Results

See the lecture notes for a proof of part (b) Total sum of squares (TSS)  (Y i – Y MEAN ) 2 Residual sum of squares (RSS)   (Y i – Y i E ) 2 Explained sum of squares (ESS)   (Y i E – Y MEAN ) 2

To prove that TSS = RSS + ESS TSS ≡  (Y i – Y MEAN ) 2 =  {(Y i – Y i E + Y i E – Y MEAN )} 2 =  (Y i – Y i E ) 2 +  (Y i E – Y MEAN )} 2  (Y i – Y i E )(Y i E – Y MEAN ) = RSS + ESS  (Y i – Y i E )(Y i E – Y MEAN )

 (Y i – Y i E )(Y i E – Y MEAN )  Y i – Y i E )(Y i E ) -Y MEAN  Y i – Y i E )   Y i – Y i E )(Y i E ) [by (a) above]  Y i – Y i E )(Y i E ) =  Y i – Y i E )(  ^ 1  ^ 2 X i ) =  ^ 1  Y i – Y i E )  ^ 2  X i  Y i – Y i E ) = 0 [by (a) and (b) above]

R 2 ≡ ESS/TSS Since TSS = RSS + ESS, it follows that 0  R 2 

Topic 5 Properties of Estimators

In the discussion that follows,  ^  is an estimator of the parameter of interest,  Bias of  ^ ≡ E(  ^) -   ^ is unbiased if Bias of  ^ = 0.  ^ is negatively biased if Bias of  ^ < 0.  ^ is positively biased if Bias of  ^ > 0.

Mean Squared Errors (MSE) of estimation for  ^ is given as MSE   ^ ≡ E[(  ^-  )] 2 MSE   ^ ≡ E[(  ^-  ) 2 ] ≡ E[{  ^-E(  ^) +E(  ^)-    ≡ E[{  ^-E(  ^)} 2 ] + E[{E(  ^)-    2E[{  ^-E(  ^)}*{E(  ^)-  ≡ Var(  ^) + {E(  ^)-    2E[{  ^-E(  ^)}*{E(  ^)- 

Now, E[{  ^-E(  ^)}*{E(  ^)-  ≡ {E(  ^)-E(  ^)}*{E(  ^)-  MSE   ^ ≡ Var(  ^) + {E(  ^)-   MSE   ^ ≡ Var(  ^) + (bias) 2. ≡ 0*{E(  ^)- 

If  ^ is unbiased, that is, if E(  ^)-  = 0. then we have, MSE   ^ ≡ Var(  ^) An unbiased estimator  ^ of a parameter  is efficient if and only if it has the smallest variance of all unbiased estimators TT hat is, for any other unbiased estimator p of  Var(  ^)≤ Var(p)

An estimator  ^ is said to be consistent if it converges in probability to . That is, Lim n  Prob(|  ^-  | >  ) = 0 for every  > 0.

When the above condition holds,  ^ is said to be the probability limit of , that is, plim  ^  Sufficient conditions for consistency: If the mean of  ^ converges to  and var(  ^) converges to zero (as n approaches  ) then  ^ ii s consistent.

That is,  ^ n is consistent if it can be shown that Lim n  E(  ^ n  And Lim n  Var(  ^ n 

The Regression Model with TWO Variables The Model :: Y =     X +  Y is the DEPENDENT variable X is the INDEPENDENT variable Y i    X 1i   X 2i  i 

The OLS estimates  ^ 1 and  ^ 2 are sample statistics used to estimate  1  and   2 respectively Y i    X 1i   X 2i  i  Here X 1i ≡ 1 for all i and X 2 is nothing but X.

Assumptions about X 2 : (1a) X 2 is non-random (chosen by the investigator) (1b) Random sampling is performed from a population of fixed values of X 2. (1c) : Lim (1/n)  x 2 2i ) = Q > 0 n  [ where x 2i  X 2i – X 2MEAN.] (1c) : Lim (1/n)  X 2i ) = P > 0 n 

Assumptions about the disturbance term  2a. E(  ) = 0 2b. Var(  i ) =  2 for all i. 2c. Cov(  i  j ) = 0 for i  j. (The  values are uncorrelated across observations). 2d. The  i all have a normal distribution Homoskedasticity

Result  ^ 2 is linear in the dependent variable Y i  ^ 2 = Covariance(X,Y) Variance(X)  ^ 2 =  Y i –Y MEAN )  X i –X MEAN )  X i –X MEAN ) 2 Proof:

 ^ 2 =  Y i  X i –X MEAN )  X i –X MEAN ) 2 + K   C i Y i  K where the C i and  K are constants

Therefore,  ^ 2 is a linear function of Y i Since, Y i    X 1i   X 2i  i   ^ 2 is a linear function of  i and hence is normally distributed

Similarly,  ^ 1 is a linear function of Y i (and hence  i ) and is normally distributed Both  ^ 1 and  ^ 2 are unbiased estimates of  1 and  2 respectively. That is, E(  ^ 1 ) =  1 and E(  ^ 2 ) =  2

Each of  ^ 1 and  ^ 2 is an efficient estimators of  1 and  2 respectively. Thus, each of  ^ 1 and  ^ 2 is a Best (efficient) Linear (in the dependent variable Y i ) Unbiased Estimator of  1 and  2 respectively. Each of  ^ 1 and  ^ 2 is a consistent estimator of  1 and  2 respectively. Also,

Var(  ^ 1 ) =   (1/n +X 2mean 2  x 2i 2 ) Var(  ^ 2 ) =    x 2i 2 ). Cov(  ^ 1,  ^ 2 ) = -   X 2mean  x 2i 2

LimVar(  ^ 2 ) n  = Lim    x 2i 2 n  = Lim   /n  x 2i 2 /n n  = 0/Q [using assumption (1c)] = 0

Because  ^ 2 is an unbiased estimator of  2 and LimVar(  ^ 2 ) = 0 n   ^ 2 is a consistent estimator of  2

The variance of the random term,  , is not known To perform statistical analysis, we estimate    by  ^ 2  RSS/(n-2) This is because  ^ 2 is an unbiased estimator of  2