Statistical Techniques I EXST7005 Simple Linear Regression

n Measuring & describing a relationship between two variables è Simple Linear Regression allows a measure of the rate of change of one variable relative to another variable. è Variables will always be paired, one termed an independent variable (often referred to as the X variable) and a dependent variable (termed a Y variable). è There is a change in the value of variable Y as the value of variable X changes.

Simple Linear Regression (continued) n For each value of X there is a population of values for the variable Y (normally distributed). Y X

Simple Linear Regression (continued) n The linear model which discribes this relationship is given as è Yi = b0 + b1Xi è this is the equation for a straight line è where; b0 is the value of the intercept (the value of Y when X = 0) è b1 is the amount of change in Y for each unit change in X. (i.e. if X changes by 1 unit, Y changes by b1 units). b1 is also called the slope or REGRESSION COEFFICIENT

Simple Linear Regression (continued) n Population Parameters   y.x = the true population mean of Y at each value of X   0 = the true value of the Y intercept   1 = the true value of the slope, the change in Y per unit of X   y.x =  0 +  1Xi è this is the population equation for a straight line

Simple Linear Regression (continued) n The sample equation for the line describes a perfect line with no variation. In practice there is always variation about the line. We include an additional term to represent this variation.  y.x =  0 +  1Xi +  i for a population n Yi = b0 + b1Xi + ei for a sample è when we put this term in the model, we are describing individual points as their position on the line, plus or minus some deviation

Simple Linear Regression (continued) Y X n

n the SS of deviations from the line will form the basis of a variance for the regression line n when we leave the ei off the sample model, we are describing a point on the regression line predicted from the sample. To indicate this we put a HAT on the Yi value Simple Linear Regression (continued)

Characteristics of a Regression Line The line will pass through the point  X,  Y (also the point 0, b0) n The sum of squared deviations (measured vertically) of the points from the regression line will be a minimum. n Values on the line can be described by the equation Y = b0 + b1Xi

Fitting the line n Fitting the line starts with a corrected SSDeviation, this is the SSDeviation of the observations from a horizontal line through the mean. Y X

Fitting the line (continued) n The fitted line is pivoted on the point until it has a minimum SSDeviations. Y X

Fitting the line (continued) How do we know the SSDeviations are a minimum? Actually, we solve the equation for ei, and use calculus to determine the solution that has a minimum of  ei2.

Fitting the line (continued) n The line has some desirable properties  E(b0) =  0  E(b1) =  1  E(  YX) =  X.Y n Therefore, the parameter estimates and predicted values are unbiased estimates.

The regression of Y on X n Y = the "dependent" variable, the variable to be predicted n X = the "independent" variable, also called the regressor or predictor variable. n Assumptions - general assumptions è Y variable is normally distributed at each value of X è The variance is homogeneous (across X). è Observations are independent of each other and ei independent of the rest of the model.

The regression of Y on X (continued) n Special assumption for regression. n Assume that all of the variation is attributable to the dependent variable (Y), and that the variable X is measured WITHOUT ERROR. n Note that the deviations are measured vertically, not horizontally or perpendicular to the line.

Derivation of the formulas n Any observation can be written as è Yi = b0 + b1Xi + ei for a sample è where; ei = a deviation fo the observed point from the regression line n note, the idea of regression is to minimize the deviation of the observations from the regression line, this is called a Least Squares Fit

Derivation of the formulas (continued)  ei = 0 n the sum of the squared deviations   ei2 =  (Yi - Yhat)2   ei2 =  (Yi - b0 + b1Xi )2 The objective is to select b0 and b1 such that  ei2 is a minimum, this is done with calculus n You do not need to know this derivation!

A note on calculations n We have previously defined the uncorrected sum of squares and corrected sum of squares of a variable Yi The uncorrected SS is  Yi2 The correction factor is (  Yi)2/n The corrected SS is  Yi2 - (  Yi)2/n n Your book calls this SYY, the correction factor is CYY n We could define the exact same series of calculations for Xi, and call it SXX

A note on calculations (continued) n We will also need a crossproduct for regression, and a corrected crossproduct n The crossproduct is XiYi The Sum of crossproducts is  XiYi, which is uncorrected The correction factor is (  Xi)(  Yi) / n = CXY The corrected crossproduct is  XiYi-(  Xi)(  Yi)/n n Which you book calls SXY

n the partial derivative is taken with respect to each of the parameters for b0 Derivation of the formulas (continued)

n set the partial derivative to 0 and solve for b0  2  (Yi-b0-b1Xi)(-1) = 0  -  Yi + nb0 + b1  Xi = 0  nb0 =  Yi - b1  Xi  b0 =  Y - b1  X n So b0 is estimated using b1 and the means of X and Y

Derivation of the formulas (continued) n Likewise for b1 we obtain the partial derivative

Derivation of the formulas (continued) n set the partial derivative to 0 and solve for b1  2  (Yi-b0-b1Xi)(-Xi) = 0  -  (YiXi + b0Xi + b1 Xi2) = 0  -  YiXi + b0  Xi + b1  Xi2) = 0  and since b0 =  Y - b1  X ), then   YiXi = (  Yi/n - b1  Xi/n )  Xi + b1  Xi2   YiXi =  Xi  Yi/n - b1 (  Xi)2/n + b1  Xi2   YiXi -  Xi  Yi/n = b1 [  Xi2 - (  Xi)2/n]  b1 = [  YiXi -  Xi  Yi/n] / [  Xi2 - (  Xi)2/n]

Derivation of the formulas (continued) b1 = [  YiXi -  Xi  Yi/n] / [  Xi2 - (  Xi)2/n] n b1 = SXY / SXX n so b1 is the corrected crossproducts over the corrected SS of X The intermediate statistics needed to solve all elements of a SLR are  Xi,  Yi, n,  Xi2,  YiXi and  Yi2 (this last term we haven't seen in the calculations above, but we will need later)

Derivation of the formulas (continued) n Review n We want to fit the best possible line, we define this as the line that minimizes the vertically measured distances from the observed values to the fitted line. n The line that achieves this is defined by the equations  b0 =  Y - b1  X  b1 = [  YiXi -  Xi  Yi/n] / [  Xi2 - (  Xi)2/n]

Derivation of the formulas (continued) n These calculations provide us with two parameter estimates that we can then use to get the equation for the fitted line.

Numerical example n See Regression handout

About Crossproducts n Crossproducts are used in a number of related calculations. n a crossproduct = YiXi Sum of crossproducts =  YiXi = SXY Covariance =  YiXi / (n-1) n Slope = SXY / SXX n SSRegression = S2XY / SXX Correlation = SXY /  SXXSYY n R2 = r2 = S2XY / SXXSYY = SSRegression/SSTotal