1. The simple linear regression model and parameter estimation

2. Introduction
In the two-sample problems of Chapter 9, we compared parameters of the x and y distributions.
Even when the observations were paired, we did not use information about one of the variables to study the other variable.
That is the objective of regression analysis.
The variables considered are related in a non-deterministic fashion, i.e. knowing one variable doesn’t tell give the exact value of the other.

3. Introduction (continued)
The available bivariate data consists of n independent pairs
The variable x is called the independent (predictor, or explanatory) variable, and y is the dependent (or response) variable.
Associated with is the random variable and its observed value
A scatter plot gives a preliminary impression of the relationship between the variables.

4. The simple linear regression model
When a linear model seems to be a reasonable one based on the plot, we can fit the model
The quantity is assumed to be normally distributed with and
Model parameters are (intercept) and (slope).

5. The simple linear regression model (continued)
The only random variable on the right-hand side of is
The inclusion of the random error term allows the observations to lie either above or below the true regression line

6. Implications of the model
Let (the expected value of Y when X=x) and (the variance of Y when X=x).
Then
The value at the true line gives the expected value when X=x, and the variance is constant.

7. Implications of the model (continued)
For fixed x, Y is the sum of a constant and a random variable that is normally distributed with mean 0 and variance
Thus Y is normally distributed with mean
and variance
The slope is the expected change in Y with a 1-unit increase in
The intercept is the expected value of Y when X=0.
The intercept has little meaning if 0 is far outside of the range of the X values (this is called extrapolation).

8. Estimating model parameters
Our estimate of the regression line should be the line that in some sense provides the best fit to the data.
We choose the line that minimizes the sum of squared vertical deviations from the line.
Denote the least squares estimates by , .

9. Principle of least squares
The estimates are found to minimize the sum of squared deviations
Taking partial derivatives with respect to and , setting the results to zero, and solving gives the estimates given next.

10. Least squares estimates
The least squares estimate of the slope is
Formulas for easier computation are:
The estimate of the intercept is

11. Fitted values and residuals
The fitted (or predicted) values are .
The residuals are the differences
between the observed and fitted y values.

12. Error sum of squares and estimated variance
In regression analysis, is estimated by using the squared residuals.
The estimate of is
The error sum of squares SSE can be interpreted as how much of the total variation of Y cannot be attributed to the linear relationship.

13. Total sum of squares and proportion of explained variation
Let
Note that since the sum of squared deviations about the least squared line is as small as possible.
The ratio is the proportion of the total variation that cannot be explained by the simple linear regression model, and
is the proportion of variation explained by the model.

14. Coefficient of determination
The coefficient of determination, denoted by , is given by
(Note that SST = SSE + SSR)

15. Interpretation of coefficient of determination
The higher the value of , the more successful is the simple linear regression model in explaining the variation of y.
The coefficient of determination also gives an idea of how closely the line fits the points. It will be 1 when all of the points lie exactly on the line (in which case SSE=0).