2. 6-1 Introduction To Empirical Models

3. 6-1 Introduction To Empirical Models
Based on the scatter diagram, it is probably reasonable to assume that the mean of the random variable Y is related to x by the following straight-line relationship:
where the slope and intercept of the line are called regression coefficients.
The simple linear regression model is given by
where  is the random error term.

4. 6-1 Introduction To Empirical Models

5. 6-1 Introduction To Empirical Models
We think of the regression model as an empirical model.
Suppose that the mean and variance of  are 0 and 2, respectively, then
The variance of Y given x is

6. 6-1 Introduction To Empirical Models
The true regression model is a line of mean values:
where 1 can be interpreted as the change in the mean of Y for a unit change in x.
Also, the variability of Y at a particular value of x is determined by the error variance, 2.
This implies there is a distribution of Y-values at each x and that the variance of this distribution is the same at each x.

7. 6-1 Introduction To Empirical Models

8. 6-1 Introduction To Empirical Models
A Multiple Linear Regression Model:
𝑌= 𝛽 0 + 𝛽 1 𝑥 1 + 𝛽 2 𝑥 2 +𝜖
where
𝛽 0 = the intercept of the plane
𝛽 1 , 𝛽 2 = partial regression coefficients

9. 6-2 Simple Linear Regression
6-2.1 Least Squares Estimation
The case of simple linear regression considers a single regressor or predictor x and a dependent or response variable Y.
The expected value of Y at each level of x is a random variable:
We assume that each observation, Y, can be described by the model

10. 6-2 Simple Linear Regression
6-2.1 Least Squares Estimation
Suppose that we have n pairs of observations (x1, y1), (x2, y2), …, (xn, yn).
The method of least squares is used to estimate the parameters, 0 and 1 by minimizing the sum of the squares of the vertical deviations in Figure 6-6.

11. 6-2 Simple Linear Regression
6-2.1 Least Squares Estimation
Using Equation 6-8, the n observations in the sample can be expressed as
The sum of the squares of the deviations of the observations from the true regression line is

12. 6-2 Simple Linear Regression
6-2.1 Least Squares Estimation

13. 6-2 Simple Linear Regression
6-2.1 Least Squares Estimation

14. Cross-products Matrix
6-2 Simple Linear Regression
Sums of Squares and
Cross-products Matrix
The Sums of squares and cross-products matrix is a convenient way to summarize the quantities needed to do the hand calculations in regression. It also plays a key role in the internal calculations of the computer. It is outputed from PROC REG and PROC GLM if the XPX option is included on the model statement. The elements are
X’X
Intercept X Y n
𝑖=1 𝑛 𝑋 𝑖
𝑖=1 𝑛 𝑌 𝑖
𝑖=1 𝑛 𝑋 𝑖 2
𝑖=1 𝑛 𝑋 𝑖 𝑌 𝑖
𝑖=1 𝑛 𝑌 𝑖 2

15. 6-2 Simple Linear Regression
6-2.1 Least Squares Estimation

16. 6-2 Simple Linear Regression
6-2.1 Least Squares Estimation

17. 6-2 Simple Linear Regression
6-2.1 Least Squares Estimation

18. 6-2 Simple Linear Regression
6-2.1 Least Squares Estimation

19. 6-2 Simple Linear Regression
6-2.1 Least Squares Estimation

20. 6-2 Simple Linear Regression
6-2.1 Least Squares Estimation

21. 6-2 Simple Linear Regression
6-2.1 Least Squares Estimation

22. 6-2 Simple Linear Regression
Regression Assumptions and Model Properties

23. 6-2 Simple Linear Regression
Regression Assumptions and Model Properties

24. 6-2 Simple Linear Regression
Regression and Analysis of Variance
𝑆𝑆𝑇=𝑆𝑦𝑦= 𝑖=1 𝑛 ( 𝑦 𝑖 − 𝑦 ) 2 = 𝑖=1 𝑛 𝑦 2 − ( 𝑖=1 𝑛 𝑦𝑖)2 𝑛

25. 6-2 Simple Linear Regression
Regression and Analysis of Variance

26. 6-2 Simple Linear Regression
Example 6-1
OPTIONS NOOVP NODATE NONUMBER LS=140;
DATA ex61;
INPUT salt area
LABEL salt='Salt Conc' area='Roadway area';
CARDS;
PROC REG DATA=EX61;
MODEL SALT=AREA/XPX R;
PLOT SALT*AREA;
TITLE 'SCATTER PLOT';
RUN; QUIT;

27. 6-2 Simple Linear Regression

28. 6-2 Simple Linear Regression

29. 6-2 Simple Linear Regression

30. 6-2 Simple Linear Regression

31. 6-2 Simple Linear Regression

32. 6-2 Simple Linear Regression

33. 6-2 Simple Linear Regression

34. 6-2 Simple Linear Regression
6-2.2 Testing Hypothesis in Simple Linear Regression
Use of t-Tests
Suppose we wish to test
An appropriate test statistic would be

35. 6-2 Simple Linear Regression
6-2.2 Testing Hypothesis in Simple Linear Regression
Use of t-Tests
We would reject the null hypothesis if

36. 6-2 Simple Linear Regression
6-2.2 Testing Hypothesis in Simple Linear Regression
Use of t-Tests
Suppose we wish to test
An appropriate test statistic would be

37. 6-2 Simple Linear Regression
6-2.2 Testing Hypothesis in Simple Linear Regression
Use of t-Tests
We would reject the null hypothesis if

38. 6-2 Simple Linear Regression
6-2.2 Testing Hypothesis in Simple Linear Regression
Use of t-Tests
An important special case of the hypotheses of Equation 6-23 is
These hypotheses relate to the significance of regression.
Failure to reject H0 is equivalent to concluding that there is no linear relationship between x and Y.

39. 6-2 Simple Linear Regression
6-2.2 Testing Hypothesis in Simple Linear Regression
Use of t-Tests

40. 6-2 Simple Linear Regression
6-2.2 Testing Hypothesis in Simple Linear Regression
Use of t-Tests

41. 6-2 Simple Linear Regression

42. 6-2 Simple Linear Regression
The Analysis of Variance Approach

43. 6-2 Simple Linear Regression
6-2.2 Testing Hypothesis in Simple Linear Regression
The Analysis of Variance Approach

44. 6-2 Simple Linear Regression
6-2.3 Confidence Intervals in Simple Linear Regression

45. 6-2 Simple Linear Regression
6-2.3 Confidence Intervals in Simple Linear Regression

46. 6-2 Simple Linear Regression

47. 6-2 Simple Linear Regression

48. 6-2 Simple Linear Regression
6-2.4 Prediction of Future Observations

49. 6-2 Simple Linear Regression

50. 6-2 Simple Linear Regression
6-2.4 Prediction of Future Observations

51. 6-2 Simple Linear Regression
6-2.5 Checking Model Adequacy
Fitting a regression model requires several assumptions.
Errors are uncorrelated random variables with mean zero;
Errors have constant variance; and,
Errors be normally distributed.
The analyst should always consider the validity of these assumptions to be doubtful and conduct analyses to examine the adequacy of the model

52. 6-2 Simple Linear Regression
6-2.5 Checking Model Adequacy
The residuals from a regression model are ei = yi - ŷi , where yi is an actual observation and ŷi is the corresponding fitted value from the regression model.
Analysis of the residuals is frequently helpful in checking the assumption that the errors are approximately normally distributed with constant variance, and in determining whether additional terms in the model would be useful.

53. 6-2 Simple Linear Regression
6-2.5 Checking Model Adequacy
As an approximate check of normality, construct a frequency histogram or a normal probability plot of residuals.
Standardize the residuals by computing 𝑑 𝑖 = 𝑒 𝑖 / 𝜎 2 , i = 1, 2, …, n. If the errors are normally distributed, approximately 95% of the standardized residuals should fall in the interval (−2, +2).

54. 6-2 Simple Linear Regression
6-2.5 Checking Model Adequacy
Plot the residuals (1) in time sequence (if known), (2) against the 𝑦 𝑖 , and (3) against the independent variable 𝑥.
𝑦 , ln 𝑦, 1 𝑦

55. 6-2 Simple Linear Regression
6-2.5 Checking Model Adequacy

56. 6-2 Simple Linear Regression
6-2.5 Checking Model Adequacy

57. 6-2 Simple Linear Regression
Standardized
Residual
𝑑 10 = 𝑒 𝜎 =
𝑑 12 =2.186

58. 6-2 Simple Linear Regression
6-2.5 Checking Model Adequacy

59. 6-2 Simple Linear Regression
Example 6-1 (cont.)
OPTIONS NOOVP NODATE NONUMBER LS=140;
DATA ex61;
INPUT salt area
LABEL salt='Salt Conc' area='Roadway area';
CARDS;
DATA EX61N;
AREA=1.25;
OUTPUT;
DATA EX61N1;
SET EX61 EX61N;
PROC REG DATA=EX61N1;
MODEL SALT=AREA/CLM CLI r;
/* CLM for (100-α) % confidence limits for the expected value of the dependent variable,
CLI for (100- α) % confidence limits for an individual predicted value */
TITLE 'CIs FOR MEAN RESPONSE AND FUTURE OBSERVATION';
RUN; QUIT;

60. 6-2 Simple Linear Regression

61. 6-2 Simple Linear Regression

62. 6-2 Simple Linear Regression

63. 6-2 Simple Linear Regression

64. 6-2 Simple Linear Regression

65. 6-2 Simple Linear Regression

66. 6-2 Simple Linear Regression
6-2.6 Correlation and Regression
𝜌 as the population correlation coefficient, which is a measure of the strength of the linear relationship between Y and X in the population or joint distribution.
The sample correlation coefficient between X and Y is
𝑟= 𝑆 𝑥𝑦 𝑆 𝑥𝑥 𝑆 𝑦𝑦

67. 6-2 Simple Linear Regression
6-2.6 Correlation and Regression
The sample correlation coefficient is also closely related to the slope in a linear regression model

68. 6-2 Simple Linear Regression
Correlation Analysis
Correlation is a measure of the linear relationship between two random variables, X and Y. It is a parameter of the bivariate distribution (joint distribution) of X & Y and will be denoted by the Greek letter rho, 𝜌.
𝜌 = Corr(X, Y)
−1≤𝜌≤1
If 𝜌=1 then X and Y have a perfect direct linear relationship.
If 𝜌=−1 then X and Y have a perfect inverse linear relationship.
If 𝜌=0 then X and Y have no linear relationship.
If 0≤𝜌≤1 then X and Y are directly related. The closer to 1 the more perfect the linear relationship.
If −1≤𝜌≤0 then X and Y are inversely related. The closer to −1 the more perfect the linear relationship.

69. 6-2 Simple Linear Regression

70. 6-2 Simple Linear Regression
6-2.6 Correlation and Regression
It is often useful to test the hypotheses
The appropriate test statistic for these hypotheses is
Reject H0 if |t0| > t/2,n-2.

71. 6-2 Simple Linear Regression
Example 6-1 (Continued)
OPTIONS NOOVP NODATE NONUMBER LS=80;
DATA ex61;
INPUT salt area
LABEL salt='Salt Conc' area='Roadway area';
CARDS;
PROC CORR DATA=EX61;
VAR SALT AREA;
TITLE 'Correlation between SALT and AREA';
PROC REG;
MODEL salt=area/XPX;
TITLE 'Linear Regression of SALT vs AREA';
RUN; QUIT;

72. 6-2 Simple Linear Regression

73. 6-2 Simple Linear Regression