Download presentation
Presentation is loading. Please wait.
Published bySophia Ellis Modified over 9 years ago
1
1 Chapter 3 Multiple Linear Regression
2
2 3.1 Multiple Regression Models Suppose that the yield in pounds of conversion in a chemical process depends on temperature and the catalyst concentration. A multiple regression model that might describe this relationship is This is a multiple linear regression model in two variables.
3
3 3.1 Multiple Regression Models Figure 3.1 (a) The regression plane for the model E(y)= 50+10x 1 +7x 2. (b) The contour plot.
4
4 3.1 Multiple Regression Models In general, the multiple linear regression model with k regressors is
5
5 3.1 Multiple Regression Models
6
6 Linear regression models may also contain interaction effects: If we let x 3 = x 1 x 2 and 3 = 12, then the model can be written in the form
7
7 3.1 Multiple Regression Models
8
8
9
9 3.2 Estimation of the Model Parameters 3.2.1 Least Squares Estimation of the Regression Coefficients Notation n – number of observations available k – number of regressor variables, p-- k+1( number of regression coefficients) y – response or dependent variable x ij – ith observation on jth regressor j.
10
10 3.2.1 Least Squares Estimation of Regression Coefficients
11
11 3.2.1 Least Squares Estimation of the Regression Coefficients The sample regression model can be written as
12
12 3.2.1 Least Squares Estimation of the Regression Coefficients The least squares function is The function S must be minimized with respect to the coefficients.
13
13 3.2.1 Least Squares Estimation of the Regression Coefficients The least squares estimates of the coefficients must satisfy
14
14 3.2.1 Least Squares Estimation of the Regression Coefficients Simplifying, we obtain the least squares normal equations: The ordinary least squares estimators are the solutions to the normal equations.
15
15 3.2.1 Least Squares Estimation of the Regression Coefficients Matrix notation is more convenient to find the estimiates Let where
16
16 3.2.1 Least Squares Estimation of the Regression Coefficients
17
17 3.2.1 Least Squares Estimation of the Regression Coefficients These are the least-squares normal equations. The solution is
18
Linear Regression Analysis 5E Montgomery, Peck & Vining 18 3.2.1 Least Squares Estimation of the Regression Coefficients
19
19 3.2.1 Least Squares Estimation of the Regression Coefficients The n residuals can be written in matrix form as There will be some situations where an alternative form will prove useful Where H is called hat matrix
20
20 Example 3-1. The Delivery Time Data The model of interest is y = 0 + 1 x 1 + 2 x 2 +
21
21 Example 3-1. The Delivery Time Data Figure 3.4 Scatterplot matrix for the delivery time data from Example 3.1. R codes for the figure in “Chapter_3_nu lti_reg.txt”
22
22 Example 3-1 The Delivery Time Data Figure 3.5 Three-dimensional scatterplot of the delivery time data from Example 3.1.
23
23 Example 3-1 The Delivery Time Data
24
24 Example 3-1 The Delivery Time Data
25
25 Example 3-1 The Delivery Time Data
26
26
27
27 R Output
28
28 3.2.3 Properties of Least-Squares Estimators Statistical Properties Variances/Covariances p×p matrix Diagonal entities Cjj are variances, And the remaining Cij are covariance of two regression coefficients
29
Linear Regression Analysis 5E Montgomery, Peck & Vining 29 3.2.4 Estimation of 2 The residual sum of squares can be shown to be: The residual mean square for the model with p parameters is:
30
Linear Regression Analysis 5E Montgomery, Peck & Vining 30 3.2.4 Estimation of 2 Recall that the estimator of 2 is model dependent - that is, change the form of the model and the estimate of 2 will invariably change. –Note that the variance estimate is a function of the errors; “unexplained noise about the fitted regression line”
31
Which model is better? Let’s calculate the variance of errors of different models Model 1; consider two reggressors ( case and distance) Model 2; only consider reggressor “case” We would usually prefer a model with a small residual mean square (estimated variance of error). 31
32
Linear Regression Analysis 5E Montgomery, Peck & Vining 32 Example 3.2 Delivery Time Data
33
Linear Regression Analysis 5E Montgomery, Peck & Vining 33 Example 3.2 Delivery Time Data
34
Linear Regression Analysis 5E Montgomery, Peck & Vining 34 3.2.5 Inadequacy of Scatter Diagrams in Multiple Regression Scatter diagrams of the regressor variable(s) against the response may be of little value in multiple regression. –These plots can actually be misleading –If there is an interdependency between two or more regressor variables, the true relationship between x i and y may be masked.
35
35 Illustration of the Inadequacy of Scatter Diagrams in Multiple Regression
36
There is only one (of few) dominate reggressor The regressors operate nearly independent !!! Scartterplot could be misleading when several important regressors are related. ( we will discuss the analytical methods for sorting out the relationships between regressors in later chapter. Scatterplot is useful if… 36
37
37 3.3 Hypothesis Testing in Multiple Linear Regression Once we have estimated the parameters in the model, we face two immediate questions: 1. What is the overall adequacy of the model? 2. Which specific regressors seem important?
38
Linear Regression Analysis 5E Montgomery, Peck & Vining 38 3.3 Hypothesis Testing in Multiple Linear Regression Test for Significance of Regression (sometimes called the global test of model adequacy) Tests on Individual Regression Coefficients (or groups of coefficients) Next we will consider:
39
Linear Regression Analysis 5E Montgomery, Peck & Vining 39 3.3.1 Test for Significance of Regression The test for significance is a test to determine if there is a linear relationship between the response and any of the regressor variables The hypotheses are H 0 : 1 = 2 = …= k = 0 H 1 : j 0 for at least one j
40
40 3.3.1 Test for Significance of Regression As in Chapter 2, the total sum of squares can be partitioned in two parts: SS T = SS R + SS Res This leads to an ANOVA procedure with the test (F) statistic
41
41 3.3.1 Test for Significance of Regression The standard ANOVA is conducted with
42
Linear Regression Analysis 5E Montgomery, Peck & Vining 42 3.3.1 Test for Significance of Regression ANOVA Table: Reject H 0 if or p-1 or n-p
43
43 3.3.1 Test for Significance of Regression R 2 –R 2 is calculated exactly as in simple linear regression –R 2 can be inflated simply by adding more terms to the model (even insignificant terms) Adjusted R 2 –Penalizes you for added terms to the model that are not significant
44
Linear Regression Analysis 5E Montgomery, Peck & Vining 44 Example 3.3 Delivery Time Data
45
Linear Regression Analysis 5E Montgomery, Peck & Vining 45 Example 3.3 Delivery Time Data To test H 0 : 1 = 2 = 0, we calculate the F–statistic:
46
46 Example 3.3 Delivery Time Data R 2 = 0.9596 Adjusted R 2 = 0.9559 To look at the overall significance of regression: p-value of F test R 2 Adjusted R 2
47
Adding a variable will always result in increase of R –squared. Our goal is to only add necessary regressors that will reduce the residual variability.. But we do not want over-fitting( add un necessary variables ( will learn variable selection procedure in later chapters). 47
48
48 3.3.2 Tests on Individual Regression Coefficients Hypothesis test on any single regression coefficient: Test Statistic: –Reject H 0 if |t 0 | > –This is a partial or marginal test!
49
Linear Regression Analysis 5E Montgomery, Peck & Vining 49 The Extra Sum of Squares method can also be used to test hypotheses on individual model parameters or groups of parameters Full model
50
Linear Regression Analysis 5E Montgomery, Peck & Vining 50
51
Linear Regression Analysis 5E Montgomery, Peck & Vining 51
52
Linear Regression Analysis 5E Montgomery, Peck & Vining 52 3.3.3 Special Case of Orthogonal Columns in X If the columns X 1 are orthogonal to the columns in X 2, the sum of squares due to 2 that is free of any dependence on the the regressors in X 1.
53
Linear Regression Analysis 5E Montgomery, Peck & Vining 53 Example Consider a dataset with four regressor variables and a single response. Fit the equation with all regressors and find that: y = - 19.9 + 0.0123x 1 + 27.3x 2 - 0.0655x 3 - 0.196x 4 Looking at the t-tests, suppose that x3 is insignificant. So it is removed. What is the equation now? Generally, it is not y = - 19.9 + 0.0123x 1 + 27.3x 2 - 0.196x 4
54
Linear Regression Analysis 5E Montgomery, Peck & Vining 54 Example The model must be refit with the insignificant regressors left out of the model. The regression equation is y = - 24.9 + 0.0117x 1 + 31.0x 2 - 0.217x 4 The refitting must be done since the coefficient estimates for an individual regressor depend on all of the regressors, x j
55
Linear Regression Analysis 5E Montgomery, Peck & Vining 55 Example However, if the columns are orthogonal to each other, then there is no need to refit. Can you think of some situations where we would have orthogonal columns?
56
Linear Regression Analysis 5E Montgomery, Peck & Vining 56 3.4.1. Confidence Intervals on the Regression Coefficients A 100(1- ) percent C.I. for the regression coefficient, j is: Or,
57
Linear Regression Analysis 5E Montgomery, Peck & Vining 57
58
Linear Regression Analysis 5E Montgomery, Peck & Vining 58 3.4.2. Confidence Interval Estimation of the Mean Response 100(1- ) percent CI on the mean response at the point x 01, x 02, …, x 0k is See Example 3-9 on page 95 and the discussion that follows
59
Linear Regression Analysis 5E Montgomery, Peck & Vining 59
60
Linear Regression Analysis 5E Montgomery, Peck & Vining 60
61
Linear Regression Analysis 5E Montgomery, Peck & Vining 61
62
Linear Regression Analysis 5E Montgomery, Peck & Vining 62 3.4.3. Simultaneous Confidence Intervals on Regression Coefficients It can be shown that From this result, the joint confidence region for all parameters in is
63
Linear Regression Analysis 5E Montgomery, Peck & Vining 63 3.5 Prediction of New Observations A 100(1- ) percent prediction interval for a future observation is
64
Linear Regression Analysis 5E Montgomery, Peck & Vining 64
65
Linear Regression Analysis 5E Montgomery, Peck & Vining 65 3.6 Hidden Extrapolation in Multiple Regression In prediction, exercise care about potentially extrapolating beyond the region containing the original observations. Figure 3.10 An example of extrapolation in multiple regression.
66
Linear Regression Analysis 5E Montgomery, Peck & Vining 66 3.6 Hidden Extrapolation in Multiple Regression We will define the smallest convex set containing all of the original n data points (x i1, x i2, … x ik ), i = 1, 2, …, n, as the regressor variable hull RVH. If a point x 01, x 02, …, x 0k lies inside or on the boundary of the RVH, then prediction or estimation involves interpolation, while if this point lies outside the RVH, extrapolation is required.
67
Linear Regression Analysis 5E Montgomery, Peck & Vining 67 3.6 Hidden Extrapolation in Multiple Regression Diagonal elements of the matrix H = X(X’X) -1 X’ can aid in determining if hidden extrapolation exists: The set of points x (not necessarily data points used to fit the model) that satisfy is an ellipsoid enclosing all points inside the RVH.
68
Linear Regression Analysis 5E Montgomery, Peck & Vining 68 3.6 Hidden Extrapolation in Multiple Regression Let x 0 be a point at which prediction or estimation is of interest. Then If h 00 > h max then the point is a point of extrapolation.
69
Linear Regression Analysis 5E Montgomery, Peck & Vining 69 Example 3.13 Consider prediction or estimation at:
70
Linear Regression Analysis 5E Montgomery, Peck & Vining 70 Figure 3.10 Scatterplot of cases and distance for the delivery time data. #9 a b c d
Similar presentations
© 2024 SlidePlayer.com Inc.
All rights reserved.