1 11 Simple Linear Regression and Correlation 11-1 Empirical Models 11-2 Simple Linear Regression 11-3 Properties of the Least Squares Estimators 11-4 Hypothesis Test in Simple Linear Regression Use of t-tests Analysis of variance approach to test significance of regression 11-5 Confidence Intervals Confidence intervals on the slope and intercept Confidence interval on the mean response 11-6 Prediction of New Observations 11-7 Adequacy of the Regression Model Residual analysis Coefficient of determination (R 2 ) 11-8 Correlation 11-9 Regression on Transformed Variables Logistics Regression CHAPTER OUTLINE Chapter 11 Table of Contents

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. Learning Objectives for Chapter 11 After careful study of this chapter, you should be able to do the following: 1.Use simple linear regression for building empirical models to engineering and scientific data. 2.Understand how the method of least squares is used to estimate the parameters in a linear regression model. 3.Analyze residuals to determine if the regression model is an adequate fit to the data or to see if any underlying assumptions are violated. 4.Test the statistical hypotheses and construct confidence intervals on the regression model parameters. 5.Use the regression model to make a prediction of a future observation and construct an appropriate prediction interval on the future observation. 6.Apply the correlation model. 7.Use simple transformations to achieve a linear regression model. 2Chapter 11 Learning Objectives

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-1: Empirical Models Many problems in engineering and science involve exploring the relationships between two or more variables. Regression analysis is a statistical technique that is very useful for these types of problems. For example, in a chemical process, suppose that the yield of the product is related to the process-operating temperature. Regression analysis can be used to build a model to predict yield at a given temperature level. 3

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-1: Empirical Models 4

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-1: Empirical Models Figure 11-1 Scatter Diagram of oxygen purity versus hydrocarbon level from Table

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-1: Empirical Models Based on the scatter diagram, it is probably reasonable to assume that the mean of the random variable Y is related to x by the following straight-line relationship: where the slope and intercept of the line are called regression coefficients. The simple linear regression model is given by where  is the random error term. 6

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-1: Empirical Models We think of the regression model as an empirical model. Suppose that the mean and variance of  are 0 and  2, respectively, then The variance of Y given x is 7

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-1: Empirical Models The true regression model is a line of mean values: where  1 can be interpreted as the change in the mean of Y for a unit change in x. Also, the variability of Y at a particular value of x is determined by the error variance,  2. This implies there is a distribution of Y-values at each x and that the variance of this distribution is the same at each x. 8

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-1: Empirical Models Figure 11-2 The distribution of Y for a given value of x for the oxygen purity- hydrocarbon data. 9

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-2: Simple Linear Regression The case of simple linear regression considers a single regressor or predictor x and a dependent or response variable Y. The expected value of Y at each level of x is a random variable: We assume that each observation, Y, can be described by the model 10

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-2: Simple Linear Regression Suppose that we have n pairs of observations (x 1, y 1 ), (x 2, y 2 ), …, (x n, y n ). Figure 11-3 Deviations of the data from the estimated regression model. 11

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-2: Simple Linear Regression The method of least squares is used to estimate the parameters,  0 and  1 by minimizing the sum of the squares of the vertical deviations in Figure Figure 11-3 Deviations of the data from the estimated regression model. 12

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-2: Simple Linear Regression Using Equation 11-2, t he n observations in the sample can be expressed as The sum of the squares of the deviations of the observations from the true regression line is 13

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-2: Simple Linear Regression 14

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-2: Simple Linear Regression 15

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-2: Simple Linear Regression Definition 16

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-2: Simple Linear Regression 17

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-2: Simple Linear Regression Notation 18

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-2: Simple Linear Regression Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-2: Simple Linear Regression Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-2: Simple Linear Regression Example 11-1 Figure 11-4 Scatter plot of oxygen purity y versus hydrocarbon level x and regression model ŷ = x. 21

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-2: Simple Linear Regression Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 23

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-2: Simple Linear Regression Estimating  2 The error sum of squares is It can be shown that the expected value of the error sum of squares is E(SS E ) = (n – 2)  2. 24

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-2: Simple Linear Regression Estimating  2 An unbiased estimator of  2 is where SS E can be easily computed using 25

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-3: Properties of the Least Squares Estimators Slope Properties Intercept Properties 26

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-4: Hypothesis Tests in Simple Linear Regression Use of t-Tests Suppose we wish to test An appropriate test statistic would be 27

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-4: Hypothesis Tests in Simple Linear Regression Use of t-Tests We would reject the null hypothesis if The test statistic could also be written as : 28

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-4: Hypothesis Tests in Simple Linear Regression Use of t-Tests Suppose we wish to test An appropriate test statistic would be 29

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-4: Hypothesis Tests in Simple Linear Regression Use of t-Tests We would reject the null hypothesis if 30

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-4: Hypothesis Tests in Simple Linear Regression Use of t-Tests An important special case of the hypotheses of Equation is These hypotheses relate to the significance of regression. Failure to reject H 0 is equivalent to concluding that there is no linear relationship between x and Y. 31

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-4: Hypothesis Tests in Simple Linear Regression Figure 11-5 The hypothesis H 0 :  1 = 0 is not rejected. 32

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-4: Hypothesis Tests in Simple Linear Regression Figure 11-6 The hypothesis H 0 :  1 = 0 is rejected. 33

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-4: Hypothesis Tests in Simple Linear Regression Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-4: Hypothesis Tests in Simple Linear Regression Analysis of Variance Approach to Test Significance of Regression The analysis of variance identity is Symbolically, 35

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-4: Hypothesis Tests in Simple Linear Regression Analysis of Variance Approach to Test Significance of Regression If the null hypothesis, H 0 :  1 = 0 is true, the statistic follows the F 1,n-2 distribution and we would reject if f 0 > f ,1,n-2. 36

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-4: Hypothesis Tests in Simple Linear Regression Analysis of Variance Approach to Test Significance of Regression The quantities, MS R and MS E are called mean squares. Analysis of variance table: 37

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-4: Hypothesis Tests in Simple Linear Regression Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-4: Hypothesis Tests in Simple Linear Regression 39

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-5: Confidence Intervals Confidence Intervals on the Slope and Intercept Definition 40

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-5: Confidence Intervals Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-5: Confidence Intervals Confidence Interval on the Mean Response Definition 42

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-5: Confidence Intervals Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-5: Confidence Intervals Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-5: Confidence Intervals Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-5: Confidence Intervals Figure 11-7 Figure 11-7 Scatter diagram of oxygen purity data from Example 11-1 with fitted regression line and 95 percent confidence limits on  Y|x0. 46

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-6: Prediction of New Observations If x 0 is the value of the regressor variable of interest, is the point estimator of the new or future value of the response, Y 0. 47

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-6: Prediction of New Observations Definition 48

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-6: Prediction of New Observations Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-6: Prediction of New Observations Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-6: Prediction of New Observations Figure 11-8 Figure 11-8 Scatter diagram of oxygen purity data from Example 11-1 with fitted regression line, 95% prediction limits (outer lines), and 95% confidence limits on  Y|x0. 51

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-7: Adequacy of the Regression Model Fitting a regression model requires several assumptions. 1.Errors are uncorrelated random variables with mean zero; 2.Errors have constant variance; and, 3.Errors be normally distributed. The analyst should always consider the validity of these assumptions to be doubtful and conduct analyses to examine the adequacy of the model 52

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-7: Adequacy of the Regression Model Residual Analysis The residuals from a regression model are e i = y i - ŷ i, where y i is an actual observation and ŷ i is the corresponding fitted value from the regression model. Analysis of the residuals is frequently helpful in checking the assumption that the errors are approximately normally distributed with constant variance, and in determining whether additional terms in the model would be useful. 53

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-7: Adequacy of the Regression Model Residual Analysis Figure 11-9 Patterns for residual plots. (a) satisfactory, (b) funnel, (c) double bow, (d) nonlinear. [Adapted from Montgomery, Peck, and Vining (2006).] 54

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-7: Adequacy of the Regression Model Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-7: Adequacy of the Regression Model Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-7: Adequacy of the Regression Model Example 11-7 Figure Normal probability plot of residuals, Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-7: Adequacy of the Regression Model Example 11-7 Figure Plot of residuals versus predicted oxygen purity, ŷ, Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-7: Adequacy of the Regression Model Coefficient of Determination (R 2 ) The quantity is called the coefficient of determination and is often used to judge the adequacy of a regression model. 0  R 2  1; We often refer (loosely) to R 2 as the amount of variability in the data explained or accounted for by the regression model. 59

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-7: Adequacy of the Regression Model Coefficient of Determination (R 2 ) For the oxygen purity regression model, R 2 = SS R /SS T = / = Thus, the model accounts for 87.7% of the variability in the data. 60

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-8: Correlation 61

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-8: Correlation We may also write: 62

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-8: Correlation It is often useful to test the hypotheses The appropriate test statistic for these hypotheses is Reject H 0 if |t 0 | > t  /2,n-2. 63

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-8: Correlation The test procedure for the hypothesis where  0  0 is somewhat more complicated. In this case, the appropriate test statistic is Reject H 0 if |z 0 | > z  /2. 64

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-8: Correlation The approximate 100(1-  )% confidence interval is 65

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-8: Correlation Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-8: Correlation Figure Scatter plot of wire bond strength versus wire length, Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-8: Correlation Minitab Output for Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-8: Correlation Example 11-8 (continued) 69

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-8: Correlation Example 11-8 (continued) 70

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-8: Correlation Example 11-8 (continued) 71

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-9: Transformation and Logistic Regression 72

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-9: Transformation and Logistic Regression Example 11-9 Table 11-5 Observed Values and Regressor Variable for Example

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-9: Transformation and Logistic Regression Example 11-9 (Continued) 74

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-9: Transformation and Logistic Regression Example 11-9 (Continued) 75

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. 11-9: Transformation and Logistic Regression Example 11-9 (Continued) 76

© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger. Important Terms & Concepts of Chapter 11 Analysis of variance test in regression Confidence interval on mean response Correlation coefficient Empirical model Confidence intervals on model parameters Intrinsically linear model Least squares estimation of regression model parameters Logistics regression Model adequacy checking Odds ratio Prediction interval on a future observation Regression analysis Residual plots Residuals Scatter diagram Simple linear regression model standard error Statistical test on model parameters Transformations 77Chapter 11 Summary