MathematicalMarketing Slide 5.1 OLS Chapter 5: Ordinary Least Square Regression We will be discussing  The Linear Regression Model  Estimation of the Unknowns in the Regression Model  Some Special Cases of the Model

MathematicalMarketing Slide 5.2 OLS The Basic Regression Model The model expressed in scalars The model elaborated in matrix terms A succinct matrix expression for it y = X  + e

MathematicalMarketing Slide 5.3 OLS Prediction Based on a Linear Combination The E(y) is given by Key Question: How Do We Get Values for the  vector? Together with the previous slide, we can say that

MathematicalMarketing Slide 5.4 OLS Parameter Estimation of  We will cover two philosophies of parameter estimation, the least squares principle and maximum likelihood. Each of these has the following steps:  Pick an objective function to optimize  Find the derivative of f with respect to the unknown parameters  Find values of the unknowns where that derivative is zero The two differ in the first step. The least squares principle would have us pick f = e′e as our objective function which we will minimize.

MathematicalMarketing Slide 5.5 OLS We Wish to Minimize e′e We want to minimize f = e′e over all possible values of elements in the vector  The function f depends on 

MathematicalMarketing Slide 5.6 OLS Minimizing ee Cont'd These two terms are the same so f = yy – 2yX  +  XX .

MathematicalMarketing Slide 5.7 OLS What Is the Derivative? The derivative of a sum is equal to the sum of the derivatives, so we can handle it in pieces. Now we need to determine the derivative, and set it to a column of k zeroes. Our objective function is the sum f = yy – 2yX  +  XX 

MathematicalMarketing Slide 5.8 OLS A Quickie Review of Some Derivative Rules The derivative of a quadratic form: The derivative of a linear combination: The derivative of a transpose: The derivative of a constant:

MathematicalMarketing Slide 5.9 OLS The Derivative of the Sum Is the Sum of the Derivatives f = yy – 2yX  +  XX  (The derivative of a quadratic form) (The derivative of a constant) (The derivative of a linear combination and the derivative of a transpose)

MathematicalMarketing Slide 5.10 OLS Beta Gets a Hat Add these all together and set equal to zero And with some algebra (This one has a name) (This one has a hat)

MathematicalMarketing Slide 5.11 OLS Is Our Formula Any Good?

MathematicalMarketing Slide 5.12 OLS What Do We Really Mean by “Good”? as n  . does not depend on  is smaller than other estimators Unbiasedness Consistency Sufficiency Efficiency

MathematicalMarketing Slide 5.13 OLS Two Key Assumptions The behavior of the estimator is driven by the error input According to the Gauss-Markov Assumption, V(e) =  looks like

MathematicalMarketing Slide 5.14 OLS The Likelihood Principle Consider a sample of 3: 10, 11 and 12. What is  ?

MathematicalMarketing Slide 5.15 OLS Maximum Likelihood According to ML, we should pick values for the parameters that maximize the probability of the sample. To do this we need to follow these steps:  Derive the probability of an observation  Assuming independent observations, calculate the likelihood of the sample using multiplication  Take the log of the sample likelihood  Derive the derivative of the log likelihood with respect to the parameters  Figure out what the parameters must be so that the derivative is equal to a vector of zeroes With linear models we can do this analytically using algebra With non-linear models we sometimes have to use brute force hill “climbing” routines

MathematicalMarketing Slide 5.16 OLS The Probability of Observation y i

MathematicalMarketing Slide 5.17 OLS Multiply Out the Probability of the Whole Sample

MathematicalMarketing Slide 5.18 OLS ln exp(a) = a Take the Log of the Sample Likelihood ln 1 = 0 ln a b = b ln a

MathematicalMarketing Slide 5.19 OLS Figure Out the Derivative and Set Equal to Zero From here we are just a couple of easy algebraic steps away from the normal equations, and the least squares formula, If ML estimators exist for a model, that estimator is guaranteed to be consistent, asymptotically normally distributed and asymptotically efficient.

MathematicalMarketing Slide 5.20 OLS Sums of Squares SS Error = yy - yX(XX) -1 Xy SS Error = SS Total - SS Predictable

MathematicalMarketing Slide 5.21 OLS Using the Covariance Matrix Instead of the Raw SSCP Matrix The covariance matrix of the y and x variables looks like

MathematicalMarketing Slide 5.22 OLS Using Z Scores We can calculate a standard version of using Z scores Or use the correlation matrix of all the variables

MathematicalMarketing Slide 5.23 OLS The Concept of Partialing Imagine that we divided up the x variables into two sets: so that the Beta's were also divided the same way: The model becomes

MathematicalMarketing Slide 5.24 OLS The Normal Equations The normal equations would then be Subtract from the first equation gives us or

MathematicalMarketing Slide 5.25 OLS The Estimator for the First Set Solving for the Estimator for the first set yields Factoring What is this? The usual formula

MathematicalMarketing Slide 5.26 OLS The P and M Matrices Define P = X(XX) -1 X and define M = I – P, = I - X(XX) -1 X.

MathematicalMarketing Slide 5.27 OLS An Intercept Only Model

MathematicalMarketing Slide 5.28 OLS The Intercept Only Model 2 The model becomes

MathematicalMarketing Slide 5.29 OLS The P Matrix in This Case

MathematicalMarketing Slide 5.30 OLS The M Matrix

MathematicalMarketing Slide 5.31 OLS Response Surface Models: Linear vs Quadratic

MathematicalMarketing Slide 5.32 OLS Response Surface Models: The Sign of the Betas