Download presentation

Presentation is loading. Please wait.

Published byHugo Ward Modified over 4 years ago

1
The General Linear Model

2
The Simple Linear Model Linear Regression

3
Suppose that we have two variables 1. Y – the dependent variable (response variable) 2. X – the independent variable (explanatory variable, factor)

4
X, the independent variable may or may not be a random variable. Sometimes it is randomly observed. Sometimes specific values of X are selected

5
The dependent variable, Y, is assumed to be a random variable. The distribution of Y is dependent on X The object is to determine that distribution using statistical techniques. (Estimation and Hypothesis Testing)

6
These decisions will be based on data collected on both variable Y (the dependent variable) and X (the independent variable). Let (x 1, y 1 ), (x 2, y 2 ), …,(x n, y n ) denote n pairs of values measured on the independent variable (X) and the dependent variable (Y) The scatterplot: The graphical plot of the points: (x 1, y 1 ), (x 2, y 2 ), …,(x n, y n )

7
Assume that we have collected data on two variables X and Y. Let ( x 1, y 1 ) ( x 2, y 2 ) ( x 3, y 3 ) … ( x n, y n ) denote the pairs of measurements on the on two variables X and Y for n cases in a sample (or population)

8
1.independent random variables. 2.Normally distributed. 3.Have the common variance, . 4.The mean of y i is i = + x i The assumption will be made that y 1, y 2, y 3 …, y n are Data that satisfies the assumptions above is to come from the Simple Linear Model

9
Each y i is assumed to be randomly generated from a normal distribution with mean i = + x i and standard deviation . yiyi + x i xixi

10
When data is correlated it falls roughly about a straight line.

11
The density of y i is: The joint density of y 1,y 2, …,y n is:

12
Estimation of the parameters the intercept the slope the standard deviation (or variance 2 )

13
The Least Squares Line Fitting the best straight line to “linear” data

14
Let Y = a + b X denote an arbitrary equation of a straight line. a and b are known values. This equation can be used to predict for each value of X, the value of Y. For example, if X = x i (as for the i th case) then the predicted value of Y is:

15
Define the residual for each case in the sample to be: The residual sum of squares (RSS) is defined as: The residual sum of squares (RSS) is a measure of the “goodness of fit of the line Y = a + bX to the data

16
One choice of a and b will result in the residual sum of squares attaining a minimum. If this is the case than the line: Y = a + bX is called the Least Squares Line

17
To find the Least Squares estimates, a and b, we need to solve the equations: and

18
Note: or and

19
Note: or

20
Hence the optimal values of a and b satisfy the equations: and From the first equation we have: The second equation becomes:

21
Solving the second equation for b: and where and

22
Note: and Proof

23
Summary: Slope and intercept of the least squares Line and

24
Maximum Likelihood Estimation of the parameters the intercept the slope the standard deviation

25
Recall The joint density of y 1,y 2, …,y n is: = the Likelihood function

26
the log Likelihood function To find the maximum Likelihood estimates of , and we need to solve the equations:

27
These are the same equations for the least squares line which have solution: becomes

28
The third equation: becomes

29
Summary: Maximum Likelihood Estimates and

30
A computing formula for the estimate of 2 and Hence

31
Now Hence

32
It also can be shown that Thus, the maximum likelihood estimator of 2, is a biased estimator of 2. This estimator can be easily converted into an unbiased estimator of 2 by multiply by the ratio n/(n – 2)

33
Estimators in Linear Regression and

34
The major computation is :

35
Computing Formulae:

36
Application Of Statistical Theory to simple Linear Regression We will now use statistical theory to prove optimal properties of the estimators. Recall, the joint density of y 1,y 2, …,y n is:

40
Also and Thus all three estimators are functions of the set of complete sufficient statistics. If they are also unbiased then they are Uniform Minimum Variance Unbiased (UMVU) estimators (using the Lehmann-Scheffe theorem)

41
and We have already shown that s 2 is an unbiased estimator of 2. We need only show that: are unbiased estimators of and .

42
Thus are unbiased estimators of and .

43
Also Thus

44
The General Linear Model

45
Consider the random variable Y with 1. E[Y] = 1 X 1 + 2 X 2 +... + p X p (alternatively E[Y] = 0 + 1 X 1 +... + p X p, intercept included) and 2. var(Y) = 2 where 1, 2,..., p are unknown parameters and X 1,X 2,..., X p are nonrandom variables. Assume further that Y is normally distributed.

46
Thus the density of Y is: f(Y| 1, 2,..., p, 2 ) = f(Y| , 2 )

47
Now suppose that n independent observations of Y, (y 1, y 2,..., y n ) are made corresponding to n sets of values of (X 1,X 2,..., X p ) - (x 11,x 12,..., x 1p ), (x 21,x 22,..., x 2p ),... (x n1,x n2,..., x np ). Then the joint density of y = (y 1, y 2,... y n ) is: f(y 1, y 2,..., y n | 1, 2,..., p, 2 ) =

49
Thus is a member of the exponential family of distributions And is a Minimal Complete set of Sufficient Statistics.

50
Matrix-vector formulation The General Linear Model

53
Geometrical interpretation of the General Linear Model

54
Geometical interpretation of the General Linear Model

55
Estimation The General Linear Model

56
Least squares estimates of Let

57
The Equations for the Least squares estimates

58
Written out in full These equations are called the Normal Equations

59
Matrix development of the Normal Equations Now The Normal Equations

60
Summary (the Least Squares Estimates) The Least Squares estimates satisfy The Normal Equations

61
Note: Some matrix properties Rank rank(AB) = min(rank(A), rank(B)) rank(A) ≤ min(# rows of A, # cols of A ) rank(A) = rank(A) Consider the normal equations

62
then the solution to the normal equations

63
Maximum Likelihood Estimation General Linear Model

64
The General Linear Model

65
The Maximum Likelihood estimates of and 2 are the values that maximize or equivalently

66
This yields the system of linear equations (The Normal Equations)

67
yields the equation:

68
If [X'X] -1 exists then the normal equations have solution: and

69
Summary and

70
Comments The matrices are symmetric idempotent matrices also

71
Comments (continued)

72
Geometry of Least Squares

73
Example Data is collected for n = 15 cases on the variables Y (the dependent variable) and X 1, X 2, X 3 and X 4. The data and calculations are displayed on the next page:

76
Properties of The Maximum Likelihood Estimates Unbiasedness, Minimum Variance

77
Note: and Thus is an unbiased estimator of. Since is also a function of the set of complete minimal sufficient statistics, it is the UMVU estimator of. (Lehman-Scheffe)

78
Note: where In general

79
Thus: where

80
Thus:

81
Let Then Thus s 2 is an unbiased estimator of 2. Since s 2 is also a function of the set of complete minimal sufficient statistics, it is the UMVU estimator of 2.

82
Distributional Properties Least square Estimates (Maximum Likelidood estimates)

83
1.If then where A is a q × p matrix Recall ~

84
The General Linear Model and The Estimates

85
Theorem Proof

89
Finally and

90
Thus are independent. Summary

91
Example Data is collected for n = 15 cases on the variables Y (the dependent variable) and X 1, X 2, X 3 and X 4. The data and calculations are displayed on the next page:

95
Compare with SPSS output Estimates of the coefficients

96
The General Linear Model with an intercept

97
Consider the random variable Y with 1. E[Y] = 0 + 1 X 1 + 2 X 2 +... + p X p (intercept included) and 2. var(Y) = 2 where 1, 2,..., p are unknown parameters and X 1,X 2,..., X p are nonrandom variables. Assume further that Y is normally distributed.

98
The matrix formulation (intercept included) Then the model becomes Thus to include an intercept add an extra column of 1’s in the design matrix X and include the intercept in the parameter vector

99
The matrix formulation of the Simple Linear regression model

100
and Now

101
thus

102
Finally Thus

103
The Gauss-Markov Theorem An important result in the theory of Linear models Proves optimality of Least squares estimates in a more general setting

104
Assume the following model Linear Model We will not necessarily assume Normality. Consider the least squares estimate of is an unbiased linear estimator of

105
The Gauss-Markov Theorem Assume Consider the least squares estimate of, an unbiased linear estimator of and Let denote any other unbiased linear estimator of

106
Proof Now

107
Now is an unbiased estimator of if Also Thus

108
Thus The Gauss-Markov theorem states that is the Best Linear Unbiased Estimator (B.L.U.E.) of

109
Hypothesis testing for the GLM The General Linear Hypothesis

110
Testing the General Linear Hypotheses The General Linear Hypothesis H 0 :h 11 1 + h 12 2 + h 13 3 +... + h 1p p = h 1 h 21 1 + h 22 2 + h 23 3 +... + h 2p p = h 2... h q1 1 + h q2 2 + h q3 3 +... + h qp p = h q where h 11 h 12, h 13,..., h qp and h 1 h 2, h 3,..., h q are known coefficients. In matrix notation

111
Examples 1.H 0 : 1 = 0 2.H 0 : 1 = 0, 2 = 0, 3 = 0 3.H 0 : 1 = 2

112
Examples 4.H 0 : 1 = 2, 3 = 4 5.H 0 : 1 = 1/2( 2 + 3 ) 6.H 0 : 1 = 1/2( 2 + 3 ), 3 = 1/3( 4 + 5 + 6 )

113
TheLikelihood Ratio Test The joint density of is: The likelihood function The log-likelihood function

114
Defn (Likelihood Ratio Test of size ) Rejects H 0 : against the alternative hypothesis H 1 : . when and K is chosen so that

115
Note We will maximize. The Lagrange multiplier technique will be used for this purpose

116
We will maximize.

117
or finally or

118
Thus the equations for are Now or and

Similar presentations

© 2019 SlidePlayer.com Inc.

All rights reserved.

To make this website work, we log user data and share it with processors. To use this website, you must agree to our Privacy Policy, including cookie policy.

Ads by Google