If the rank of X is equal to p then the columns of X are linearly independent and there is a unique way of representing If the rank of X is strictly less than p then there is no unique way of representing
Comment: Usually the situation where the rank of X, r < p, arises in the following instances. 1.The design of the study (the choice of the values of X 1, X 2, …, X p ) was not careful enough to ensure that X had full rank. 2.Observations were missing causing the model to be altered Elements of are deleted along with corresponding rows of X, reducing the number of linear independent rows from p to r. 3.The model was defined in such a way that: i = 1 x i1 + 2 x i2 + … + p x ip is not uniquely determined by 1, 2, …, p.
Two Basic approaches: 1.Impose p – r linear restrictions on the parameters This allows us to reduce the number of parameters to r. will have a unique representation if the p – r restrictions are added. This technique is usually used with ANOVA, MANOVA, ANACOVA models. 2.Live with the singularity. Restrict our attention to linear combinations of the parameters that have unique estimators. The two approaches are essentially the same (lead to the same conclusions).
Recall: Linear Equations theory Consider the system of linear equations M (A), the linear space spanned by the columns of A
Then the general solution to the system of linear equations is
Maximum Likelihood Estimation leads to the system of linear equations p equations in p unknowns called the Normal equations
Theorem The Normal equations are consistent. Proof It can be shown that M (XX) M (X) M(X)M(X) M(XX)M(XX) Theorem The general solution to the Normal equations is
Theorem is the same for all solutions of the Normal equations Proof: the general solution to the Normal equations is Since M (XX) M (X) there exists a p × n matrix L such that X = XXL or X = LXX
Example: One-way ANOVA (Analysis of Variance) Suppose we have k normal populations Let y i1, y i2, …, y in denote a sample of n from Let ij = y ij - ( i ), then i1, i2, …, in denotes a sample of n from distribution. where 11, 12, …, kn are kn independent observations from N(0, 2 ) distribution.
= then linear space spanned by the vectors M(X)M(X)
Thus the estimable parameters are of the form: The common approach is to add the restriction This reduces the number of parameters to k, and converts the model to full rank.
Properties of estimable functions: 1.All linear functions are estimable M (X) = E p = p-dimensional Euclidean space (which contains all p-dimansional vectors) Proof If rank(X) = p then 2. is estimable if Proof (unique for all solutions of the normal equations) Hence is estimable.
3. If and are estimable then Proof since and are estimable then