1. Statistical Modelling Chapter XI 1 XII. Justifying the ANOVA- based hypothesis test XII.AThe sources for an ANOVA XII.BThe sums of squares for an ANOVA XII.CDegrees of freedom of the sums of squares for an ANOVA XII.DExpected mean squares for an ANOVA XII.EThe distribution of the F statistics for an ANOVA XII.FApplication of theory for ANOVA- based hypothesis test to an example

2. Statistical Modelling Chapter XI 2 Introduction In chapter VI we gave a procedure for determining an ANOVA table. In this chapter we look at the theory that justifies this procedure. In particular, examine basis of the SSqs, dfs, E[MSq]s and the distribution of the F statistic.

3. Statistical Modelling Chapter XI 3 XII.AThe sources for an ANOVA Various methods for determining the sources to include in an ANOVA table. Perhaps most popular is to try to identify the situation that is closest to yours in a textbook. Big disadvantage is not exact match and use wrong analysis. Our approach based on dividing the factors into unrandomized and randomized factors

4. Statistical Modelling Chapter XI 4 Our approach 1.identify the unrandomized and randomized factors; 2.determine the structure formulae that describe the crossing and nesting relations a) amongst the unrandomized factors and b) amongst the randomized factors (and in some cases between the randomized and some unrandomized factors); 3.expand the structure formulae to obtain two sets of sources; 4.form the lines in the ANOVA table by a) listing the unrandomized sources in the table, b) entering the randomized sources indented under the appropriate unrandomized sources, and c) adding Residuals for unrandomized sources where there are df left over. This extracts the key features of the experiment arising from the randomization and captures these in the structure formulae for use in determining the terms in the analysis. The ANOVA table produced using this method displays the confounding arising from the randomization, something that traditional tables do not do.

5. Statistical Modelling Chapter XI 5 XII.BThe SSqs for an ANOVA Derivation of SSqs, given the sources in the ANOVA, is based on the S matrices. For a structure formula, one must identify the generalized factors corresponding to the sources derived from that formula. For each generalized factor there is an S that specifies the units that are related in having the same level of that generalized factor. For example, S Blocks has a one for every pair of units that occur in the same block. Generally have two sets of relationship (S) or, equivalently, mean- operator matrices (M  (n/f) -1 S  g -1 S): one for the unrandomized factors and one for the randomized factors. (relationship matrices also called summation matrices) Now a set of such matrices forms the basis for an algebra that is called a relationship algebra. Associated with a set of Ss is a set of Qs, again one Q for each generalized factor, that are the idempotents of the relationship algebra and provide the quadratic-form matrices for the analysis. Under certain conditions, met by all our examples, a set of Qs derived from a structure formula forms a complete set of mutually- orthogonal idempotents (CSMOI — see defn XI.10). So we form SSqs of the projections of the data vector into the subspaces projected onto by the Qs.

6. Statistical Modelling Chapter XI 6 Obtaining expressions for Qs and Ms Rule VI.6 used to obtain expressions for Qs in terms of Ms. Following rule used to obtain expressions for the S (& Ms) as the direct products of I and J matrices. (generalizes Rule XI.1) Rule XII.1: The S for a generalized factor, formed as the subset of a set of s equally-replicated factors that uniquely index the units, –is the direct product of s matrices, provided the s factors are arranged in standard order. Taking the s factors in the sequence specified by the standard order, –for each factor in the generalized factor include an I matrix in the direct product and –J matrices for those that are not. The order of a matrix in the direct product is equal to the number of levels of the corresponding factor. The M matrix is the same direct product of I and J matrices, except that each J is multiplied by the reciprocal of its order. Rule applies to factors from the unrandomized structure.

7. Statistical Modelling Chapter XI 7 Ms are symmetric and idempotent Prove this useful result for those that can be expressed as the direct product of I and J matrices in the next lemma. Lemma XII.1: Let M be the direct product of I and J matrices, the latter multiplied by the reciprocal of their order, as prescribed in rule XII.1. Then M is symmetric and idempotent. Proof: Since (A  B)'  A'  B' and I and J are both symmetric, M must also be symmetric. Since (A  B)(C  D)  (AC  BD) and in particular (A  B)(A  B)  (A 2  B 2 ), M 2 must be the direct product of I and the square of J matrices, each J multiplied by the reciprocal of its order. Consequently, M 2 and M will be same direct product of I and J matrices, latter multiplied by the reciprocal of their order.

8. Statistical Modelling Chapter XI 8 XII.CDegrees of freedom of the SSqs for an ANOVA Definition XII.1: The degrees of freedom of a sum of squares is the rank of the idempotent of its quadratic form. That is the degrees of freedom of Y'QY is given by rank(Q). The following definition and lemma establish that the trace of an idempotent is the same as rank and hence its df. Definition XII.2: The trace of a square matrix is the sum of its diagonal elements. Lemma XII.2: For B idempotent, trace(B)  rank(B). Proof: The proof is based on the following facts: –the trace of a matrix is equal to the sum of its eigenvalues; –the rank of a matrix is equal to the number of nonzero eigenvalues; and –the eigenvalues of an idempotent can be proved to be either 1 or zero and so sum = number.

9. Statistical Modelling Chapter XI 9 Results on traces of matrices Clearly, the dfs can be established by determining the trace of a matrix so some results on the traces. Lemma XII.3: Let c be a scalar and A, B and C be matrices. Then, when the appropriate operations are defined, we have –trace(A')  trace(A) –trace(cA)  c  trace(A) –trace(A + B)  trace(A) + trace(B) –trace(AB)  trace(BA) –trace(ABC)  trace(CAB)  trace(BCA) –trace(A  B)  trace(A)  trace(B) –trace(A'A)  0 if and only if A  0.

10. Statistical Modelling Chapter XI 10 Trace of a Q Each Q matrix is a linear combination of M matrices so that –the trace(Q) will be the same linear combination of the traces of the Ms. In the next lemma we prove that the trace of an M is equal to the no. of levels in its generalized factor, –when corresponding summation matrix is direct product of I and Js. Lemma XII.4: Let M F be a mean operator matrix for a generalized factor F that has f levels each replicated n/f = g times and S F be the corresponding summation matrix that is direct product of I and Js. Then trace(M F )  f. Proof: Now trace(M F )  g -1 trace(S F ). But S F is the direct product of I and Js. However, trace(A  B)  trace(A)  trace(B) and the traces of I and J matrices are equal to their orders. As the product of the orders of the I and Js for any S must be n, trace(S F )  n and so trace(M F )  g -1 trace(S F )  n/g  f.

11. Statistical Modelling Chapter XI 11 XII.DExpected mean squares for an ANOVA As has been previously stated, the E[MSq]s are just the average or mean value of the MSqs under sampling from a population whose Y variables behave as described by the linear model. –i.e. the true mean value of a Mean Square –It depends on the model parameters To derive the expected values, we note that the general form of an MSq is a quadratic form divided by a df, Y'QY/. So we first establish an expression for the expectation of any quadratic form.

12. Statistical Modelling Chapter XI 12 Expectation of a quadratic form Theorem XII.1: Let Y be an n  1 vector of random variables with E[Y]   and var[Y]  V, where  is a n  1 vector of expected values and V is an n  n matrix. Let A an n  n matrix of real numbers. Then E[Y'AY]  trace(AV) +  'A .

13. Statistical Modelling Chapter XI 13 Proof of theorem XII.1 Firstly note that, as Y'AY is a scalar, Y'AY  trace(Y'AY) Also, recall that trace of matrix products are cyclically commutative. Thus, E[Y'AY]  E[trace(Y'AY)]  E[trace(AYY')]. Now, for an n  n matrix Z Also, lemma XI.5 (E[Y] results) states that E[AY]  AE[Y] and this can be extended to E[AYY']  AE[YY']. Hence, E[trace(AYY')]  trace(E[AYY'])  trace(AE[YY']). Now we require an expression for E[YY'] which we obtain by considering definition I.7 of V  E[(Y   ) (Y   )']. From this

14. Statistical Modelling Chapter XI 14 Proof of theorem XII.1 (continued) Now, E[  ]   as the elements of  are population quantities and are constants with respect to expectation. Thus E[Y  ']  E[Y]  '   '  E[  Y]. Hence, V  E[YY']  E[Y  ']  E[  Y] +  '  E[YY']   ' so that E[YY']  V +  '. This leads to

15. Statistical Modelling Chapter XI 15 Theorem for E[MSq] Theorem XII.2: Let Y be an n  1 vector of random variables with E[Y]   and var[Y]  V, where  is a n  1 vector of expected values and V is an n  n matrix. Let Y'QY/ be the mean square where Q is an n  n symmetric, idempotent matrix and is the degrees of freedom of the sums of squares. Then E[Y'QY/ ]  (trace(QV) +  'Q  /. Proof: Since is a constant, E[Y'QY/ ]  E[Y'QY]/ and the result follows straightforwardly using theorem XII.1. So to derive E[MSq] for a particular source under a specific model, one substitutes –the Q matrix for the source and –the  and V for the model into the expression E[Y'QY/ ]  (trace(QV) +  'Q  / given by theorem XII.2.

16. Statistical Modelling Chapter XI 16 XII.EThe distribution of the F statistics for an ANOVA Each F statistic in an ANOVA is used to assess whether or not a null hypothesis is likely to be true by determining if the value of our test statistic is unlikely when H 0 is true. To do this we need to establish the sampling distribution of the test statistic when the null hypothesis is true. Test statistic is of the following general form It is the ratio of two MSqs each of which is a quadratic form divided by its df. We will in the following three theorems: 1.Give the distribution of a quadratic form. 2.Establish the relationship between several quadratic forms. 3.Obtain the distribution of the ratio of two independent quadratic forms.

17. Statistical Modelling Chapter XI 17 Distribution of a quadratic form Theorem XII.3: Let A be an n  n symmetric matrix of rank and Y be an n  1 normally distributed random vector with E[AY]  0, var[Y]  V and E[Y'AY/ ] . Then (1/ )Y'AY follows a chi-squared distribution with degrees of freedom if and only if A is idempotent. Proof: not given The chi-square probability distribution function for the random variable U is: Probability distribution function for

18. Statistical Modelling Chapter XI 18 Relationships between idempotents Theorem XII.4: Let Y be an n  1 vector of random variables with E[Y]  X  and var[Y]  V. Let Y'A 1 Y, Y'A 2 Y, …, Y'A h Y be a collection of h quadratic forms where, for each i  1, 2, …, h, –A i is symmetric, of rank i, –E[A i Y]  0 and –E[Y'A i Y/ i ]  i. If any two of the following three statements are true (any two implies the other), 1.All A i are idempotent 2. is idempotent 3. A i A j  0, i  j then not only does (1/ i )Y'A i Y, for each i, follows a chi- squared distribution with i degrees of freedom as Theorem XII.3 establishes, but –Y'A i Y are independent for i  j and – where denotes the rank of. Proof: not given

19. Statistical Modelling Chapter XI 19 Distribution of the ratio of two independent quadratic forms Theorem XII.5: Let U 1 and U 2 be two random variables distributed as chi-squares with 1 and 2 degrees of freedom. Then, provided U 1 and U 2, are independent, the random variable is distributed as Snedecor's F with 1 and 2 degrees of freedom. Proof: not given

20. Statistical Modelling Chapter XI 20 F distribution for W Probability distribution function for F 3,46

21. Statistical Modelling Chapter XI 21 XII.FApplication of theory for ANOVA-based hypothesis test to an example a)The sources for an ANOVA b)The sums of squares for an ANOVA c)Degrees of freedom of the sums of squares for an ANOVA d)Expected mean squares for an ANOVA e)The distribution of the F statistics for an ANOVA

22. Statistical Modelling Chapter XI 22 Example XII.1 Randomized Complete Block Design a)The sources for an ANOVA 1.For the RCBD, the unrandomized factors are Blocks and Units and the randomized factors are Treatments. 2.The unrandomized structure formula is b Blocks/t Units and the randomized structure formula t Treatments. 3.The unrandomized sources are Blocks and Units[Blocks] and the randomized source is Treatments. 4.The sources for the ANOVA table for this experiment are:

23. Statistical Modelling Chapter XI 23 Example XII.1 Randomized Complete Block Design (continued) b)The sums of squares for an ANOVA Generalized factors –unrandomized: (G), Blocks and Blocks  Units and –randomized: (G) and Treatments. So Qs are –unrandomized: Q G, Q B and Q BU and –randomized: Q G and Q T. Since each of these is derived from a structure formula consisting of only nesting and crossing operators, they each form a CSMOI.

24. Statistical Modelling Chapter XI 24 Example XII.1 Randomized Complete Block Design (continued) b)The sums of squares for an ANOVA (continued) The Hasse diagrams of generalized-factor marginalities, that includes expressions for the Qs in terms of the Ms, are as follows:

25. Statistical Modelling Chapter XI 25 Example XII.1 Randomized Complete Block Design (continued) b)The sums of squares for an ANOVA (continued) If the data is ordered in standard order for Blocks and Treatments and if Units are given the same numbering as Treatments (this doesn't affect the analysis), rule XII.1 (S, M as direct products) yields: –M BU  I b  I t, and ; – and. Latter requires inclusion of a factor for replicates, here Blocks; this factor never occurs in its generalized factors. So the ANOVA table with SSqs added is:

26. Statistical Modelling Chapter XI 26 Example XII.1 Randomized Complete Block Design (continued) c)Degrees of freedom of the SSqs for an ANOVA The following theorem establishes that the degrees of freedom of the sums of squares are as given in the analysis of variance table. Theorem XII.6: Let quadratic forms for the SSqs be –where M BU  I b  I t,, and Their degrees of freedom are n  1, b  1, b(t  1), t  1 and (b  1)(t  1), respectively, –where n is the no. of observations, b is the no. of blocks and t is the no. of treatments.

27. Statistical Modelling Chapter XI 27 Proof of theorem XII.6 (continued) First we establish that all Q matrices are symmetric and idempotent so that we can utilise lemma XII.2 to conclude that the ranks of the Q matrices are equal to their traces. Now Q B, Q BU and Q T are symmetric and idempotent as they are in the complete sets of mutually-orthogonal idempotents. It remains to demonstrate that Q U and are symmetric and idempotent.

28. Statistical Modelling Chapter XI 28 Proof of theorem XII.6 (continued) Firstly, from lemma I.1 (transpose properties), (cA + dB)'  cA' + dB' and we have from lemma XII.1 that the Ms are symmetric and idempotent so that Now M BU  I n so that That is Q U is symmetric and idempotent.

29. Statistical Modelling Chapter XI 29 Proof of theorem XII.6 (continued) Secondly, as each M is symmetric. Further,

30. Statistical Modelling Chapter XI 30 Proof of theorem XII.6 (continued) Now and so

31. Statistical Modelling Chapter XI 31 Proof of theorem XII.6 (continued) Consequently lemma XII.2 (rank  trace) applies to the Qs so that the ranks of all the Q matrices are equal to their trace. We also note that using lemma XII.4 (trace(M F )  f) we have that Since, from lemma XII.3 (traces) we have that

32. Statistical Modelling Chapter XI 32 ANOVA table with degrees of freedom added

33. Statistical Modelling Chapter XI 33 d)Expected mean squares for an ANOVA We now prove the E[MSq]s for the case of Blocks random. But first a useful lemma. Lemma XII.5: Let Then

34. Statistical Modelling Chapter XI 34 Proof of lemma XII.5 First we have so that

35. Statistical Modelling Chapter XI 35 Theorem XII.7 Then, the expected mean squares are where,,  j is the jth element of the t-vector , b is the number of blocks and t is the number of treatments. Let

36. Statistical Modelling Chapter XI 36 For E[SS B /(b  1)], we use theorem XII.2 (E[Y'QY/ ]  (trace(QV) +  'Q  / ) and that it can be shown that Q B M BU  Q B and Q B M B  Q B to obtain the following expressions: Proof of theorem XII.7 First note that we can write

37. Statistical Modelling Chapter XI 37 Proof of theorem XII.7 (continued) Now from theorem XII.6 (RCDD dfs) we have that trace(Q B )  (b  1) and from lemma XII.5 Hence, the expected mean square is The proof that is left as an exercise for you.

38. Statistical Modelling Chapter XI 38 Proof of theorem XII.7 (continued) Now, for, again we use theorem XII.2 (E[MSq]s) and also that it can be shown that so that we have

39. Statistical Modelling Chapter XI 39 Proof of theorem XII.7 (continued) Now from theorem XII.6 (RCBD dfs) we have that and from lemma XII.5 so that as claimed.

40. Statistical Modelling Chapter XI 40 ANOVA table with E[MSq]s added

41. Statistical Modelling Chapter XI 41 e)The distribution of the F statistics for an ANOVA We next derive the sampling distribution of the F- statistics for testing treatment and block differences. The following theorems involve establishing the sampling distribution of the F-statistics under these models. Note that the model –under the null hypothesis of no treatment effects is E[Y]  X G  and –while under the null hypothesis of no block variation is E[Y]  X T  and

42. Statistical Modelling Chapter XI 42 Theorem XII.8 Let Then, the ratio of these two mean squares, given by is distributed as a Snedecor's F with (t  1) and (b  1)(t  1) degrees of freedom. where

43. Statistical Modelling Chapter XI 43 Proof of theorem XII.8 We have to show that 1. and under the expectation and variation models above, so that theorem XII.3 (dist n Y'AY) can be invoked to conclude that follow chi-square distributions; 2.the quadratic forms and are independent as in theorem XII.4 (several Y'A i Ys); 3.then theorem XII.5 (F distribution) can be invoked to obtain the distribution of the F test statistic.

44. Statistical Modelling Chapter XI 44 Proof of theorem XII.8 (continued) Firstly,

45. Statistical Modelling Chapter XI 45 Proof of theorem XII.8 (continued) Also, using theorem XII.2 (E[MSq]s) and –as Q T M BU  Q T I n  Q T –Q T M B  (M T  M G )M B  M G  M G  0 (M B M T  M T M B  M B M G  M G M B  M G M T  M T M G  M G from the proof of theorem XII.6 {RCBD dfs}) –Q T X G  0 (see above in this proof), and –trace(Q T )  t  1 (from theorem XII.6 {RCBD dfs}), we have

46. Statistical Modelling Chapter XI 46 Proof of theorem XII.8 (continued) Now, using theorem XII.2 (E[MSq]s), and as – (M B M T  M T M B  M B M G  M G M B  M G M T  M T M G  M G from the proof of theorem XII.6 {RCBD dfs}) – (see above in this proof), and – (from theorem XII.6 {RCBD dfs}), we have

47. Statistical Modelling Chapter XI 47 Proof of theorem XII.8 (continued) Secondly, to show that and are independent quadratic forms we have to show that Q T and meet two of the three conditions outlined in theorem XII.4 (several Y'A i Ys). As outlined in the proof of theorem XII.6 (RCBD dfs), Q T and are idempotent so that condition 1 is met. For condition 3, we require that. Now,

48. Statistical Modelling Chapter XI 48 Proof of theorem XII.8 (continued) Thirdly, theorem XII.5 (F distribution) means that, as are distributed as independent chi-squares with degrees of freedom (t  1) and (b  1)(t  1), respectively, then follows an F distribution with (t  1) and (b  1)(t  1) degrees of freedom.

49. Statistical Modelling Chapter XI 49 Theorem XII.9 Let where Then, the ratio of these two mean squares, given by is distributed as a Snedecor's F with (b  1) and (b  1)(t  1) degrees of freedom. Proof: parallels that for the treatment differences.

50. Statistical Modelling Chapter XI 50 XII.GExercises Ex. 12-1 requires the proofs of some properties of Q and M matrices for the RCBD. Ex. 12-2 asks you to derive expressions for Q and M matrices for a CRD and to prove that the Residual operator is symmetric and derive its df. Ex. 12-3 involves the proof of the E[MSq] for the RCBD. Ex. 12-4 asks you to prove that the ratio of the Rows and Residual Msq for a Latin square is distributed as Snedecor's F under the null hypothesis.