1. A tale of randomization: randomization versus mixed model analysis for single and chain randomizations Chris Brien Phenomics & Bioinformatics Research Centre, University of South Australia. The Australian Centre for Plant Functional Genomics, University of Adelaide. This work was supported by the Australian Research Council.

2. A tale of randomization: outline 1.Once upon a time. 2.Randomization model for a single randomization. 3.Randomization analysis for a single randomization. 4.Randomization model for a chain of randomizations. 5.Randomization analysis for a chain of randomizations. 6.Some issues. 7.Conclusions. 2

3. 1.Once upon a time In the 70s I was a true believer: We are talking randomization inference. 3

4. Purism These books demonstrate that p-value from randomization analysis is approximated by p-value from analyses assuming normality for CRDs & RCBDs; Welch (1937) & Atiqullah (1963) show that true, provided the observed data actually conforms to the variance for the assumed normal model (e.g. homogeneity between blocks). 4 Kempthorne (1975):

5. Randomization analysis: what is it? A randomization model is formulated. It specifies the distribution of the response over all randomized layouts possible for the design. A test statistic is identified. I will use test statistics from parametric analyses (e.g. F-statistics). The value of the test statistic is computed from the data for: all possible randomized layouts, or a random sample (with replacement) of them randomization distribution of the test statistic, or an estimate; the randomized layout used in the experiment: the observed test statistic. The p-value is the proportion of all possible values that are as, or more, extreme than the observed test statistic. Different to a permutation test in that it is based on the randomization employed in the experiment. 5

6. Sex created difficulties … and time Preece (1982, section 6.2): Is Sex a block or a treatment factor? Semantic problem: what is a block factor? Often Sex is unrandomized, but is of interest – I believe this to be the root of the dilemma. If it is unrandomized, it cannot be tested using a randomization test (at all?). In longitudinal studies, Time is similar. Sites also. What about incomplete block designs with recombination of information? Missing values? Seems that not all inference possible with randomization analysis. 6

7. Fisher (1935, Section 21) first proposed randomization tests: It seems clear that Fisher intended randomization tests to be only a check on normal theory tests. 7

8. Fisher (1960, 7 th edition) added Section 21.1 that includes: Less intelligible test nonparametric test. 8

9. Conversion I became a modeller, BUT, I did not completely reject randomization inference. I have advocated randomization-based mixed models: a mixed model that starts with the terms that would be in a randomization model (Brien & Bailey, 2006; Brien & Demétrio, 2009). This allowed me to: test for block effects and block-treatment interactions; model longitudinal data. I comforted myself that when testing a model that has an equivalent randomization test, the former is an approximation to the latter and so robust. 9

10. More recently …. Cox, Hinkelmann and Gilmour pointed out, in the discussion of Brien and Bailey (2006), no one had so far indicated how a model for a multitiered experiment might be justified by the randomizations employed. Rosemary Bailey and I have been working for some time on the analysis of experiments with multiple randomizations, using randomization-based (mixed) models; Brien and Bailey (201?) details estimation & testing. I decided to investigate randomization inference for such experiments, but first single randomizations. 10

11. 2.Randomization model for a single randomization Additive model of constants: y w + X h where y is the vector of observed responses; w is the vector of constants representing the contributions of each unit to the response; and is a vector of treatment constants; X h is design matrix showing the assignment of treatments to units. Under randomization, i.e. over all allowable unit permutations applied to w, each element of w becomes a random variable, as does each element of y. Let W and Y be the vectors of random variables and so we have Y W + X h. The set of Y forms the multivariate randomization distribution, our randomization model. Now, we assume E R [W] 0 and so E R [Y] X h. 11

12. Randomization model (contd) Further, 12 is the set of generalized factors (terms) derived from a poset of factors on the units; H is the covariance between variables with the same levels of generalized factor H; H is the canonical component of excess covariance for H; H is the eigenvalue of V R for H and is its contribution to E[MSq]; B H, S H, and Q H are known matrices. This model has the same terms as a randomization-based mixed model (Brien & Bailey, 2006; Brien & Demétrio, 2009) However, the distributions differ.

13. Randomization by permutation of units & unit factors UnitBlocksUnitsTreatments 1111 2122 3211 4222 13 Permutations for an RCBD with b 2, k v 2. The allowable permutations are: those that permute the blocks as a whole, and those that permute the units within a block; there are b!(k!) b 2!(2!) 2 8. UnitBlocksUnitsTreatmentsPermutation 11114 21223 32111 42222 Permuted unitBlocksUnitsTreatmentsPermutationBlocksUnits 1111422 2122321 3211111 4222212 Equivalent to Treatments randomization 1, 2, 2, 1.

14. Null randomization distribution: RCBD Under the assumption of no treatment effects, Y * W +. In which case, the randomization distribution of Y * is termed the null randomization distribution Actual distribution obtained by applying each unit permutation to y: 14 PermutationY * 11 Y * 12 Y * 21 Y * 22 1y 11 y 12 y 21 y 22 2y 12 y 11 y 21 y 22 3y 11 y 12 y 22 y 21 4y 12 y 11 y 22 y 21 5 y 22 y 11 y 12 6y 21 y 22 y 12 y 11 7y 22 y 21 y 11 y 12 8y 22 y 21 y 12 y 11 Can show that 1 st & 2 nd order parameters of the distribution,, G, B and BU, are equal to sample statistics. For example, for all Y * ij : Y * ij for Unit j in Block i. The distribution of gives the distribution of W.

15. V R for the RCBD example The matrices in the expressions for are known. 15

16. 3.Randomization analysis for a single randomization Estimation and hypothesis testing based on the randomization distribution. Will focus on hypothesis testing. Propose to use I-MINQUE to estimate the s and use these estimates to estimate via EGLS. I-MINQUE yields the same estimates as REML, but without the need to assume a distributional form for the response. 16

17. Test statistics Have a set of idempotents specifying a treatment decomposition. For an R, to test H 0 : RX h 0, use a Wald F, a Wald test statistic divided by its numerator df: 17 Numerator is a quadratic form: (est) (var(est)) -1 (est). For an orthogonal design, F Wald is the same as the F from an ANOVA. Otherwise, it is a combined F test statistic. For nonorthogonal designs, an alternative test statistic is an intrablock F-statistic. For a single randomization, let Q H be the matrix for H that projects on the eigenspace of V from which RX h is to be estimated. Then The intrablock

18. Randomization distribution of the test statistic To obtain it: Apply, to the unit factors and y, but not the treatment factors, all allowable unit permutations for the design employed: effects a rerandomization of the treatments; Compute the test statistic for each allowable permutation; This set of values is the required distribution. Number of allowable permutations. For our RCBD, there are 8 permutations and so computing the 8 test statistics is easy. For b 10 and k 3, there are 1.4 x 10 35 not so easy. An alternative is random data permutation (Edgington, 1995): take a Monte Carlo sample of the permutations. 18

19. Null distribution of the test statistic under normality Under normality of the response, the null distribution of F Wald is: for orthogonal designs, an exact F-distribution; for nonorthogonal designs, an F-distribution asymptotically. Under normality of the response, the null distribution of an intrablock F-statistic is an exact F-distribution. 19

20. Wheat experiment in a BIBD (Joshi, 1987) Six varieties of wheat are assigned to plots arranged in 10 blocks of 3 plots. The efficiency factor for this design is 0.80. The ANOVA with the intrablock F and p: 20 plots tiertreatments tier sourced.f.sourced.f.MSFp-value Blocks9Varieties539.320.580.718 Residual467.591.17 Plots[B]20Varieties5231.294.020.016 Residual1557.53 F Wald 3.05 with p 0.035 ( 1 5, 2 19.1). Estimates: B 14.60 (p 0.403); BP 58.28.

21. 21 Test statistic distributions 50,000 randomly selected permutations of blocks and plots within blocks selected. Intrablock F-statisticCombined F-statistic Peak on RHS is all values 10.

22. 22 Combined F-statistic Part of the discrepancy between F- and the randomization distributions is that combined F-statistic is only asymptotically distributed as an F. Differs from Kenward & Rogers (1997) & Schaalje et al (2002) for nonorthogonal designs. Randomization distributionParametric bootstrap

23. Two other examples Rabbit experiment using the same BIBD (Hinkelmann & Kempthorne, 2008). 6 Diets assigned to 10 Litters, each with 3 Rabbits. Estimates: L 21.70 (p 0.002), LR 10.08. Casuarina experiment in a latinized row-column design (Williams et al., 2002). 4 Blocks of 60 provenances arranged in 6 rows by 10 columns. Provenances grouped according to 18 Countries of origin. 2 Inoculation dates each applied to 2 of the blocks. Estimates: C 0.2710; B, BR, BC < 0.06; BRC 0.2711. 23

24. ANOVA for Casuarina experiment Provenance represents provenance differences within countries. 24 plots tiertreatments tier sourced.f.sourced.f.Eff.MSFp-value Blocks3Innoculation111.5411.460.077 Residual21.011.17 Columns9Country97.25 Rows[B]20Country170.90 Provenance30.43 B#C27Country170.69 Provenance100.48 R#C[B]176Country170.7612.4610.25<0.001 Provenance410.6850.291.220.235 I#C170.6810.130.540.917 I#P410.5220.150.630.938 Residual600.24

25. Comparison of p-values For intrablock F, p-values from F and randomization distributions generally agree. For F Wald, p-values from F-distribution generally underestimates that from randomization distribution: (Rabbit Diets an exception – little interblock contribution). 25 ExampleSourceIntrablock FF Wald (Combined) 2 F-distri- bution Randomiz- ation 2 F-distri- bution Randomiz -ation WheatVarieties150.0160.01219.10.0350.096 RabbitDiets150.038 16.00.0320.034 TreeCountry60<0.001 79.3<0.0010.008 Provenance600.2350.23879.00.3380.454 Innoc#C600.9170.91884.80.9630.976 Innoc#P600.938 81.10.9430.966

26. 4.Randomization model for a chain of randomizations A chain of two randomizations consists of: the randomization of treatments to the first set of units; The randomization of the first set of units to a second set of units. For example, a two-phase sensory experiment (Brien & Payne, 1999; Brien & Bailey, 2006, Example 15) involves two randomizations: Field phase: 8 treatments to 48 halfplots using split-plot with 2 Youden squares for main plots. Sensory phase: 48 halfplots randomized 576 evaluations, using Latin squares and an extended Youden square. 26 2Occasions 3Intervals in O 6Judges 4Sittings in O, I 4Positions in O, I, S, J 576 evaluations 48 halfplots 2Squares 3Rows 4Columns in Q 2Halfplots in Q, R, C 8 treatments 4Trellis 2Methods (Q = Squares) Three sets of objects: treatments ( ), halfplots ( ) & evaluations ( ).

27. Randomization model Additive model of constants: y z + X f w + X f X h where y is the vector of observed responses; z is the vector of constants representing the contributions of each unit in the 2 nd randomization ( ) to the response; w is the vector of constants representing the contributions of each unit in the 1 st randomization ( ) to the response; and is a vector of treatment constants; X f & X h are design matrices showing the randomization assignments. Under the two randomizations, each element of z and of w become random variables, as does each element of y. Y Z + X f W + X f X h where Y, Z and W are the vectors of random variables. Now, we assume E R [Z] E R [W] 0 and so E R [Y] X f X h. 27

28. Randomization model (contd) Further, 28 C & C are the contributions to the variance arising from and, respectively. & are the sets of generalized factors (terms) derived from the posets of factors on and ; are the covariances; are the canonical component of excess covariance; are the eigenvalues of C and C, respectively; are known matrices.

29. Forming the null randomization distribution of the response Under the assumption of no treatment effects, Y * Z + X f W +. There are two randomizations, to and to ; to effect to, and are permuted, and to effect to, and are permuted. However, in this model X f is fixed and reflects the actual randomization employed in the experiment. Hence, we do not apply the second randomization and consider the null randomization distribution, conditional on the observed randomization of to. 1) Apply the permutations of to and y, to effect a rerandomization of to. o must also be applied to so that it does not effect a rerandomization of to. 29

30. 5.Randomization analysis for a chain of randomizations Again, based on the randomization distribution of the response. Use the same test statistics as for a single randomization: F Wald and intrablock F-statistics. Obtain or estimate the randomization distributions of these test statistics Based on randomization of to and is conditional on the observed randomization of to. 30

31. A Two-Phase Sensory Experiment (Brien & Bailey, 2006, Example 15) Involves two randomizations: 31 (Brien & Payne, 1999) 2Occasions 3Intervals in O 6Judges 4Sittings in O, I 4Positions in O, I, S, J 576 evaluations 48 halfplots 2Squares 3Rows 4Columns in Q 2Halfplots in Q, R, C 8 treatments 4Trellis 2Methods (Q = Squares) The randomization distribution will be based on the randomization of treatments to halfplots and is conditional on the actual randomization of halfplots to evaluations. Permuting evaluations and y will almost certainly result in unobserved combinations of halfplots and evaluations, so that the randomization model is no longer valid.

32. ANOVA table for sensory exp't 32 evaluations tier sourcedf Occasions1 Judges5 O#J5 Intervals[O]4 I#J[O]20 Sittings[O I] 18 S#J[O I] 90 Positions[O I S J] 432 treatments tier effsourcedf 1/27Trellis3 Residual3 2/27Trellis3 Residual3 8/9Trellis3 Residual9 Method1 T#M3 Residual20 Intrablock Trellis Orthogonal sources halfplots tier effsourcedf Squares1 Rows2 Q#R2 Residual16 1/3Columns[Q]6 Residual12 2/3Columns[Q]6 R#C[Q]12 Residual72 Halfplots[R C Q] 24 Residual408

33. Comparison of p-values Note the difference in denominator df for Trellis. 33 SourceIntrablock FF Wald (Combined) 2 F- distribution Randomiz- ation 2 F- distribution Randomiz- ation Trellis90.0010.00414.9<0.0010.004 Method200.6270.626 Trellis#Method200.0090.005

34. F 5.10 p F 0.009 p R 0.005 F 0.24 p F 0.627 p R 0.626 F comb 25.59 p F <0.001 p R 0.004 F intra 13.47 p F 0.001 p R 0.004 Comparison of distributions Trellis 34 Method Trellis Trellis# Method

35. 6.Some issues Size of permutations sample A controversy: sometimes pooling Unit-treatment additivity 35

36. Size of permutations sample A study of subsamples of the 50,000 randomly selected permutations revealed that: the estimates of p-values from samples of 25,000 or more randomized layouts have a range < 0.005. samples of 5,000 randomized layouts will often be sufficiently accurate – the estimates of p-values o around 0.01 or less, exhibit a range < 0.005; o in excess of 0.20, show a range about 0.03; o around 0.05, display a range of 0.01. 36

37. Unit-treatment additivity Cox and Reid (2000) allow random unit-treatment interaction; Test hypothesis that treatment effects are greater than unit- treatment interaction. Nelder (1977) suggests the random form is questionable. The Iowa school allows arbitrary (fixed) unit-treatment interactions. Test difference between the average treatment effects over all units, which is biased in the presence of unit-treatment interaction. Such a test ignores marginality/hierarchy. Questions: Which form applies? How to detect unit-treatment interaction? Often impossible, but, when it is possible, cannot be part of a randomization analysis. Randomization analysis requires unit-treatment additivity. If not appropriate, use a randomization-based mixed model. 37

38. A controversy Should nonsignificant (??) unit sources of variation be removed and hence pooled with other unit sources? The point is that effects hypothesized to occur at the planning stage have not eventuated. A modeller would remove them; Indeed, in mixed-model fitting using REML will have no option if the fitting process does not converge. Some argue, because in randomization model, must stay. Seems reasonable if doing randomization inference. Sometimes-pooling may disrupt power and coverage properties of the analysis (Janky, 2000). 38

39. 7.Conclusions Fisher was right: One should employ meaningful models; Randomization analyses provides a check on parametric analyses. I am still a modeller, with the randomization-based mixed model as my starting point. I am happy that, for single-stratum tests, the normal theory test approximates an equivalent randomization test, when one exists. However, the p-values for combined test-statistics from the F-distribution are questionable: novel that depends on interblock components; need to do bootstrap or randomization analysis for F Wald when denominator df for intrablock-F and F Wald differ markedly; this has the advantage of avoiding the need to pool nonsignificant (??) unit sources of variation, although fitting can be challenging. 39

40. References Atiqullah, M. (1963) On the randomization distribution and power of the variance ratio test. J. Roy. Statist. Soc., Ser. B (Methodological), 25: 334-347. Brien, C.J. & Bailey, R.A. (2006) Multiple randomizations (with discussion). J. Roy. Statist. Soc., Ser. B (Statistical Methodology), 68: 571-609. Brien, C.J. & Demétrio, C.G.B. (2009) Formulating Mixed Models for Experiments, Including Longitudinal Experiments." J. Agric. Biol. Environ. Statist., 14: 253-280. Cox, D.R. & Reid, N. (2000). The theory of the design of experiments. Boca Raton, Chapman & Hall/CRC. Edgington, E.S. (1995) Randomization tests. New York, Marcel Dekker. Fisher, R.A. (1935, 1960) The Design of Experiments. Edinburgh, Oliver and Boyd. Hinkelmann, K. & Kempthorne, O. (2008) Design and analysis of experiments. Vol I. Hoboken, N.J., Wiley-Interscience. Janky, D.G. (2000) Sometimes pooling for analysis of variance hypothesis tests: A review and study of a split-plot model. The Amer. Statist. 54: 269-279. Joshi, D.D. (1987) Linear estimation and design of experiments. Delhi, New Age Publishers. 40

41. References (contd) Kempthorne, O. (1975) Inference from experiments and randomization. A Survey of Statistical Design and Linear Models. J. N. Srivastava. Amsterdam., North Holland. Mead, R., S. G. Gilmour & Mead, A.. (2012). Statistical principles for the design of experiments. Cambridge, Cambridge University Press. Nelder, J.A. (1965) The analysis of randomized experiments with orthogonal block structure. I. Block structure and the null analysis of variance. Proc. Roy. Soc. Lon., Series A, 283: 147-162. Nelder, J. A. (1977). A reformulation of linear models (with discussion). J. Roy. Statist. Soc., Ser. A (General), 140: 48-77. Preece, D.A. (1982) The design and analysis of experiments: what has gone wrong?" Util. Math., 21A: 201-244. Schaalje, B. G., J. B. McBride, et al. (2002). Adequacy of approximations to distributions of test statistics in complex mixed linear models. J. Agric. Biol, Environ. Stat., 7: 512-524. Welch, B.L. (1937) On the z-test in randomized blocks and Latin squares. Biometrika, 29: 21-52. Williams, E.R., Matheson, A.C. & Harwood, C.E. (2002). Experimental design and analysis for tree improvement. Collingwood, Vic., CSIRO Publishing. 41