1. Factorial Design of Experiments Kevin Leyton-Brown

2. Experimental Design2 Terminology: variables response variable –outcome of an experiment. The thing being measured factor –a variable upon which the experimenter believes that one or more response variables may depend, and which the experimenter can control the design of the experiment will largely consist of a policy for determining how to set the factors in each experimental trial –possible values of a factor are called levels. in other literature, the word “version” is used for a qualitative (not quantitative) level experimental region / factor space –all possible levels of the factors which are candidates for inclusion in the experiment covariates –these are like factors which are not under the control of the experimenter. Like factors, it is believed that they might affect the value of the response. Unlike factors, they cannot be controlled, though they can be observed.

3. Experimental Design3 Terminology: experiments test run –a single factor/level combination for which an observation of the response is recorded repeat tests –two or more test runs. Every effort is made to ensure that the individual test runs take place under identical circumstances replications –two or more repetitions of an experiment or a portion of an experiment under potentially different conditions experimental unit –a measurement (or a material upon which a measurement is made) –it is important that the experimental units be homogeneous i.e., they do not exhibit systematic variation

4. Experimental Design4 Terminology: Experimental Design confounding –if two factors are varied simultaneously, we can say the effect of the factors is confounded. Likewise, bad experimental design can lead to the effect of one or more factors being confounded with one or more covariates. blocking –if experimental units can be partitioned into groups (blocks) so that experimental units within groups are more homogeneous than experimental units across groups, the experiment can be designed differently to minimize the effects of the non-homogeneity of experimental units. For example, the experiment could be replicated in each block. experimental design / experimental layout –choice of the factor-level combinations to be examined –number of repeat tests, replications; blocking –assignment of factor-level combinations to experimental units –sequencing of test runs effect of factors on response –the change in the average response under two or more factor-level combinations

5. Experimental Design5 Examples 1.Suspended-Particulate study –measure three factors –consider all combination of factor levels (possible because each factor takes on only two or three values) –random sequencing –temperature measured as a covariate 2.Soil treatment study –effect of two factors (fertilizer; plowing) on soybean yield –three different fields must be used, so blocking is done –thus, each factor-level combination will be tested on each field, with the experiment replicated three times

6. Experimental Design6 Common Design Problems 1.Masking factor effects –if variation in test results is on the same order of magnitude as the factor effects, the latter may go undetected. This can be addressed through appropriate choice of sample size. –unmeasured covariates (e.g., the effect of time passing, as an instrument degrades) can also lead to variation that masks factor effects 2.Uncontrolled factors –it’s pretty obvious that all variables of interest should be included as factors –sometimes, though, it can be tricky to choose the appropriate level of model granularity. Sometimes a “high-level feature” (i.e., some function of the levels of many factors) could be chosen in place of the many factors that influence it this can make it difficult to vary the factor appropriately, and can limit analysis of the experimental results. 3.One-factor-at-a-time testing –it can be tempting to vary each factor value independently, holding the others constant. However, this does not explore the factor space very effectively. It would be a terrible local search strategy! –Looking at the same point in another way, it neglects the possibility that interactions between factors could be important

7. Experimental Design7 Experimental Design: Overview 1.Consideration of Objectives –consider the sorts of hypotheses that should be tested by the experiment –determine which variables are observable 2.Factor Effects –select factors, covariates –guess at whether factors act independently of each other or not 3.Precision and Efficiency –Precision: the least precise definition of this term that could possibly be given. My best guess: “variability in a statistic of interest” variance in measured response values given identical repeat tests variance in measured feature levels under homogeneous conditions –ways of improving precision: blocking, repeat tests, replication, accounting for covariates that is, either increasing sample size or accounting for previously unconsidered sources of variation –Efficiency: for efficiency reasons, the authors leave this one out! 4.Randomization –…in the sequence of test runs or the assignment of factor-level combinations can help protect against systematic bias. Do it.

8. Experimental Design8 Factorial Experiments in Completely Randomized Designs A (complete) factorial experiment includes all possible factor-level combinations One of the most straightforward experimental designs to implement in this setting is a completely randomized design –in such a design, factor-level combinations are sequenced / assigned to experimental units uniformly at random Randomization can prevent all kinds of problems, most of which we probably don’t have in a computational setting: –“instrument drift, power surges, equipment malfunctions, technician or operator errors, …” –note, of course, that randomization doesn’t prevent these problems from occurring. It just eliminates bias While many of these problems don’t affect us in empirical algorithmic experiments, some still can, and in any case randomization should be very easy for us to do. –Amazingly enough, they suggest that we obtain random numbers by consulting a table in the appendix!

9. Experimental Design9 Repeat Tests Experiment designs should include repeat test runs –that is, runs where all the parameters are exactly the same –can be used to estimate variance due from uncontrolled / random causes while we often do this when studying randomized algorithms, we probably do it very rarely when studying non-randomized algorithms. –Is it a good idea? balance – having each factorial-level combination repeated an equal number of times is desirable –when this is not possible or appropriate, a good approach is to randomly (without replacement) select factor-level combinations to repeat once –simply repeating some factor-level combinations many times while not repeating others is a bad idea –for some reason, they state that six repeated tests is a good amount for estimating variance. In practice, I’m sure it depends on dimensionality of factor space, degree of interaction between factors, etc.

10. Experimental Design10 Interactions between Factors Here we investigate situations where joint factor effects are present –in other words, the value of the response variable does not depend only on a function of this kind: –instead, the response variable depends on a function of this kind: –how do we calculate the magnitudes of the interaction effects?

11. Experimental Design11 Calculation of Factor Effects: Notation factors are denoted using uppercase Latin letters: A, B, C, … an individual response is a lowercase letter, usually subscripted –one subscript for each factor; designates factor-level combination used to generate response. Possibly other subscripts to denote which repeat trial. example: 3 factors, A, B, C. –Domain of A is {1,2}, subscripted by i; domain of B is {1,2}, subscripted by j. –Domain of C is {1,2,3,4}, subscripted by k. –r repeat trials were conducted; the repeat trial is denoted by a fourth subscript l, with domain {1, …, r} –responses would be written as y ijkl To denote averaging over one or more of the subscripts, we replace the subscript by a dot. In our example (where n = 2 ¢ 2 ¢ 4 ¢ r):

12. Experimental Design12 Calculation of Factor Effects Effects representation: –in the case of two-level factors, encode the domain of each factor as {-1,1} –the AB column is then the product of A and B Main effect for factor A (two factors, two levels each): Interaction between two factors A and B:

13. Experimental Design13 Calculation of Factor Effects Based on these definitions, it seems pretty easy to calculate the factor effects by summing appropriately. However, they give us: –Algorithm for calculating effects of two-level factors: construct the effects representation for each main effect and each interaction calculate linear combinations of the responses, using the signs in the effects column for ech main effect and for each interaction Divide the results from (2) by 2 k-1 where k is the total number of factors Then, we can compare the calculated effects for each interaction with the standard error. If an effect is much larger (e.g., more than twice as large) the authors seem to treat it as significant. –they calculate standard error as 2s e / n 1/2, where s e is the estimated standard deviation of the uncontrolled experimental error, and n is the total number of test runs (including repeat tests) What if we have more than two factor levels? Basically, things get messy. One solution (if numbers of factor levels are powers of two) is to represent each bit of the factor level using a new, derived two-level factor. Now we need interactions between these simpler factors just to get the main effects of the original factor. However, everything else works as before.