Lecture 9 Page 1 CS 239, Spring 2007 More Experiment Design CS 239 Experimental Methodologies for System Software Peter Reiher May 8, 2007

Lecture 9 Page 2 CS 239, Spring 2007 Outline Multiplicative experiment design models 2 k r factorial experiment designs 2 k-p fractional factorial designs Confounding in fractional factorial designs

Lecture 9 Page 3 CS 239, Spring 2007 Assumptions of additive models Example of a multiplicative situation Handling a multiplicative model When to choose multiplicative model Multiplicative example Multiplicative Models for 2 2 r Experiments

Lecture 9 Page 4 CS 239, Spring 2007 Assumptions of Additive Models Last time’s analysis used additive model: –y ij = q 0 + q A x A + q B x B + q AB x A x B + e ij Assumes all effects are additive: –Factors –Interactions –Errors This assumption must be validated!

Lecture 9 Page 5 CS 239, Spring 2007 Example of a Multiplicative Situation Testing processors with different workloads Most common multiplicative case Consider 2 processors, 2 workloads –Use 2 2 r design Response is time to execute w j instructions on processor that takes v i seconds/instruction w j and v i sound like good factors to test Without interactions, time is y ij = v i w j

Lecture 9 Page 6 CS 239, Spring 2007 Handling a Multiplicative Model Take logarithm of both sides: y ij = v i w j so log(y ij ) = log(v i ) + log(w j ) Use additive model on logarithms X A is log(v i ), X B is log(w j ) –Choose your high and low levels for each Resulting model is: –log(y ij ) = q0 +q A X A + q B X B + q AB X A X B +e ij But we care about y ij, not log(y ij )

Lecture 9 Page 7 CS 239, Spring 2007 Converting Back to y ij Take antilog of both sides of equation U A = 10 q A U B = 10 q B U AB = 10 q AB

Lecture 9 Page 8 CS 239, Spring 2007 Meaning of a Multiplicative Model Model is Here,  A = 10 q A is ratio of MIPS ratings of processors,  B = 10 q B is ratio of workload size Antilog of q 0 is geometric mean of responses: where n = 2 2 r

Lecture 9 Page 9 CS 239, Spring 2007 When to Choose a Multiplicative Model? Physical considerations (see previous slides) Range of y is large –Making arithmetic mean unreasonable –Calling for log transformation Plot of residuals shows large values and increasing spread Quantile-quantile plot doesn’t look like normal distribution

Lecture 9 Page 10 CS 239, Spring 2007 Multiplicative Example Consider additive model of processors A 1 and A 2 running benchmarks B 1 and B 2 : Note large range of y values

Lecture 9 Page 11 CS 239, Spring 2007 Error Scatter of Additive Model

Lecture 9 Page 12 CS 239, Spring 2007 Quantile-Quantile Plot of Additive Model

Lecture 9 Page 13 CS 239, Spring 2007 Multiplicative Model Taking logs of everything, the model is:

Lecture 9 Page 14 CS 239, Spring 2007 Error Residuals of Multiplicative Model

Lecture 9 Page 15 CS 239, Spring 2007 Quantile-Quantile Plot for Multiplicative Model

Lecture 9 Page 16 CS 239, Spring 2007 Summary of the Two Models Which suggests the time to run a benchmark depends only on the processor speed and benchmark size Sounds about right

Lecture 9 Page 17 CS 239, Spring 2007 General 2 k r Factorial Design Simple extension of 2 2 r Just k factors, not 2 See Box 18.1 for summary Always do visual tests Remember to consider multiplicative model as alternative

Lecture 9 Page 18 CS 239, Spring 2007 Example of 2 k r Factorial Design Consider a design 3 factors 2 levels for each 3 replications of each combination There will be more factor interaction terms, of course

Lecture 9 Page 19 CS 239, Spring 2007 Sign Table for a Sample 2 k r Factorial Design

Lecture 9 Page 20 CS 239, Spring 2007 Allocation of Variation for Design Percent variation explained: 90% confidence intervals

Lecture 9 Page 21 CS 239, Spring 2007 Error Residuals for Design

Lecture 9 Page 22 CS 239, Spring 2007 Quantile-Quantile Plot for All Points for Design

Lecture 9 Page 23 CS 239, Spring 2007 Quantile-Quantile Plot for Means R 2 for this one is.94

Lecture 9 Page 24 CS 239, Spring 2007 Concerns With These Kinds of Designs They don’t test all possible levels –Only test two, in fact –Solved by full factorial designs Which we’ll cover later They are a lot of work –Especially if there are many factors –Solved by fractional factorial design

Lecture 9 Page 25 CS 239, Spring 2007 Fractional Designs What if there are many factors? You can’t afford to test all combinations Well, then, test only some of them How should you determine which combinations to test? –Losing least information

Lecture 9 Page 26 CS 239, Spring k-p Fractional Factorial Designs Introductory example of a 2 k-p design Preparing the sign table for a 2 k-p design Confounding Algebra of confounding Design resolution

Lecture 9 Page 27 CS 239, Spring 2007 What Is A 2 k-p Fractional Factorial Design? As before, test only two levels of each factor But instead of testing all 2 k factors, Only test 2 k-p of them –The larger p is, the fewer combinations tested –E.g., for k = 5 and p = 2, reduces tests from 32 to 8

Lecture 9 Page 28 CS 239, Spring 2007 Introductory Example of a 2 k-p Design Exploring 7 factors in only 8 experiments –k = 7, p = 4 Full factorial design would take 128 experiments Won’t we save time! Would be nice to know what price we paid –We can’t know everything –But we can get some control

Lecture 9 Page 29 CS 239, Spring 2007 Sign Table for Example

Lecture 9 Page 30 CS 239, Spring 2007 Analysis of Design Column sums are zero: Sum of 2-column product is zero: Sum of column squares is = 8 Orthogonality allows easy calculation of effects:

Lecture 9 Page 31 CS 239, Spring 2007 Effects and Confidence Intervals for 2 k-p Designs Effects are as in 2 k designs: % variation proportional to squared effects For standard deviations and confidence intervals: –Use formulas from full factorial designs –Replace 2 k with 2 k-p

Lecture 9 Page 32 CS 239, Spring 2007 Preparing the Sign Table for a 2 k-p Design Start by preparing a sign table for k-p factors –Assign first k-p factors as before Then assign remaining factors –In the place of some (or all) of the combined effects columns

Lecture 9 Page 33 CS 239, Spring 2007 Sign Table for k-p Factors Same as table for experiment with k-p factors –I.e., 2 (k-p) table –2 k-p rows and 2 k-p columns –First column is I, contains all 1’s –Next k-p columns get k-p selected factors –Rest are products of factors

Lecture 9 Page 34 CS 239, Spring 2007 Assigning Remaining Factors 2 k-p -(k-p)-1 product columns remain Choose any p columns –Assign remaining p factors to them –Any others stay as-is, measuring interactions

Lecture 9 Page 35 CS 239, Spring 2007 An Example Let’s build a table So there are five factors A, B, C, D, and E But we only want to run 8 experiments –p = 2 –5-2=3

Lecture 9 Page 36 CS 239, Spring 2007 Start With a 2 3 Table Exp #IABCABACBCABC

Lecture 9 Page 37 CS 239, Spring 2007 Now Add the Remaining Factors D and E Exp #IABCABACBCABC Exp #IABCABACBCD Exp #IABCABACED

Lecture 9 Page 38 CS 239, Spring 2007 Our Final Sign Table Exp #IABCABACED

Lecture 9 Page 39 CS 239, Spring 2007 Running Experiments With This Sign Table Use it just as before Run the set of experiments the table indicates –E.g., run A,B,C,D at low level, E at high level –Then A and E at high, B, C, and D at low –And so on

Lecture 9 Page 40 CS 239, Spring 2007 Calculating Effects With the Sign Table Just like before Multiply experiment results by columns Add up results Divide by number of experiments There are your q values

Lecture 9 Page 41 CS 239, Spring 2007 What Have We Paid? What about all the other effect combinations? The fourth column shows the combined effects of A and B The fifth column shows the combined effects of A and C Exp #IABCABACED

Lecture 9 Page 42 CS 239, Spring 2007 The other combined effects were confounded The confounding problem An example of confounding Confounding notation Choices in fractional factorial design Confounding

Lecture 9 Page 43 CS 239, Spring 2007 The Confounding Problem Fundamental to fractional factorial designs Some effects produce combined influences –Limited experiments mean only some combinations can be calculated Problem of combined influence is confounding –Inseparable effects called confounded effects

Lecture 9 Page 44 CS 239, Spring 2007 An Example of Confounding Consider this table: Extend it with an AB column:

Lecture 9 Page 45 CS 239, Spring 2007 Analyzing the Confounding Example Effect of C is same as that of AB: q C = (y 1 -y 2 -y 3 +y 4 )/4 q AB = (y 1 -y 2 -y 3 +y 4 )/4 Formula for q C really gives combined effect: q C +q AB = (y 1 -y 2 -y 3 +y 4 )/4 No way to separate q C from q AB –Not a problem if q AB is known to be small

Lecture 9 Page 46 CS 239, Spring 2007 Let’s Go Back to Our Example Where are combined effects AD, AE, BC, BD, BE, CD, CE, and DE? Not to mention ABC, ABD, ABE, ACD, ACE, ADE, BCD, BDE, BCE, CDE, ABCD, ABCE, ACDE, ABDE,BCDE, and ABCDE? Exp #IABCABACED

Lecture 9 Page 47 CS 239, Spring 2007 Confounding Notation Previous confounding is denoted by equating confounded effects: C = AB Other effects are also confounded in this design: A = BC, B = AC, I = ABC –Last entry indicates ABC is confounded with overall mean, or q 0

Lecture 9 Page 48 CS 239, Spring 2007 What Does Confounding Really Mean? Each effect is a combination of several effects from a full experiment Impossible to pull out one from the other –Unless you change design and make more runs Must be aware of what’s getting confounded

Lecture 9 Page 49 CS 239, Spring 2007 Getting Concrete on The Meaning of Confounding Consider our generic fractional factorial experiment What if we’re measuring computer performance? With three factors: 1.CPU speed (A) 2.Memory size (B) 3.Disk speed (C)

Lecture 9 Page 50 CS 239, Spring 2007 Using Our Fractional Design, We have combined the effect of disk speed with the interaction of CPU speed and memory size (C=AB) And the effect of CPU speed with the combined effects of disk speed and memory size (A=BC) Among several others

Lecture 9 Page 51 CS 239, Spring 2007 Does This Matter? Maybe What if disk speed had little effect on system performance? But the interaction between memory size and CPU speed had a lot Might conclude that you can speed up the system by increasing disk speed When actually that makes no difference

Lecture 9 Page 52 CS 239, Spring 2007 Choosing a Fractional Factorial Design Many fractional factorial designs possible –Chosen when assigning remaining p signs –2 p different designs exist for 2 k-p experiments Some designs better than others –Desirable to confound significant effects with insignificant ones –Usually means low-order with high-order

Lecture 9 Page 53 CS 239, Spring 2007 Algebra of Confounding Rules of the algebra Generator polynomials

Lecture 9 Page 54 CS 239, Spring 2007 Rules of Confounding Algebra Particular design can be characterized by single confounding –Traditionally, the I = wxyz... confounding Others can be found by multiplying by various terms –I acts as unity (e.g., I times A is A) –Squared terms disappear (AB 2 C becomes AC)

Lecture 9 Page 55 CS 239, Spring 2007 Example: Confoundings Design is characterized by I = ABC Multiplying by A gives A = A 2 BC = BC Multiplying by B, C, AB, AC, BC, and ABC: B = AB 2 C = AC, C = ABC 2 = AB, AB = A 2 B 2 C = C, AC = A 2 BC 2 = B, BC = AB 2 C 2 = A, ABC = A 2 B 2 C 2 = I

Lecture 9 Page 56 CS 239, Spring 2007 Generator Polynomials Polynomial I = wxyz... is called generator polynomial for the confounding A 2 k-p design confounds 2 p effects together –So generator polynomial has 2 p terms –Can be found by considering interactions replaced in sign table

Lecture 9 Page 57 CS 239, Spring 2007 Example of Finding a Generator Polynomial Consider a design Sign table has 2 3 = 8 rows and columns First 3 columns represent A, B, and C Columns for D, E, F, and G replace AB, AC, BC, and ABC columns respectively –So confoundings are necessarily: D = AB, E = AC, F = BC, and G = ABC

Lecture 9 Page 58 CS 239, Spring 2007 The Design Exp #IABCDEFG ABACBC ABC

Lecture 9 Page 59 CS 239, Spring 2007 Turning Basic Terms into Generator Polynomial Basic confoundings are D = AB, E = AC, F = BC, and G = ABC Multiply each equation by left side: I = ABD, I = ACE, I = BCF, and I = ABCG or I = ABD = ACE = BCF = ABCG

Lecture 9 Page 60 CS 239, Spring 2007 Finishing the Generator Polynomial Any subset of above terms also multiplies out to I –E.g., ABD times ACE = A 2 BCDE = BCDE Expanding all possible combinations gives the 16- term generator (book is wrong, ABDG isn’t one of them): I = ABD = ACE = BCF = ABCG = BCDE = ACDF = CDG = ABEF = BEG = AFG = DEF = ADEG = BDFG = CEFG = ABCDEFG

Lecture 9 Page 61 CS 239, Spring 2007 Fractional Design Resolution What is resolution? Finding resolution Choosing a resolution

Lecture 9 Page 62 CS 239, Spring 2007 What Is Resolution? Design is characterized by its resolution Resolution measured by order of confounded effects Order of effect is number of factors in it –E.g., I is order 0, and ABCD is order 4 Order of confounding is sum of effect orders –E.g., AB = CDE would be of order 5

Lecture 9 Page 63 CS 239, Spring 2007 Definition of Resolution Resolution is minimum order of any confounding in design Denoted by uppercase Roman numerals –E.g, with resolution of 3 is called R III –Or more compactly,

Lecture 9 Page 64 CS 239, Spring 2007 Finding Resolution Find minimum order of effects confounded with mean –I.e., search generator polynomial Consider earlier example: I = ABD = ACE = BCF = ABCG = BCDE = ACDF = CDG = ABEF = BEG = AFG = DEF = ADEG = BDFG = CEFG = ABCDEFG So it’s R III design

Lecture 9 Page 65 CS 239, Spring 2007 Choosing a Resolution Generally, higher resolution is better Because usually higher-order interactions are smaller Exception: when low-order interactions are known to be small –Then choose design that confounds those with important interactions –Even if resolution is lower

Lecture 9 Page 66 CS 239, Spring 2007 Fractional Designs With Replications You can (of course) run multiple experiments for each fractional setting –Formally, a 2 k-p r design Multiplicative factor in number of runs –Rather than exponential Is this a better use of your experiment time? Depends

Lecture 9 Page 67 CS 239, Spring 2007 Depends on What? Are confounded interactions more important than statistical variation? Running 2 replications is same work as decreasing p by 1 Maybe better to reconsider your factors –Do you need all of them? –Decreasing by 1 factor allows less confounding for same experiments

Lecture 9 Page 68 CS 239, Spring 2007 So, More Runs or Less Confounding? 2 x runs are often used for preliminary experiments, anyway To identify important factors to look at more deeply How likely is it that an “unlucky” single replication will misidentify important factor?