1. Lecture 9 Page 1 CS 239, Spring 2007 More Experiment Design CS 239 Experimental Methodologies for System Software Peter Reiher May 8, 2007

2. Lecture 9 Page 2 CS 239, Spring 2007 Outline Multiplicative experiment design models 2 k r factorial experiment designs 2 k-p fractional factorial designs Confounding in fractional factorial designs

3. Lecture 9 Page 3 CS 239, Spring 2007 Assumptions of additive models Example of a multiplicative situation Handling a multiplicative model When to choose multiplicative model Multiplicative example Multiplicative Models for 2 2 r Experiments

4. Lecture 9 Page 4 CS 239, Spring 2007 Assumptions of Additive Models Last time’s analysis used additive model: –y ij = q 0 + q A x A + q B x B + q AB x A x B + e ij Assumes all effects are additive: –Factors –Interactions –Errors This assumption must be validated!

5. Lecture 9 Page 5 CS 239, Spring 2007 Example of a Multiplicative Situation Testing processors with different workloads Most common multiplicative case Consider 2 processors, 2 workloads –Use 2 2 r design Response is time to execute w j instructions on processor that takes v i seconds/instruction w j and v i sound like good factors to test Without interactions, time is y ij = v i w j

6. Lecture 9 Page 6 CS 239, Spring 2007 Handling a Multiplicative Model Take logarithm of both sides: y ij = v i w j so log(y ij ) = log(v i ) + log(w j ) Use additive model on logarithms X A is log(v i ), X B is log(w j ) –Choose your high and low levels for each Resulting model is: –log(y ij ) = q0 +q A X A + q B X B + q AB X A X B +e ij But we care about y ij, not log(y ij )

7. Lecture 9 Page 7 CS 239, Spring 2007 Converting Back to y ij Take antilog of both sides of equation U A = 10 q A U B = 10 q B U AB = 10 q AB

8. Lecture 9 Page 8 CS 239, Spring 2007 Meaning of a Multiplicative Model Model is Here,  A = 10 q A is ratio of MIPS ratings of processors,  B = 10 q B is ratio of workload size Antilog of q 0 is geometric mean of responses: where n = 2 2 r

9. Lecture 9 Page 9 CS 239, Spring 2007 When to Choose a Multiplicative Model? Physical considerations (see previous slides) Range of y is large –Making arithmetic mean unreasonable –Calling for log transformation Plot of residuals shows large values and increasing spread Quantile-quantile plot doesn’t look like normal distribution

10. Lecture 9 Page 10 CS 239, Spring 2007 Multiplicative Example Consider additive model of processors A 1 and A 2 running benchmarks B 1 and B 2 : Note large range of y values

11. Lecture 9 Page 11 CS 239, Spring 2007 Error Scatter of Additive Model

12. Lecture 9 Page 12 CS 239, Spring 2007 Quantile-Quantile Plot of Additive Model

13. Lecture 9 Page 13 CS 239, Spring 2007 Multiplicative Model Taking logs of everything, the model is:

14. Lecture 9 Page 14 CS 239, Spring 2007 Error Residuals of Multiplicative Model

15. Lecture 9 Page 15 CS 239, Spring 2007 Quantile-Quantile Plot for Multiplicative Model

16. Lecture 9 Page 16 CS 239, Spring 2007 Summary of the Two Models Which suggests the time to run a benchmark depends only on the processor speed and benchmark size Sounds about right

17. Lecture 9 Page 17 CS 239, Spring 2007 General 2 k r Factorial Design Simple extension of 2 2 r Just k factors, not 2 See Box 18.1 for summary Always do visual tests Remember to consider multiplicative model as alternative

18. Lecture 9 Page 18 CS 239, Spring 2007 Example of 2 k r Factorial Design Consider a 2 3 3 design 3 factors 2 levels for each 3 replications of each combination There will be more factor interaction terms, of course

19. Lecture 9 Page 19 CS 239, Spring 2007 Sign Table for a Sample 2 k r Factorial Design

20. Lecture 9 Page 20 CS 239, Spring 2007 Allocation of Variation for 2 3 3 Design Percent variation explained: 90% confidence intervals

21. Lecture 9 Page 21 CS 239, Spring 2007 Error Residuals for 2 3 3 Design

22. Lecture 9 Page 22 CS 239, Spring 2007 Quantile-Quantile Plot for All Points for 2 3 3 Design

23. Lecture 9 Page 23 CS 239, Spring 2007 Quantile-Quantile Plot for Means 2 3 3 R 2 for this one is.94

24. Lecture 9 Page 24 CS 239, Spring 2007 Concerns With These Kinds of Designs They don’t test all possible levels –Only test two, in fact –Solved by full factorial designs Which we’ll cover later They are a lot of work –Especially if there are many factors –Solved by fractional factorial design

25. Lecture 9 Page 25 CS 239, Spring 2007 Fractional Designs What if there are many factors? You can’t afford to test all combinations Well, then, test only some of them How should you determine which combinations to test? –Losing least information

26. Lecture 9 Page 26 CS 239, Spring 2007 2 k-p Fractional Factorial Designs Introductory example of a 2 k-p design Preparing the sign table for a 2 k-p design Confounding Algebra of confounding Design resolution

27. Lecture 9 Page 27 CS 239, Spring 2007 What Is A 2 k-p Fractional Factorial Design? As before, test only two levels of each factor But instead of testing all 2 k factors, Only test 2 k-p of them –The larger p is, the fewer combinations tested –E.g., for k = 5 and p = 2, reduces tests from 32 to 8

28. Lecture 9 Page 28 CS 239, Spring 2007 Introductory Example of a 2 k-p Design Exploring 7 factors in only 8 experiments –k = 7, p = 4 Full factorial design would take 128 experiments Won’t we save time! Would be nice to know what price we paid –We can’t know everything –But we can get some control

29. Lecture 9 Page 29 CS 239, Spring 2007 Sign Table for Example

30. Lecture 9 Page 30 CS 239, Spring 2007 Analysis of 2 7-4 Design Column sums are zero: Sum of 2-column product is zero: Sum of column squares is 2 7-4 = 8 Orthogonality allows easy calculation of effects:

31. Lecture 9 Page 31 CS 239, Spring 2007 Effects and Confidence Intervals for 2 k-p Designs Effects are as in 2 k designs: % variation proportional to squared effects For standard deviations and confidence intervals: –Use formulas from full factorial designs –Replace 2 k with 2 k-p

32. Lecture 9 Page 32 CS 239, Spring 2007 Preparing the Sign Table for a 2 k-p Design Start by preparing a sign table for k-p factors –Assign first k-p factors as before Then assign remaining factors –In the place of some (or all) of the combined effects columns

33. Lecture 9 Page 33 CS 239, Spring 2007 Sign Table for k-p Factors Same as table for experiment with k-p factors –I.e., 2 (k-p) table –2 k-p rows and 2 k-p columns –First column is I, contains all 1’s –Next k-p columns get k-p selected factors –Rest are products of factors

34. Lecture 9 Page 34 CS 239, Spring 2007 Assigning Remaining Factors 2 k-p -(k-p)-1 product columns remain Choose any p columns –Assign remaining p factors to them –Any others stay as-is, measuring interactions

35. Lecture 9 Page 35 CS 239, Spring 2007 An Example Let’s build a 2 5-2 table So there are five factors A, B, C, D, and E But we only want to run 8 experiments –p = 2 –5-2=3

36. Lecture 9 Page 36 CS 239, Spring 2007 Start With a 2 3 Table Exp #IABCABACBCABC 11 111 211 11 31 1 1 1 4111 1 51 11 1 611 1 1 71 11 1 811111111

37. Lecture 9 Page 37 CS 239, Spring 2007 Now Add the Remaining Factors D and E Exp #IABCABACBCABC 11 111 211 11 31 1 1 1 4111 1 51 11 1 611 1 1 71 11 1 811111111 Exp #IABCABACBCD 11 111 211 11 31 1 1 1 4111 1 51 11 1 611 1 1 71 11 1 811111111 Exp #IABCABACED 11 111 211 11 31 1 1 1 4111 1 51 11 1 611 1 1 71 11 1 811111111

38. Lecture 9 Page 38 CS 239, Spring 2007 Our Final Sign Table Exp #IABCABACED 11 111 211 11 31 1 1 1 4111 1 51 11 1 611 1 1 71 11 1 811111111

39. Lecture 9 Page 39 CS 239, Spring 2007 Running Experiments With This Sign Table Use it just as before Run the set of experiments the table indicates –E.g., run A,B,C,D at low level, E at high level –Then A and E at high, B, C, and D at low –And so on

40. Lecture 9 Page 40 CS 239, Spring 2007 Calculating Effects With the Sign Table Just like before Multiply experiment results by columns Add up results Divide by number of experiments There are your q values

41. Lecture 9 Page 41 CS 239, Spring 2007 What Have We Paid? What about all the other effect combinations? The fourth column shows the combined effects of A and B The fifth column shows the combined effects of A and C Exp #IABCABACED 11 111 211 11 31 1 1 1 4111 1 51 11 1 611 1 1 71 11 1 811111111

42. Lecture 9 Page 42 CS 239, Spring 2007 The other combined effects were confounded The confounding problem An example of confounding Confounding notation Choices in fractional factorial design Confounding

43. Lecture 9 Page 43 CS 239, Spring 2007 The Confounding Problem Fundamental to fractional factorial designs Some effects produce combined influences –Limited experiments mean only some combinations can be calculated Problem of combined influence is confounding –Inseparable effects called confounded effects

44. Lecture 9 Page 44 CS 239, Spring 2007 An Example of Confounding Consider this 2 3-1 table: Extend it with an AB column:

45. Lecture 9 Page 45 CS 239, Spring 2007 Analyzing the Confounding Example Effect of C is same as that of AB: q C = (y 1 -y 2 -y 3 +y 4 )/4 q AB = (y 1 -y 2 -y 3 +y 4 )/4 Formula for q C really gives combined effect: q C +q AB = (y 1 -y 2 -y 3 +y 4 )/4 No way to separate q C from q AB –Not a problem if q AB is known to be small

46. Lecture 9 Page 46 CS 239, Spring 2007 Let’s Go Back to Our Example Where are combined effects AD, AE, BC, BD, BE, CD, CE, and DE? Not to mention ABC, ABD, ABE, ACD, ACE, ADE, BCD, BDE, BCE, CDE, ABCD, ABCE, ACDE, ABDE,BCDE, and ABCDE? Exp #IABCABACED 11 111 211 11 31 1 1 1 4111 1 51 11 1 611 1 1 71 11 1 811111111

47. Lecture 9 Page 47 CS 239, Spring 2007 Confounding Notation Previous 2 3-1 confounding is denoted by equating confounded effects: C = AB Other effects are also confounded in this design: A = BC, B = AC, I = ABC –Last entry indicates ABC is confounded with overall mean, or q 0

48. Lecture 9 Page 48 CS 239, Spring 2007 What Does Confounding Really Mean? Each effect is a combination of several effects from a full experiment Impossible to pull out one from the other –Unless you change design and make more runs Must be aware of what’s getting confounded

49. Lecture 9 Page 49 CS 239, Spring 2007 Getting Concrete on The Meaning of Confounding Consider our generic 2 3-1 fractional factorial experiment What if we’re measuring computer performance? With three factors: 1.CPU speed (A) 2.Memory size (B) 3.Disk speed (C)

50. Lecture 9 Page 50 CS 239, Spring 2007 Using Our Fractional Design, We have combined the effect of disk speed with the interaction of CPU speed and memory size (C=AB) And the effect of CPU speed with the combined effects of disk speed and memory size (A=BC) Among several others

51. Lecture 9 Page 51 CS 239, Spring 2007 Does This Matter? Maybe What if disk speed had little effect on system performance? But the interaction between memory size and CPU speed had a lot Might conclude that you can speed up the system by increasing disk speed When actually that makes no difference

52. Lecture 9 Page 52 CS 239, Spring 2007 Choosing a Fractional Factorial Design Many fractional factorial designs possible –Chosen when assigning remaining p signs –2 p different designs exist for 2 k-p experiments Some designs better than others –Desirable to confound significant effects with insignificant ones –Usually means low-order with high-order

53. Lecture 9 Page 53 CS 239, Spring 2007 Algebra of Confounding Rules of the algebra Generator polynomials

54. Lecture 9 Page 54 CS 239, Spring 2007 Rules of Confounding Algebra Particular design can be characterized by single confounding –Traditionally, the I = wxyz... confounding Others can be found by multiplying by various terms –I acts as unity (e.g., I times A is A) –Squared terms disappear (AB 2 C becomes AC)

55. Lecture 9 Page 55 CS 239, Spring 2007 Example: 2 3-1 Confoundings Design is characterized by I = ABC Multiplying by A gives A = A 2 BC = BC Multiplying by B, C, AB, AC, BC, and ABC: B = AB 2 C = AC, C = ABC 2 = AB, AB = A 2 B 2 C = C, AC = A 2 BC 2 = B, BC = AB 2 C 2 = A, ABC = A 2 B 2 C 2 = I

56. Lecture 9 Page 56 CS 239, Spring 2007 Generator Polynomials Polynomial I = wxyz... is called generator polynomial for the confounding A 2 k-p design confounds 2 p effects together –So generator polynomial has 2 p terms –Can be found by considering interactions replaced in sign table

57. Lecture 9 Page 57 CS 239, Spring 2007 Example of Finding a Generator Polynomial Consider a 2 7-4 design Sign table has 2 3 = 8 rows and columns First 3 columns represent A, B, and C Columns for D, E, F, and G replace AB, AC, BC, and ABC columns respectively –So confoundings are necessarily: D = AB, E = AC, F = BC, and G = ABC

58. Lecture 9 Page 58 CS 239, Spring 2007 The 2 7-4 Design Exp #IABCDEFG 11 111 211 11 31 1 1 1 4111 1 51 11 1 611 1 1 71 11 1 811111111 ABACBC ABC

59. Lecture 9 Page 59 CS 239, Spring 2007 Turning Basic Terms into Generator Polynomial Basic confoundings are D = AB, E = AC, F = BC, and G = ABC Multiply each equation by left side: I = ABD, I = ACE, I = BCF, and I = ABCG or I = ABD = ACE = BCF = ABCG

60. Lecture 9 Page 60 CS 239, Spring 2007 Finishing the Generator Polynomial Any subset of above terms also multiplies out to I –E.g., ABD times ACE = A 2 BCDE = BCDE Expanding all possible combinations gives the 16- term generator (book is wrong, ABDG isn’t one of them): I = ABD = ACE = BCF = ABCG = BCDE = ACDF = CDG = ABEF = BEG = AFG = DEF = ADEG = BDFG = CEFG = ABCDEFG

61. Lecture 9 Page 61 CS 239, Spring 2007 Fractional Design Resolution What is resolution? Finding resolution Choosing a resolution

62. Lecture 9 Page 62 CS 239, Spring 2007 What Is Resolution? Design is characterized by its resolution Resolution measured by order of confounded effects Order of effect is number of factors in it –E.g., I is order 0, and ABCD is order 4 Order of confounding is sum of effect orders –E.g., AB = CDE would be of order 5

63. Lecture 9 Page 63 CS 239, Spring 2007 Definition of Resolution Resolution is minimum order of any confounding in design Denoted by uppercase Roman numerals –E.g, 2 5-1 with resolution of 3 is called R III –Or more compactly,

64. Lecture 9 Page 64 CS 239, Spring 2007 Finding Resolution Find minimum order of effects confounded with mean –I.e., search generator polynomial Consider earlier example: I = ABD = ACE = BCF = ABCG = BCDE = ACDF = CDG = ABEF = BEG = AFG = DEF = ADEG = BDFG = CEFG = ABCDEFG So it’s R III design

65. Lecture 9 Page 65 CS 239, Spring 2007 Choosing a Resolution Generally, higher resolution is better Because usually higher-order interactions are smaller Exception: when low-order interactions are known to be small –Then choose design that confounds those with important interactions –Even if resolution is lower

66. Lecture 9 Page 66 CS 239, Spring 2007 Fractional Designs With Replications You can (of course) run multiple experiments for each fractional setting –Formally, a 2 k-p r design Multiplicative factor in number of runs –Rather than exponential Is this a better use of your experiment time? Depends

67. Lecture 9 Page 67 CS 239, Spring 2007 Depends on What? Are confounded interactions more important than statistical variation? Running 2 replications is same work as decreasing p by 1 Maybe better to reconsider your factors –Do you need all of them? –Decreasing by 1 factor allows less confounding for same experiments

68. Lecture 9 Page 68 CS 239, Spring 2007 So, More Runs or Less Confounding? 2 x runs are often used for preliminary experiments, anyway To identify important factors to look at more deeply How likely is it that an “unlucky” single replication will misidentify important factor?