1. 1

2. 2 2 k r Factorial Designs with Replications r replications of 2 k Experiments –2 k r observations. –Allows estimation of experimental errors Model: y = q 0 + q A x A + q B x B +q AB x A x B +e e=Experimental error

3. 3 Computation of Effects Simply use means of r measurements Effects: q 0 = 41, q A = 21.5, q B = 9.5, q AB = 5

4. 4 Estimation of Experimental Errors Estimated Response: y i = q 0 + q A x Ai + q B x Bi +q AB x Ai x Bi Experimental Error = Estimated-Measured e ij = y ij - y i = y ij - q 0 - q A x Ai - q B x Bi - q AB x Ai x Bi Sum of Squared Errors: SSE = ^ ^

5. 5 Experimental Errors: Example Estimated Response: y 1 = q 0 - q A - q B + q AB = 41 -21.5 -9.5 +5 = 15 Experimental errors: e 11 = y 11 - y 1 = 15 - 15 = 0 ^ ^ Effect EstimatedMeasured SSE = 0 2 + 3 2 + (-3) 2 + (-3) 2 +... + 4 2 = 102 ^

6. 6 Allocation of Variation Total variation or total sum of squares: SST = SST = SSA + SSB + SSAB + SSE

7. 7 Derivation Model: Since x’s, their products, and all errors add to zero

8. 8 Derivation (cont’d) Mean response:Squaring both sides of the model and ignoring cross product terms: SSY = SS0 + SSA + SSB + SSAB + SSE

9. 9 Derivation (cont’d) Total Variation: One way to compute SSE:

10. 10 Example: Memory-Cache Study

11. 11 Example: Memory-Cache Study(cont’d) SSA + SSB + SSAB + SSE =5547 + 1083 + 300 + 102 = 7032 = SST Factor A explains 5547/7032 or 78.88% Factor B explains 15.40% Interaction AB explains 4.27% 1.45% is unexplained and is attributed to errors.

12. 12 Review: Confidence Interval for the Mean Problem: How to get a single estimate of the population mean from k sample estimates? Answer: Get probabilistic bounds. Eg., 2 bounds, C 1 & C 2   There is a high probability, 1- , that the mean is in the interval (C 1, C 2 ): P r {C 1    C 2 } = 1 -  Confidence interval  (C 1, C 2 )   Significance Level 100 (1-  )  Confidence Level 1-   Confidence Coefficient.

13. 13 Confidence Interval for the Mean (cont’d) Note: Confidence Level is traditionally expressed as a percentage (near 100%); whereas, significance level , is expressed as a fraction & is typically near zero; e.g., 0.05 or 0.01.

14. 14 Confidence Interval for the Mean (cont’d) Example: Given sample with: mean = x = 3.90 SD = s = 0.95 n = 32 A 90 % CI for the mean = 3.90 + (1.645)(0.95)/ = (3.62, 4.17), used the central limit theorem. Note: A 90 % CI => We can state with 90 % confidence that the population mean is between 3.62 & 4.17. The chance of error in this statement is 10 %

15. 15 Testing for a Zero Mean Difference in processor times of two different implementations of the same algorithms was measured on 7 similar workloads. The differences are: {1.5, 2.6, -1.8, 1.3, -0.5, 1.7, 2.4} Can we say with 99 % confidence that one implementation is superior to the other

16. 16 Testing for a Zero Mean (cont’d) Sample size = n = 7 mean = x = 1.03 sample variance = s 2 = 2.57 sample deviation = s = 1.60 CI = 1.03  t x 1.60/ = 1.03  0.605t 100 (1-  ) = 99,  = 0.01, 1-  /2 = 0.995 From Table, the t value at six degrees of freedom is: t [0.995; 6] = 3.707 & the 99% CI = (-1.21, 3.27). Since the CI includes zero, we can not say with 99% confidence that the mean difference is significantly different from Zero.

17. 17 Type I & Type II Errors In testing a NULL, hypothesis, the level of significance is the probability of rejecting a true hypothesis. HYPOTHESIS Actually TrueActually False Correct  Error (Type II) Correct  Error (Type I) To Accept To Reject DECISIONDECISION Note: The letters  &  denote the probability related to these errors

18. 18 Since q 0 = Linear combination of normal variables => q 0 is normal with variance Variance of errors: Effects are random variables. Errors ~ N(0,σ e ) => y ~ N( y.., σ e ) Confidence Intervals For Effects

19. 19 Confidence Intervals For Effects (cont’d) Denominator = 2 2 (r - 1) = # of independent terms in SSE => SSE has 2 2 (r - 1) degrees of freedom. Estimated variance of q 0 : Similarly, Confidence intervals (CI) for the effects: CI does not include a zero => significant

20. 20 Example For Memory-cache study: Standard deviation of errors: Standard deviation of effects: For 90% Confidence :

21. 21 Example (cont’d) Confidence intervals: No zero crossing => All effects are significant.

22. 22 Confidence Intervals for Contrasts Contrast  Linear combination with  coefficients = 0 Variance of  h i q i : For 100 ( 1 -  ) % confidence interval, use

23. 23 u = q A + q B -2q AB Coefficients = 0,1,1, and -2 => Contrast Mean u = 21.5 + 9.5 - 2 x 5 = 21 Variance Example: Memory-cache study Standard deviation t [0.95;8] = 1.86 90% Confidence interval for u : (16.31, 25.69)

24. 24 Mean response y : y = q 0 + q A x A + q B x B + q AB x A x B The standard deviation of the mean of m response: CI for Predicted Response ^ ^ n eff = Effective deg of freedom = Total number of runs 1 + Sum of DFs of params used in y ^

25. 25 CI for Predicted Response (cont’d) 100 ( 1 -  ) % confidence interval: A single run (m =  1) : Population mean

26. 26 Example: Memory-cache Study For x A = -1 and x B = -1: A single confirmation experiment: y 1 = q 0 - q A - q B + q AB = 41 - 21.5 - 9.5 + 5 = 15 Standard deviation of the prediction: ^ Using t [0.95;8] =1.86, the 90% confidence interval is:

27. 27 Example: Memory-cache Study (cont’d) Mean response for 5 experiments in future: The 90% confidence interval is: Mean response for a large number of experiments in future: The 90% confidence interval is:

28. 28 Example: Memory-cache Study (cont’d) Current mean response: Not for future. (Use the formula for contrasts): 90% confidence interval: Notice: Confidence intervals become narrower.

29. 29 Assumptions 1. Errors are statistically independent. 2. Errors are additive. 3. Errors are normally distributed 4. Errors have a constant standard deviation  e. 5. Effects of factors are additive. => observations are independent and normally distributed with constant variance.