Manijeh Keshtgary Chapter 13

 How to report the performance as a single number? Is specifying the mean the correct way?  How to report the variability of measured quantities? What are the alternatives to variance and when are they appropriate?  How to interpret the variability?  How much confidence can you put on data with a large variability?  How many measurements are required to get a desired level of statistical confidence?  How to summarize the results of several different workloads on a single computer system?  How to compare two or more computer systems using several different workloads? Is comparing the mean sufficient? 2

 Sample Versus Population  Confidence Interval for The Mean  Approximate Visual Test  One Sided Confidence Intervals  Sample Size for Determining Mean 3

 Old French word `essample'  `sample' and `example'  One example  theory  One sample  Definite statement 4

5  Generate several million random numbers with mean  and standard deviation   Draw a sample of n observations: {x 1, x 2, …, x n } ◦ Sample mean (x)  population mean (  )  Parameters: population characteristics ◦ Unknown, Use Greek letters (   Statistics: Sample estimates ◦ Random, Use English letters (x, s)

 it is not possible to get a perfect estimate of the population mean from any finite number of finite size samples.  The best we can do is to get probabilistic bounds. Thus, we may be able to get two bounds, for instance, c1 and c2,

 k samples  k Sample means ◦ Can't get a single estimate of  ◦ Use bounds c 1 and c 2 :  Probability{c 1    c 2 } = 1-  (  is very small)  Confidence interval: [(c 1, c 2 )]  Significance level:   Confidence level: 100(1-  )  Confidence coefficient: 1-  7  c1c1 c2c2

 Use 5-percentile and 95-percentile of the sample means to get 90% Confidence interval  Need many samples (n > 30)  Central limit theorem: Sample mean of independent and identically distributed observations: Where  = population mean,  = population standard deviation  Standard Error: Standard deviation of the sample mean  100(1-  )% confidence interval for  : z 1-  /2 = (1-  /2)-quantile of N(0,1) 8  -z 1-  /2 z 1-  /2

 For example, for a two–sided confidence interval at 95%, α= 0.05 and p = 1 – α/2 = The entry in the row labeled 0.97 and column labeled gives z p =

 x = 3.90, s = 0.95 and n = 32  A 90% confidence interval for the mean =  We can state with 90% confidence that the population mean is between 3.62 and The chance of error in this statement is 10%. 10

 If we take 100 samples and construct confidence interval for each sample, the interval would include the population mean in 90 cases. 11  c1c1 c2c2 Total yes > 100(1-  )

 The preceding confidence interval applies only for large samples, that is, for samples of size greater than 30.  For smaller samples, confidence intervals can be constructed only if the observations come from a normally distributed population  100(1-  ) % confidence interval for n < 30

◦ Note: can be constructed only if observations come from a normally distributed population  t [1-  /2; n-1] = (1-  /2)-quantile of a t-variate with n-1 degrees of freedom ◦ Listed in Table A.4 in the Appendix 13

 Table A.4 lists t [p;n]. For example, the t [0.95;13] required for a two–sided 90% confidence interval of the mean of a sample of 14 observation is 1.771

 Sample ◦ -0.04, -0.19, 0.14, -0.09, -0.14, 0.19, 0.04, and  Mean = 0, Sample standard deviation =  For 90% interval: t [0.95;7] =  Confidence interval for the mean 15

16

 Difference in processor times: {1.5, 2.6, -1.8, 1.3, -0.5, 1.7, 2.4}  Question: Can we say with 99% confidence that one is superior to the other? ◦ Sample size = n = 7 ◦ Mean = 7.20/7 = 1.03 ◦ Sample variance = ( *7.20/7)/6 = 2.57 ◦ Sample standard deviation = = 1.60 t [0.995; 6] =  99% confidence interval = (-1.21, 3.27) 17

 Opposite signs  we cannot say with 99% confidence that the mean difference is significantly different from zero  Answer: They are same  Answer: The difference is zero 18

 Difference in processor times ◦ {1.5, 2.6, -1.8, 1.3, -0.5, 1.7, 2.4}.  Question: Is the difference 1?  99% Confidence interval = (-1.21, 3.27) ◦ The confidence interval includes 1 =>  Yes: The difference is 1 with 99% of confidence 19

 Paired: one-to-one correspondence between the i th test of system A and the i th test on system B ◦ Example: Performance on i th workload ◦ Straightforward analysis: the two samples are treated as one sample of n pairs ◦ Use confidence interval of the difference  Unpaired: No correspondence ◦ Example: n people on System A, n on System B  Need more sophisticated method ◦ t-test procedure 20

 Performance: {(5.4, 19.1), (16.6, 3.5), (0.6, 3.4), (1.4, 2.5), (0.6, 3.6), (7.3, 1.7)}. Is one system better?  Differences: {-13.7, 13.1, -2.8, -1.1, -3.0, 5.6}.  Answer: No. They are not different (the confidence interval includes zero) 21

 1. Compute the sample means  2. Compute the sample standard deviations 22

 3. Compute the mean difference  4. Compute the standard deviation of the mean difference  5. Compute the effective number of degrees of freedom  6. Compute the confidence interval for the mean difference  7. If the confidence interval includes zero, the difference is not significant 23

 Times on System A: {5.36, 16.57, 0.62, 1.41, 0.64, 7.26} Times on system B: {19.12, 3.52, 3.38, 2.50, 3.60, 1.74}  Question: Are the two systems significantly different?  For system A:  For System B: 24

 The confidence interval includes zero  the two systems are not different 25

26

 Times on System A: {5.36, 16.57, 0.62, 1.41, 0.64, 7.26} Times on system B: {19.12, 3.52, 3.38, 2.50, 3.60, 1.74} t [0.95, 5] =  The 90% confidence interval for the mean of A = 5.31  (2.015) = (0.24, 10.38)  The 90% confidence interval for the mean of B = 5.64  (2.015) = (0.18, 11.10)  Confidence intervals overlap and the mean of one falls in the confidence interval for the other ◦ Two systems are not different at this level of confidence 27

 Need not always be 90% or 95% or 99%  Based on the loss that you would sustain if the parameter is outside the range and the gain you would have if the parameter is inside the range  Low loss  Low confidence level is fine ◦ E.g., lottery of 5 Million, one dollar ticket cost, with probability of winning (one in 10 million) ◦ 90% confidence  buy 9 million tickets (and pay $9M) ◦ 0.01% confidence level is fine  50% confidence level may or may not be too low  99% confidence level may or may not be too high 28

 Time between crashes was measured for two systems A and B. The mean and standard deviations of the time are listed in Table To check if System A is more susceptible to failures than System B,

 Larger sample  Narrower confidence interval resulting in higher confidence  Question: How many observations n to get an accuracy of § r% and a confidence level of 100(1-  )%?  We know that for a sample of size n, the 100(1 - α)% confidence interval of the population mean is  r% accuracy implies that confidence interval should be 33

 Sample mean of the response time = 20 seconds Sample standard deviation = 5 Question: How many repetitions are needed to get the response time accurate within 1 second at 95% confidence?  Required accuracy = 1 in 20 = 5% Here, = 20, s= 5, z= 1.960, and r=5, n = A total of 97 observations are needed. 34

 All statistics based on a sample are random and should be specified with a confidence interval  If the confidence interval includes zero, the hypothesis that the population mean is zero cannot be rejected  Paired observations  Test the difference for zero mean  Unpaired observations  More sophisticated t-test 35