2 Accuracy and Precision

Accuracy How close a measurement is to the actual or “true value” high accuracy true value low accuracy true value 3

Precision How well several measurements agree with each other high precision low precision 4

Precision or the reproducibility of a set of measurements. A precise sample estimate will have a very small random error of estimation. Precision

Accuracy

Precision

Accuracy and Precision What can you say about the accuracy and precision in each of the following: High precision, low accuracy High precision, high accuracy 8

Question: How will each of the following affect accuracy and precision? 1. A meter stick that is missing the first centimeter. 2. A scale that has a zero point that is really five pounds above zero. 9

Solution How will each of the following affect accuracy and precision? 1. The shortened meter stick will produce measurements that have poor accuracy, but good precision is possible. 2. The poorly calibrated scale will give a weight that is not accurate, but good precision is possible. 10

A blood sample was taken from a patient and four different assays were used to measure blood glucose. The true value of the blood glucose was known to be 4.5 mmol/L. State whether the accuracy and the precision of each assay is high or low: Assay A: 4,6; 4,6; 4,8; 4,5; 4,5; 4,4; Assay B: 4,3; 3,5; 5,3; 4,6; 5,5; 3,7 Assay C: 3,5; 3,6; 3,3; 3,5; 3,4; 3,5 Assay D: 8,5; 6,4; 5,3; 7,6; 4,8; 9,3 11

Literature incosistency High accuracy and low precision or low accuracy and low precision?

How confident are we in the estimation of mean or proportion we have calculated?

14

15

Measures of precision: 1. Standard error of mean, SEM Standard error of proportion, SE(p) 2. Confidence interval for mean, CI Confidence interval for proportion, CI

Standard error of mean, SEM Number of patients Standard deviation, SD SEM is smaller (estimate is more precise): the larger is N (number of patients) the smaller is SD (dispersion of data)

95% confidence interval for mean, 95% CI  Together with SEM, 95% CI is also the measure of precision  Unlike SEM, 95% CI also estimates accuracy of the result ie. 95% is accurate that interval includes true (population) mean

95% confidence interval for mean If we draw a 100 samples from our population we would find the true population value within 95% confidence interval in 95 samples. 20 In example for 20 samples:

Critical values for 90%, 95% and 99% level of confidence 90% CI => mean ± 1.65 SEM 95% CI => mean ± 1.96 SEM 99% CI => mean ± 2.58 SEM Level of Confidence - Critical Value 0.75, or 75% , or 80% , or 85% , or 90% , or 95% , or 98% , or 99% 2.58

Example 1 The average systolic BP before treatment in study A, of a group of 100 hypertensive patients, was 170 mmHg. After treatment with the new drug the mean BP dropped by 20 mmHg. If the 95% CI is 15–25, this means: 22 we can be 95% confident that the true effect of treatment is to lower the BP by 15–25 mmHg.

Example 2 In study B 50 patients were treated with the same drug, also reducing their mean BP by 20 mmHg, but with a wider 95% CI of -5 to +45. This CI includes zero (no change). This means: 23 No true change in BP was determined (the drug might be actually ineffective)

Watch out for... The size of a CI is related to the sample size of the study. Larger studies usually have a narrower CI. 24

Width of CI 25

Example 3 – Meta analysis Fig. Plot of 5 studies of a new antihypertensive drug. 1.Which study showed the greatest change? 2.Did all the studies show change in favour of the intervention? 3.Were the changes statistically significant?

Proportion 27

Proportion 1. Standard error of proportion, SE(p) SE( p) = √(p(1 – p)/n) 2. Confidence interval for proportion

The standard deviation describes the variability of a sample; does not describe the sample The standard error of the mean (SEM) does not describe the sample but uncertainty describes the uncertainty of how the sample mean represents the population mean.

Misuse of standard error of the mean (SEM) when reporting variability of a sample. A critical evaluation of four anaesthesia journals. P. Nagele* British Journal of Anaesthesia 90 (4): (2001) Common mistake in the literature

SD CI Standard deviation tells us about the variability (spread) in a sample. The CI tells us the range in which the true value (the mean if the sample were infinitely large) is likely to be.

Krebs NF, Westcott JE, Culbertson DL et. al. Comparison of complementary feeding strategies to meet zinc requirements of older breastfed infants. Am J Clin Nutr. 2012; 96:30-35 CI Meat Fe&Zn Fe

Mean shoulder pain and disability index scores with 95% confidence intervals for supervised exercises and radial extracorporeal shockwave treatment 33 Radial extracorporeal shockwave treatment compared with supervised exercises in patients with subacromial pain syndrome: single blind randomised study

What does a small standard error tell us about the sample estimate of the mean? (Y/N) That it is highly variable That the population standard deviation may be small That the sample size is probably small That it is imprecise

What will tend to make the standard error larger? (Y/N) A small variance A large standard deviation Imprecise data Inaccurate data 35