Sample Size and Accuracy

Sample Size and Accuracy How do we define accuracy? Impact of sample size on Confidence Interval. Impact of Confidence Level on Confidence Interval. Impact of Most Likely Estimate (MLE). Impact of MLE on confidence interval. Impact of MLE on sample size. Impact of Population size on Sample size. Calculating Sample Size. NEXT

How do we define accuracy? Accuracy is defined using two parameters: Confidence Interval (CI) and Confidence Level (CL) NEXT

Confidence Interval Confidence Interval is a range surrounding your estimate that the true population value might be in. Smaller Confidences Intervals imply more accurate results. NEXT

Confidence Interval Example: After a survey of 50 files, you find that 15 (30%) have major omissions. 30% is the Most Likely Estimate (MLE). What is the lowest and highest possible value be for the population? 30% 0% 20% 40% 60% 80% 100% Percentage of files with omissions NEXT

Confidence Interval If the Confidence Interval (CI) is ±10%, then the lowest likely value is 20%. This is called the Lower Confidence Limit (LCL) LCL = MLE - CI LCL = 30% - 10% = 20% LCL 30% 0% 20% 40% 60% 80% 100% Percentage of files with omissions NEXT

Confidence Interval If the Confidence Interval (CI) is ± 10%, then the highest likely value is 40%. This is called the Upper Confidence Limit (UCL) UCL = MLE + CI UCL = 30% + 10% = 40% LCL 30% UCL 0% 20% 40% 60% 80% 100% Percentage of files with omissions NEXT

Confidence Level Confidence Level (CL) is the probability that the true value is between the LCL and UCL (within one CI of the MLE) If the CL is 95%, then the true population value has a 95% chance of being within one CI of the MLE. We can express CL as a percentage (95%), as a proportion (.95) or as a probability (19 times our of 20). NEXT

Examples of CI and CL The estimate is 40% (CI = ± 10% and CL = 90%) The estimate of 40% is accurate with ±10%, 18 times out of 20. The estimate is 66%, (CI = ± 5%, and CL = 95%) The estimate of 66% is accurate with ±5%, 19 times out of 20. The estimate is 80% (CI = ± 12%, and CL = 99%) The estimate of 80% is accurate with ±12%, 99 times out of 100. NEXT

Minimum level of acceptable accuracy. In general, we should never report estimates below a minimum level of accuracy. A common standard is having a CI no larger than 10%, and a CL no smaller than 90%. CI  10% CL  90% NEXT

Sample Size and Accuracy How do we define accuracy? Impact of sample size on Confidence Interval. Impact of Confidence Level on Confidence Interval. Impact of Most Likely Estimate (MLE). Impact of MLE on confidence interval. Impact of MLE on sample size. Impact of Population size on Sample size. Calculating Sample Size. NEXT

Impact of sample size on confidence interval. In general, as sample size increases, the Confidence Interval gets smaller (more accurate). Hence, larger samples yield more accurate results than smaller samples. NEXT

Impact of sample size on confidence interval. Notice how CI drops quickly with initial increases in sample size, and then levels off. This is called the “principle of diminishing returns.” NEXT

Impact of sample size on confidence interval. As sample size increases, the confidence interval gets smaller (more accurate). In the following demonstration, assume the MLE is 50%. With a sample of 10, the CI is ±26% (CL = 90%) With a sample of 30, the CI is ±15% (CL = 90%) With a sample of 50, the CI is ±11% (CL = 90%) MLE=50% 0% 100% 20% 40% 60% 80% Percentage of files with omissions NEXT

Sample Size and Accuracy How do we define accuracy? Impact of sample size on Confidence Interval. Impact of Confidence Level on Confidence Interval. Impact of Most Likely Estimate (MLE). Impact of MLE on confidence interval. Impact of MLE on sample size. Impact of Population size on Sample size. Calculating Sample Size. NEXT

Impact of Confidence Level on Confidence Interval. As you decrease the Confidence Level, the Confidence Interval also decreases. NEXT

Impact of Confidence Level on Confidence Interval. Larger confidence levels require larger confidence intervals. With a CL=99%, CI is ±17% (Sample size=50) With a CL=95%, CI is ±13% (Sample size=50) With a CL=90%, CI is ±11% (Sample size=50) MLE=50% 0% 100% 20% 40% 60% 80% Percentage of files with omissions NEXT

Sample Size and Accuracy How do we define accuracy? Impact of sample size on Confidence Interval. Impact of Confidence Level on Confidence Interval. Impact of Most Likely Estimate (MLE). Impact of MLE on confidence interval. Impact of MLE on sample size. Impact of Population size on Sample size. Calculating Sample Size. NEXT

Impact of Most Likely Estimate (MLE). The results of the survey (MLE) also has an impact on the accuracy of the results. Results from different questions can have different CIs, depending on the MLE of each question. 20% say “YES”: CI = ±8.8% 35% strongly disagree: CI = ± 10.5% 50% of files have errors: CI = ± 11% 90% have high job satisfaction: CI = ± 6.6% NEXT

Impact of MLE on confidence interval. The Confidence Interval (accuracy) of any result is affected by the MLE of that result. The Confidence Interval is largest (most inaccurate) when MLE=50%. NEXT

Impact of MLE on confidence interval. CI changes with the results (MLE). CI is largest when MLE = 50% When MLE = 10%, the CI is ±7% (CL = 90%) When MLE = 85%, the CI is ±8% (CL = 90%) When MLE = 50%, the CI is ±11% (CL = 90%) MLE=10% MLE=50% MLE=85% 0% 100% 20% 40% 60% 80% Percentage of files with omissions NEXT

Impact of MLE on sample size. When choosing a sample size, it must be large enough to meet the criteria of CI  10% & CL  90%. The minimum sample size varies depending on the results you will get. When you have no idea what results you will get, assume MLE will be 50% (at 50%, the sample size needed is at its largest). If, from past experience, you have a good idea of what the MLE will be, you can adjust your sample size to suit the expected findings. NEXT

Impact of MLE on sample size. You can use an estimate of MLE to choose an appropriate sample size. NEXT

Sample Size and Accuracy How do we define accuracy? Impact of sample size on Confidence Interval. Impact of Confidence Level on Confidence Interval. Impact of Most Likely Estimate (MLE). Impact of MLE on confidence interval. Impact of MLE on sample size. Impact of Population size on Sample size. Calculating Sample Size. NEXT

Impact of Population size on Sample size. As the population increases, the sample size needs to increase. Sample size does not increase proportionately with population size. For very small populations (<100), a very large percentage is needed for the sample. For very large populations (>1000), a very small percentage is needed for the sample. NEXT

Impact of Population size on Sample size. As the population increases, the sample size increases. Notice that the “percentage of the population” decreases as population increases. Sample size needed based on Population size. (MLE = 50%, CI = ±10% @ CL = 90%) NEXT

Sample Size and Accuracy How do we define accuracy? Impact of sample size on Confidence Interval. Impact of Confidence Level on Confidence Interval. Impact of Most Likely Estimate (MLE). Impact of MLE on confidence interval. Impact of MLE on sample size. Impact of Population size on Sample size. Calculating Sample Size. NEXT

Calculating Sample Size. In order to calculate sample size (n), you need to know: Population size (N). Most Likely Estimate (if unknown, MLE = 50%). Desired Confidence Interval (CI  10%). Desired Confidence Level (CL  90%). NEXT

Methods of Calculating Sample Size Sample size calculators are available from the Internet, but many assume MLE is only 50%, or many of these calculators are not suitable for MLE < 25%. The sample size calculator in IDEA is suitable for any size MLE. Please consult with you SME Survey for a tutorial on calculating sample sizes with IDEA.