Presentation on theme: "Sample Size And Power I Jean B. Nachega, MD, PhD Department of Medicine & Centre for Infectious Diseases Stellenbosch University"— Presentation transcript:
Sample Size And Power I Jean B. Nachega, MD, PhD Department of Medicine & Centre for Infectious Diseases Stellenbosch University
Sample Size And Power Warren Browner and Stephen Hulley The ingredients for sample size planning, and how to design them An example, with strategies for minimizing sample size
Sampling and Inference A sample is designed to represent a larger population Therefore, findings in the sample allow inferences about events in the population Problem: what if the inferences are wrong? Finding something in the sample that isn’t “real” in the population Missing something that is “real”
Inference Use a random sample to learn something about a larger population
SUMMARY OF HOW RESEARCH WORKS Hulley S, Cummings S., Newman T et al 2001 ACTUAL STUDY Actual Subjects Actual Measurements FINDINGS IN THE STUDY RESEARCH QUESTION Target population Phenomena of interest TRUTH IN THE UNIVERSE STUDY PLAN Intended Sample Intended Variables TRUTH IN THE STUDY DesignInference Implement Random + System error Random + System error
Parameter Statistic Mean: Standard deviation: Proportion: s X estimates from sample from entire population p
How Do We Generalize? Population Sample generalizeback generalizeback
Inference Two ways to make inference Estimation of parameters * Point Estimation ( X or p) * Intervals Estimation Hypothesis Testing
Preventing Wrong Inferences Difficult when caused by systematic error (bias) Easy when caused by Random error (chance) Solution: increase sample size Problem: cost, feasibility Goldilocks solution: a sample size that is big enough but not too big
Mean, , is unknown PopulationPoint estimate I am 95% confident that is between 40 & 60 Mean X = 50 Sample Interval estimate
Parameter = Statistic ± Its Error
Sampling Distribution X or P
Standard Error SE (Mean) = S n SE (p) = p(1-p) n Quantitative Variable Qualitative Variable
95% Samples Confidence Interval X _ X SE X SE SE Z-axis 1 - α α/2
95% Samples Confidence Interval SE p p SE p SE Z-axis 1 - α α/2
Interpretation of CI Probabilistic In repeated sampling 100(1- )% of all intervals around sample means will in the long run include Practical We are 100(1- )% confident that the single computed CI contains
Example (Sample size≥30) An epidemiologist studied the blood glucose level of a random sample of 100 patients. The mean was 170, with a SD of 10. SE = 10/10 = 1 Then CI: = ≥ % = X + Z SE
18 Study design Investigator assigns exposures? Experimental study Random allocation? Randomised controlled trial Observational study Comparison group? Analytical study Descriptive study Direction? Cohort study Case- control study Cross- sectional study Yes No Exposure -> outcomeExposure <- outcomeExposure & outcome at same time Non- randomised controlled trial
Ingredients For Planning Sample Size in an Analytic Study Hypothesis Null and alternative One-sided vs two-sided Statistical test Based on type of predictor and outcomes variables in the hypothesis Effect size (and its variance, if apply) Power and alpha
An assumption about the population parameter. I assume the mean SBP of participants is 120 mmHg What is a Hypothesis?
Research Hypothesis A clear statement of what you are studying Simple:one predictor, one outcome Specific: who, what, when, where Stated:in advance
Research Hypothesis In patients with early ALS seen at UCT in 2007, those randomly assigned to be treated with newmol will have a lower 1-year mortality than those randomly assigned to placebo.
The Null Hypothesis There’s nothing going on. Purpose in life: to be rejected in favor of its alternative. In patients with early ALS seen at UCSF in 2007, those randomly assigned to be treated with newmol will have the same 1-year mortality as those randomly assigned to placebo.
H 0 Null Hypothesis states the Assumption to be tested e.g. SBP of participants = 120 (H 0 : 120). H 1 Alternative Hypothesis is the opposite of the null hypothesis ( SBP of participants ≠ 120 (H 1 : ≠ 120). It may or may not be accepted and it is the hypothesis that is believed to be true by the researcher Null & Alternative Hypotheses
Defines unlikely values of sample statistic if null hypothesis is true. Called rejection region of sampling distribution Typical values are 0.01, 0.05 Selected by the Researcher at the Start Provides the Critical Value(s) of the Test Level of Significance,
Level of Significance, a and the Rejection Region 0 Critical Value(s) Rejection Regions
True Value of Population Parameter Increases When Difference Between Hypothesized Parameter & True Value Decreases Significance Level Increases When Decreases Population Standard Deviation Increases When Increases Sample Size n Increases When n Decreases Factors Increasing Type II Error n β d
Probability of Obtaining a Test Statistic More Extreme or ) than Actual Sample Value Given H 0 Is True Called Observed Level of Significance Used to Make Rejection Decision If p value Do Not Reject H 0 If p value <, Reject H 0 p Value Test
What’s This All About? A long time ago, statisticians figured out the probability that a sample of a given size would “find something” even if there were nothing going on in the population.
This means that... After a study, we can determine the likelihood that whatever we found in our sample could have occurred by chance... Even if nothing was going on in the population (i.e., the null hypothesis was true)--a “Type I error” If this is very unlikely (say < 1 in 20) we reject the null hypothesis in favor of the alternative hypothesis; we call the finding statistically significant (P <.05)
H 0 : Innocent Jury Trial Hypothesis Test Actual Situation Verdict InnocentGuilty Decision H 0 TrueH 0 False Innocent CorrectError Accept H Type II Error ( ) Guilty Error Correct H 0 Type I Error ( ) Power (1 - ) Jury Trial Example False Negative False Positive Reject
Two-sided Alternative Hypothesis In patients with early ALS seen at UCSF in 2007, those randomly assigned to be treated with newmol will have a different 1-year mortality than those randomly assigned to placebo.
Two One-sided Alternative Hypotheses Side A: In patients with early ALS seen at UCSF in 2007, those randomly assigned to be treated with newmol will have a higher 1-year mortality than those randomly assigned to placebo. Side B: In patients with early ALS seen at UCSF in 2007, those randomly assigned to be treated with newmol will have a lower 1-year mortality than those randomly assigned to placebo.
If The Null Hypothesis Is True By chance alone, each of the two one- sided alternative hypotheses is... Possible Equally likely Wrong Thus a two-sided alternative hypothesis has twice the likelihood of happening by chance alone
35 Measurements Scales in Clinical Research CharacteristicExample Descriptive Stats Info content Categorical (Nominal or Dichotomous) No natural Order Sex; blood; Vital status % Lower Categorical (Ordinal) Ordered Categories Degree of pain Cancer stage % + medians Intermediate Continuous (Interval) Ranked with specified Intervals Weight; Hb; Nb of Cigarettes; Outpatient attendance The above + means; SD, medians Higher
Next Ingredient: Statistical Test (Types of Variable) The statistical test determines how the sample size will be calculated The type of predictor and outcome variable determine which statistical test will be used to analyze the data Both dichotomous: Chi square One dichotomous, one “continuous”: t test Both “continuous”: correlation coeff or t test
Statistical Test (Types of Variable)
Statistical Tests Used in Bivariate Analysis
Statistical Test (Types of Variable) ALS study Predictor: newmol vs placebo Outcome: % dead Both are dichotomous Chi square test
Next Ingredient: Effect Sizes (dichotomous variables) How big an effect you anticipate seeing Newmol halves mortality newmol = 5%, Placebo = 10%
Penultimate Ingredient: Power The chance of finding something in your sample if it’s really going on in the population (avoiding a Type II error) “Something” = the effect size (or greater) Usually set at 80% or 90% = (1 - beta)
…and the Final Ingredient: Alpha The chance of finding something in your sample if there’s nothing going on in the population.
Alpha Explained The level of statistical significance (ie, the p-value that will be considered significant) The pre-set maximum chance of finding something, if it really isn’t there. Usually set at May be one-sided or two-sided.
Sidedness Of Alpha With a two-sided alternative hypothesis, you have two chances of finding something that isn’t really there: One (equal) chance for each side. So a one-sided alpha of 0.05 corresponds to a two-sided alpha of 0.10.
Tools to Calculate Sample Size Formulae General formulae: these can be complex Quick formulae: for particular power and significance levels and specified tests Special Tables for different tests Altman’s Nomogram Computer Software
SAMPLE SIZE: AN EXAMPLE Null hypothesis: In patients with early ALS seen at UCT in 2007, those randomly assigned to be treated with newmol will have the same 1-year mortality as those randomly assigned to placebo. Two-sided alternative hypothesis Dichotomous predictor and outcome Effect size: 10% mortality + 5% Power, alpha:90%, 0.05 (two-sided)
THE SAMPLE SIZE IS… Table 6.B Smaller of P1 and P2 = 0.05; power of 90%; alpha of 0.05 (two-sided) Difference = This is per group
Sample Size Reduction Strategy #1: Statistical Manipulation Use a lower power Use a one-sided alpha Power of 80% One-sided alpha of 0.05
The New Sample Size Is… Table 6.B Smaller of P1 and P2 = 0.05; power of 80%; alpha of 0.05 (one-sided) Difference = This is also per group
SS Reduction Strategy #2: Use A More Common Outcome Change from 1-year mortality to 2-year mortality or loss of independent living Placebo:40% Newmol:20%
The New Sample Size Is… Table 6.B Smaller of P1 and P2 = 0.20; power of 80%; alpha of 0.05 (two-sided) Difference =
SS Reduction Strategy #3: Use A Continuous Outcome Change “mortality” to “muscle strength” NOTE: Big change in research question and research hypothesis. New null hypothesis: In patients with early ALS seen at UCT in 2007, those randomly assigned to be treated with newmol will have the same grip strength at the end of six months as those treated with placebo. Two-sided alternative hypothesis
Estimate The Mean And Variability Of Grip Strength Patients with untreated ALS have a (mean ± SD) grip strength of 20 ± 10 kg after 6 months of disease Newmol may improve that by 25%
Then Grip strength Placebo:20 kg Newmol:25 kg (25% more) Effect size = 5 kg SD = 10 kg Standardized effect size: E/S = 5/10 = 0.5
The New Sample Size Is... Table 6.A E/S = 0.5 ß = 0.20, Alpha (two-sided) = 0.05 N = 64 per group
Ss Reduction Strategy #4: Use A More Precise Outcome Buy a better instrument to measure grip strength Use a well-defined protocol Repeat measurements on two consecutive days Reduce SD from 10 kg to 8 kg
The New Sample Size Is... New E/S = 5 kg/8 kg= ß = 0.20, Alpha (two-sided) = 0.05 N = about 45 per group This helped quite a bit.
SS Reduction Strategy #5: Use Paired Measurements Most of the variability in grip strength at the end of the study is likely to be due to differences between subjects in grip strength at the beginning of the study. Switch the outcome to change in grip strength from the beginning to the end of the study.
Paired Measurements Each subject contributes a pair of measurements: (before, after) The outcome variable is the difference between that pair for each subject. The SD of the change in a measurement is usually < than the SD of the measurement SD of change in grip strength is 5 kg New standardized effect size = 5/5 = 1.0
The New Sample Size Is... E/S = 1.0 ß = 0.20, Alpha (two-sided) = 0.05 N = 17 per group We now have a potentially do-able study, albeit one that is very different from the original aim.
SS Techniques for Descriptive Studies Such studies (including studies of diagnostic tests) do not have a predictor and outcomes variables, nor do they compare groups Therefore the concept of power and null and alternative hypotheses do not apply The investigator calculate calculates descriptive stat such as means and proportions. Descriptive studies commonly report confidence intervals, a range of values about the sample mean or proportion A confidence level is a measure of precision of a sample estimate (e.g. 95%, 99%)
One Group SS Techniques for Descriptive Studies Based on precision of estimation: desire confidence interval of a certain width Requires: Select the Confidence level (or =significance level) Specify desired precision (total width) of the confidence interval [w = 2d] estimate of Standard Deviation or proportion
One Group SS- Continuous For a continuous variable, the 95% CI is Need to specify d (1/2 the width of the CI) Need to estimate 2
Example: One Group SS- Continuous We are interested in estimating the mean age at cancer diagnosis for a certain group of patients. Suppose we would like to estimate the mean age within ± 2.5 years (95% CI of width 5 years). Suppose that we estimate the population’s standard deviation as 12 years.
Example: One Group SS- Continuous We would need a sample size of 89 patients in order to estimate the mean age at diagnosis to within ± 2.5 years
One Group SS - Dichotomous For a dichotomous variable, the 95% CI is Need to specify d (1/2 the width of the CI) Need to estimate p
We would need a sample size of 683 patients in order to estimate the proportion symptom-free to within ± 3% Example: One Group SS- Dichotomous
We would need a sample size of 142 patients in each group in order to detect a 5 mm difference in average blood pressure Example: Two Group SS- Continuous
What sample size is needed to estimate the proportion symptom- free to within ± 5%? What sample size is needed to estimate the proportion symptom- free if p is unknown? Example: One Group SS- Dichotomous
Other considerations and special issues Dropouts: If the investigator estimates that 20% of her sample will be lost to follow-up, then the sample size should be increased by a factor of (1:[1-0.20]) Survival analysis will use the proportion of subjects (dichotomous variable) still alive at each point in time and sample size will be estimated using chi-squared test Sample size calculation in clustered samples are more complex and require the assistance of a statistician
The Bottom Line Sample size estimation is an integral part of study planning and grant writing Almost never the last thing you do More often, one of your first tasks Consult always a statistician, especially if a grant proposal that involves substantial costs, is being submitted for funding
SAMPLE SIZE PLANNING: REVIEW OF INGREDIENTS Looking for something in a sample Hypotheses (null and alternative) Will you be able to... Know it’s there in the population if you find it in your sample ( avoid a Type I error) Test of significance, alpha Find it in your sample if it’s there in the population (avoid a type II error)? Effect size, power
Acknowledgments and Suggested Reading Authors By Stephen B. Hulley, M.D., MPH Steven R. Cummings, M.D. Warren S. Browner, M.D., MPH Deborah Grady, M.D., MPH Norman Hearst, M.D., MPH Thomas B. Neuman, M.D., MPH