1. Power, Sample Size, & Effect Size:
Finding an effect when there is one
HMS Academy
Fellowship in Medical Education Research
November 16, 2017
Amy Sullivan & Amy Cohen

2. Statistical Power
DON’T get to the end of your study and find out that you didn’t power it adequately to find an effect.
Power analysis can be used to calculate the minimum sample size required to accept the outcome of a statistical test with a particular level of confidence

3. Why not get the largest sample possible?
You want at least the number required to power your study, but not a great deal more (factoring in expected response rate and attrition, if applicable)
Collecting data with a high response rate requires lots of effort and resources. A well-designed study will aim for the optimal size to detect an effect ✔
Just right
Too big
Too small

4. Definitions Type I error Type II error
When we reject null hypothesis when H0 is true
We use α to estimate the likelihood of a Type I error
(often .05)
Type II error
When we fail to reject the null hypothesis when H0 is false
β represents the probability of a Type II error
Power is equal to 1-β (typically aim for .80)

5. Key Concepts Power (0-1) Sample Size Population Effect Size 0-1+
Alpha level

6. Knowing 3 out of 4 of these, can estimate 4th
Know power, significance level, effect size— can estimate needed sample size
Know sample size, significance level, effect size— can estimate power

7. What is an effect size?
After training one group on effective study skills, I compare the intervention and control groups on their final exam scores. One scored an average score of 86 and the other an average of 80. An independent-groups t-test shows that the mean difference between these groups is statistically significant. How do I know whether or not this is a meaningful difference?

8. Effect size, cont.
Effect sizes describe the magnitude of the difference—Cohen groups these into small (detectable, not trivial), medium, and large. For our example, we would need to know the standard deviation of the pooled test scores to estimate effect size. Let’s say it is 10. Cohen’s d= mean 1- mean 2/pooled sd = 86-80/10=.6 Is this small, medium, or large?

9. Inputs for power calculation
Power calculators will typically ask you to specify whether outcomes are means or proportions, and what the estimated mean and standard deviations are.
You will also need to specify the statistical tests you will be using (e.g., paired t-tests, unpaired t-tests, ANOVA, etc)
What if you don’t know the M and SD? Estimate this from prior research, pilot studies, or what your desired difference would be

10. From Gail Sullivan article

11. Rule of thumb
In general, a .5 SD change in a Likert scale (moderate effect size) tends to be “clinically significant” in survey research

12. Estimating power & sample size in JMP
Demo
Also: PS, G*Power, PASS

13. Sample size estimates will vary depending on your sample design
Basic power calculators will assume a one-stage sample (that is, no clustering or multi-stage samples such as a sample of schools and a sample of students within schools)
If you want to do subgroup analyses, you will also have to power your study to detect subgroup differences

14. Some basics on sample design
First decide whether you will use probability or non-probability sampling
Census versus sample

15. Some basics on sample design
Probability sampling
Simple random: each unit in the sampling frame has the same probability of being selected into the sample
Stratified: first divide the sampling frame into strata (groups, blocks), then do a simple random sample within each strata
Clustered: sample clusters of units. Eg. Residency programs
One stage: Random sample of residency programs, then survey all residents in each program
Two stage: Random sample of residency programs, then survey a sample of residents in each program

16. Stages in the Selection of a Sample Define the target population
Select a sampling frame
Determine if a probability or nonprobability
sampling method will be chosen
Plan procedure
for selecting sampling units
Determine sample size
Select actual sampling units

17. Sampling Frame
A list of elements from which the sample may be drawn

18. Sampling Units Group selected for the sample
Primary Sampling Units (PSU)
Secondary Sampling Units

19. Errors Associated with Sampling
Sampling Frame Error
Random Sampling Error
Nonresponse Error

20. Copyright © 2000 Harcourt, Inc. All rights reserved.
Probability Sampling
Simple Random Sample
Systematic Sample
Stratified Sample
Cluster Sample
Multistage Area Sample
Copyright © 2000 Harcourt, Inc. All rights reserved.

21. Simple Random Sampling
A sampling procedure that ensures that each element in the population will have an equal chance of being included in the sample
Copyright © 2000 Harcourt, Inc. All rights reserved.

22. Copyright © 2000 Harcourt, Inc. All rights reserved.
Systematic Sampling
A simple process
Every nth name from the list will be drawn
Copyright © 2000 Harcourt, Inc. All rights reserved.

23. Clustering and sample size
Clustering reduces efficiency of the design
Standard sample size calculation for individual-based studies only accommodate for variation between individuals
In cluster studies, there are two components of variation
Variation among individuals within clusters
Variation in outcome between clusters

24. Clustering and sample size
Individual-based studies assume independence of outcomes among individuals
In cluster randomization:
Individuals within a cluster are more likely to be similar
Measure of this intracluster dependence among individuals is ICC –Intraclass Correlation
Based in within-cluster variance
High when individuals in cluster are more “similar”
Not taking ICC into account may lead to under-powered study (too small sample)

25. Increasing sample size
Increasing the number of clusters vs increasing the number the individuals per cluster
Increasing the number of clusters has a much stronger effect on power and confidence
Intuitively, the sample is the number of units (clusters) at the level where the random assignment takes place. It is not the same as the number of people surveyed
Challenge is to engineer the logistics to to maximize the number of clusters, given budget

26. Adequately powered study= believable results!