1. Sampling, sample size estimation, and randomisation
PS302

2. Overview Sampling representative sampling (e.g. for surveys)
homogenous sampling (e.g. for experiments)
Sample size estimation
Based on power
Gathering the information you need
Power calculations (G*Power software)
- ANOVA
- regression
Rules of thumb for multivariate tests
Presentation of power analysis in your report
Practical randomising
Random selection (e.g. for surveys)
Random allocation (e.g. for experiments)

3. Getting a representative sample
Survey of UK Households
want a sample from each SES group
each age group
each sex
Proportions should match the population

4. Matching the population
Percent of population  percent of sample
Assume, sample size = 1200
Population = Women 60%, Men 40% 
Sample: Women 720, Men 480

5. Problem for you to try Population figures: Men 65 years+ = 1 million
Women 65 years+ = 1.5 million
Men years = 8 million
Women years = 8.5 million
Men < 25 years = 5 million
Women < 25 years = 5.2 million
Total population size = 29.2 million
Percent W25-65 = (5.2 / 29.2) * 100
= 17.8%
Given a sample size of 200, how many women <25 years should be included?

6. quota sampling Recruiters are given a quota of each stratum
Problem – biased selection by recruiter/interviewer
Advantage – random selection very difficult to achieve, quota sampling a good compromise

7. Homogenous sampling Restrict sampling to a narrow group
Sample only Warwick students
Sample only one Sex
Sample only one Age group
Advantages
reduces error variance by reducing individual differences

8. Homogenous sampling ctd
Disadvantage – may reduce generalisability
generalisability will need to be considered and assessed separately
Suitability
experimental work
studies where individual differences are not directly relevant and power is more important concern

9. power
Probability that any particular (random) sample will produce a statistically significant effect
Eg. power = 0.9
 90% chance of detecting an effect if there really is an effect
Researchers usually aim to have power at 80-90%

10. Power and sample size
All else being equal, to get more power you need more participants
Where “all else” means:
reliability of measures
other sources of error variance
p-value
the true size of the effect

11. These concepts are inter-related
Desired power ↑ N ↑
Acceptable p-value ↓ N ↑
Effect size to detect ↓ N ↑
Reliability of measures ↓ N ↑
Other error variance ↑ N ↑

12. if you know these…
effect size
variance of measures
you can often work out what the sample size should be
So where can you find them?
Previous research studies

13. Calculating using G-power
First step, assemble the figures needed
For between subjects ANOVA:
Effect size (Cohen’s f, or partial eta squared)
Significance level [.05, usually]
Power [.8, usually]
Numerator degrees of freedom (df)
Number of cells in design (groups)

14. 1. Effect size … from previous studies
Easy – they reported effect sizes
“There was not a significant main effect of Sex on response time, F(1, 42) = 2.03, p = .16, η2 = 0.046”
Harder – they reported only the F and df, so you have to make a calculation
partial η2 = (dfeffect * F) / [(dfeffect * F) + dferror]
= (1 * 2.03) / [(1 * 2.03) + 42]
= 0.046

15. measures of effect size for ANOVA
Roughly, the correlation between an effect
and the outcome (DV)
eta squared
The proportion of variance in the outcome variable (DV) that is explained by the IV
SSeffect / SS[corrected] total
partial eta squared (SPSS prints this out)
The proportion of the effect + error variance explained by the effect
SSeffect / (SSeffect + SSerror)

16. 4. Numerator df
“There was a non-significant main effect of Gender on response time, F(1, 42) = 2.03, p < .05, η2 = 0.09”

17. 5. Number of cells (groups)
Two way ANOVA
2 x 3 ANOVA  6 cells
4 x 2 ANOVA  8 cells
Etc.

18. Calculating using G-power
First step, assemble the figures needed
For this 2 X 3 between subjects ANOVA:
Effect size (η2 = 0.046)
Significance level [.05, as usual]
Power [.8, normal]
Numerator degrees of freedom (df = 1, 2 for the respective main effects, or 2 for the interaction)
Number of cells in design (groups = 6)

19. tip: power & ANOVA Each effect in the ANOVA has its own power
Eg. 2 x 3 ANOVA
Main effect A
Main effect B
Interaction effect A * B
Tip: power is lower for interactions
than for main effects

20. Sample size – ethical issues
Too small a sample
-- can’t detect significant effects
 waste all participants’ time
Too large a sample
-- waste resources
-- waste the extra participants’ time

21. Sample size – practical issues
Resources
Time
Cost of running each participant
Availability
Clinical populations are often small
Access can take time & require permission

22. Choosing an appropriate sample size for established laboratory paradigms
Shortcut
Base sample size on sample size used in previous research
This is often perfectly appropriate
(but make sure the previous research is of high quality!)

23. Rules of thumb for multivariate tests
multiple regression
cases (N) / predictors (p)
N at least p for R2
N at least p for testing a predictor
Need more cases if outcome is skewed, anticipated effect size is small, measures less reliable…

24. Rules of thumb for multivariate tests
PCA (exploratory FA)
50 no good
100 poor
300 good, but ideally need more

25. Random allocation For example
3 between subjects conditions (e.g. control, happy, sad)
Who does which condition?
first come? Interviewer choice?
Must avoid confounds. But can’t check all possible. Solution is random allocation.

26. Random allocation needs truly random numbers
Different ways to do that
SPSS
random.org
Research randomiser
scripting language like python

27. to randomly assign 9 participants to 3 conditions:
Python
to randomly assign 9 participants to 3 conditions:
from random import shuffle
numbers = [1,1,1,2,2,2,3,3,3]
shuffle(numbers)
numbers
[3, 2, 1, 2, 3, 1, 3, 1, 2]

28. Research randomiser http://www.randomizer.org/form.htm
3 conditions, 48 planned participants
randomly: allocate each participant (identified by order of recruitment) to one of 3 conditions
How many sets of numbers to generate? [1]
numbers per set? [48]
Number range? [From 1 To 3]
Do you wish each number in a set to remain unique? [No]
[Don’t “sort”!]

29. Result
Set #1:3, 3, 1, 3, 3, 1, 2, 2, 2, 1, 2, 2, 1, 1, 2, 1, 3, 3, 1, 3, 2, 1, 3, 3, 1, 2, 1, 3, 1, 2, 2, 2, 3, 3, 1, 3, 3, 1, 1, 3, 1, 3, 3, 2, 3, 3, 1, 2

30. Research randomiser http://www.randomizer.org/form.htm
3 sentence types, 48 sentences
16 in each group, create a random sequence, but limit runs of the same type
How many sets of numbers to generate? [16]
numbers per set? [3]
Number range? [From 1 To 3]
Do you wish each number in a set to remain unique? [Yes]

31. Research randomiser http://www.randomizer.org/form.htm
3 types, 48 sentences, 16 of each type
limit run of a given type, while still randomising order of presentation
16 Sets of 3 Unique Numbers Per Set Range: From 1 to 3 -- Unsorted Job Status:      
Set #1:2, 3, 1           
Set #2:3, 1, 2           
Set #3: ….

32. Web links

33. measures of effect size for ANOVA
Roughly, the correlation between an effect
and the outcome (DV)
eta squared
The proportion of variance in the outcome variable (DV) that is explained by the IV
SSeffect / SS[corrected] total
partial eta squared (SPSS prints this out)
The proportion of the effect + error variance explained by the effect
SSeffect / (SSeffect + SSerror)