1. Random error, Confidence intervals and P-values
Simon Thornley 1

2. Overview Simulated, repeated, epidemiological study to understand:
Random error
confidence intervals
p-values
Sample size calculations 2

3. Interpreting study results
Estimate OR/RR and 95% C. I.
Is there an association between exposure and outcome?
Is P <0.05 or 95% CI for measure of association contain null value (1)?
Hypothesis likely false No
Consider type-2 error; confounding, bias, other studies
Yes
Exposure is associated with disease
Is there another explanation?
Bias
Confounding
Type-1 error
(consider strength of association)
Information
(recall)
Shared common cause of exposure and disease?
Selection
(survivor; loss to follow up, hosp. controls)
How does my study compare with others?
Could study design be improved?
Regression or stratified analysis

4. What is random error?
Only one type of error in epidemiological studies – others?
We rarely have whole population,
Rely on sample
Who is picked? chance?

5. 20 sided dice Rolling dice = outcome e.g. Diarrhoea
10 rolls =exposed subjects
6 to 20 = diarrhoea in exposed
10 rolls unexposed.
16 to 20 = diarrhoea in unexposed

6. a b c d Total a+c b+d Diarrhoea No diarrhoea Exposed Unexposed
Exposed (10 rolls)
1 roll = 1 participant
Unexposed (10 rolls)
≥6 (diarrhoea)
<6
(no diarrhoea)
≥16 (diarrhoea)
<16 a b c d
Diarrhoea
No diarrhoea
Exposed a b
Unexposed c d
Total
a+c
b+d

7. What is Risk/odds ratio?
Assume dice fair
calculate ‘true’ risk ratio and odds ratio of the outcome?

8. What is the true odds ratio?
Risk in exposed?
Risk in unexposed?
Odds in exposed?
Odds in unexposed? 8

9. Risk in exposed?

10. =15/20

11. Risk in unexposed?

12. =5/20

13. Odds among exposed?
1/3
4/3 3

14. Odds among unexposed?
1/3 3 4

15. True odds ratio? 3 4 5 6 8 9

16. What is odds ratio? Odds in exposed is 15/5 = 3
Odds in unexposed is 5/15 = 1/3
Odds ratio = 3/(1/3) = 9
Risk ratio is three, but bounded by prevalence in unexposed (denominator), which is ¼. Reciprocal of this (4) is upper limit of risk ratio.

17. Aggregate outcomes Rolling dice = defines outcome e.g. Diarrhoea
10 students exposed; 10 unexposed.
Roll dice 20 times each; ≥6 diarrhoea in exposed; ≥16 diarrhoea in unexposed
Simulated epidemiological study, n=10 exposed, n=10 unexposed. For exposed, >5 = disease; for unexposed >15 = disease; otherwise no-disease.

18. Every pair: Two by two table
Diarrhoea
No diarrhoea
Exposed a b
Unexposed c d
a+c
b+d
Get students to prepare two by two table to summarise results. 18

19. Odds ratio odds of exposure in cases
odds of exposure in controls
= a*d
b*c
Also odds ratio, if time.

20. 95% confidence interval for odds ratio
se(log OR) = √[1/a+1/b+1/c+1/d]
Error factor (EF) = exp[1.96 x s.e.(log OR)]
95%CI = OR / EF to OR x EF
Calculate ‘z’ statistic =log (OR)/se(log OR)
Use tables to convert to p-value
This can be difficult, no need to calculate p-value. 20

21. My example Diarrhoea No diarrhoea Exposed 6 4 Unexposed 2 8 12
Example for students to work through, if having trouble.
OR =(6/4)/(2/8)=(6*8) /(2*4)=48/8 = 6
SE (log(OR)) = √[1/6+1/4+1/2+1/8] =1.02
Error factor = exp[1.96 x s.e.(1.02)] =7.39
95% CI = 6/7.39 to 6* = 0.81 to 44.4
z-statistic = log(OR)/se(log(OR) =1.76
P =0.08
True odds ratio is... 9
Does calculated 95% odds ratio contain true value?

22. No significant diff. What does this mean?
What is the “null” hypothesis?
What is the “alternate” hypothesis?

23. No significant diff. Null hypothesis (no difference)
That P(Diarrhoea | Exposed) = 0.5
P(Diarrhoea | Unexposed) = 0.5
OR P(D|E) - P(D|no E) = 0
Alternate hypothesis (yes, a difference)
P(D|E)- P(D|no E) = delta (δ)

24. Which of the following are not affected by the study sample size?
95% confidence interval
P-value
Bias
Random error

25. Notice that the point estimates for larger study samples are getting closer to the true value, and the 95% CI is narrower. P(type-2 error) is lower if sample size ↑
True odds ratio (9)
This are the odds ratios from the student dice rolling exercises. 19/20 95% confidence intervals should include the true effect (OR=9)
Doesn’t contain true value (95% of intervals should!) see definition of confidence interval
Type-2 errors: cross null value of 1 (3/10 or 30%). All other groups made correct inference from study data (rejected null hypothesis)

26. Errors in hypothesis testing
Test result (P or 95% CI)
Null Hypothesis
(No difference)
Accept Null
(P < 0.05 or 95% CI includes 1)
Reject Null
(P ≥ 0.05 or 95% CI excludes 1)
True
(exposure doesn’t cause disease) OK
Type-1 error
False
(exposure causes disease)
Type-2 error
This is the so called “Neyman-Pearson” approach to hypothesis testing. We fix the probability of making a type 1 error (concluding an effect is present, when there is none), and type 2 error (concluding no effect is present, when the exposure truly does influence outcome), and calculate the sample size required, based on this.

27. v/2 u (~10%) v/2 (~5%) H0: exp doesn’t cause disease
HAlt: exp. associated with disease
0.75
Distribution of risk of disease if no effect of exposure (null hypothesis)
Distribution of risk of disease if exposure associated with disease (alternative hypothesis)
When we perform a study, our results come from a hypothesised distribution. The null in our sample, represents 0.5. Our estimate for the exposed, is most likely to land at 0.75, but as you have seen, this may be different, due to chance. We have a small chance of getting a result that suggests our exposure has no effect on the outcome. The width of the probability curves can be reduced by increasing the numbers in the study. To accomplish this we fix the shaded and solid areas, estimate the likely treatment effect (0.75) and then use a calculation to find what width of curve is produced by a sample size that we recruit.
v/2
v/2
0.50
0.75
u (~10%)
v/2 (~5%)

28. Sample size n=sample in each group π0=risk in unexposed (1/4)
π1=risk in exposed (3/4)
u=1 sided AUC of normal dist. Corresponding to 100%-power (eg. 10%; u=1.28)
v=2 sided z level corresponding to % of AUC of normal dist. for two sided significance level required (5%, v=1.96)
Ugly formula for sample size. Put in for interest, not examinable, but principles are.

29. Our example... n = 18.8 per group (38 total)
We were unlucky not to find a significant difference between the two samples

30. Simpler formula... n=16 per group or 32 total
Simplification, pi hat = average of null (pi 0) and alternative (pi 1). 16*0.6*0.4/(0.25)^2 = 16

31. Sample size vs. power
The statistical power (1-probability of type 2 error) increases as sample size increases. Most investigators consider 90% power (0.9) standard for most study designs.

32. 5 different effect measures
We have 10 repetitions of the study; How many of these 95% confidence intervals would you expect to contain the true value?
How many studies draw the incorrect conclusion of no difference between the treatment and no treatment groups using p-values?

33. Two approaches Yes/No→ ‘p-value’
What is the true difference? →’95% confidence interval’ 33

34. Confidence Interval What is the true difference?
95% confidence interval is not
“95% probability that the true effect estimate lies within the confidence interval”
“If we repeated the study over and over, and calculate 95% confidence intervals on each, 19/20 times, these intervals will contain the true effect estimate”
Population
Confidence interval
Sample 34

35. 95% CI
“a series of values compatible with the true value”

36. Confidence Interval
If 95% confidence interval does not contain 0 (difference between means) or 1 (ratio effect measures) the null hypothesis can be rejected.
Increasing popularity over p-values...

37. P-value Propose ‘null’ hypothesis:
no effect of eating contaminated food on risk of diarrhoea
Is observed difference
True vs. spurious (sampling error, chance)?
Gives Yes or No answer
P-value = P(observed results or more extreme | null true)
If P-value low (P < 0.05):
Chance is unlikely to explain result
Reject null hypothesis 37

38. Problems with P-values
NOT- probability that the null hypothesis is true
Wrong conditional interpretation.
Affected by both sample size and effect size
Sample size is by design…
Meta-analyses
Is the effect clinically significant?
Are the null and alternate hypotheses sufficiently different?

39. Problems with P-values
Imagine study; All given two treatments. At follow up which drug do you prefer?
Total
Preference
Treatment: placebo
% Preference
Two sided p-value 20
15:5
75%
0.04
200
115:86
57.50%
2000
1046:954
52.30% :
50.07%

40. Which of the following statements about p-values are true?
A) Probability of the observed results (or more extreme) if we assume there is no association
B) Probability null hypothesis is true
C) Unaffected by the sample size
D) Used to adjudicate the presence of confounding.

41. Imagine…. Redesigned the dice rolling expt.
The risks are the same in both cases and controls of getting disease (1 to 10 = disease; 11 to 20 = no disease).
What is the chance of a type-1 (false positive) error?
Is the type-1 error rate improved by increasing the sample size?

42. Summary
Confidence interval – range of values within which we are reasonably confident that the population measure lies P-value – strength of evidence against the null hypothesis that the true difference in the population is zero.

43. Type-2 error is best described as:
Not seeing an effect that truly is there.
Detecting an effect that really isn’t there
Bias
Increases with increasing sample size

44. Type-1 error is: Detecting an effect that really isn’t there
Not detecting an effect that really is there
Bias
Not fixed.

45. The probability of a type-1 error is usually...
20% 5% 1%
10%

46. a) It is a true effect of the exposure on the outcome.
During the analysis of the results of this epidemiological study, you derive a crude odds ratio with a 95% which excludes the null value of 1. Which of the following could not explain the result you have found?
a) It is a true effect of the exposure on the outcome.
b) It is a type-2 error
c) It is a type-1 error
d) It is due to information bias
e) It is due to the presence of unadjusted confounding factors.

47. Which of the following measures of disease frequency is able to be estimated from a cross-sectional study?
Incidence rate
Cumulative incidence
Prevalence
Risk ratio