Random error, Confidence intervals and P-values Simon Thornley 1

Overview Simulated, repeated, epidemiological study to understand: Random error confidence intervals p-values Sample size calculations 2

Interpreting study results Estimate OR/RR and 95% C. I. Is there an association between exposure and outcome? Is P <0.05 or 95% CI for measure of association contain null value (1)? Hypothesis likely false No Consider type-2 error; confounding, bias, other studies Yes Exposure is associated with disease Is there another explanation? Bias Confounding Type-1 error (consider strength of association) Information (recall) Shared common cause of exposure and disease? Selection (survivor; loss to follow up, hosp. controls) How does my study compare with others? Could study design be improved? Regression or stratified analysis

What is random error? Only one type of error in epidemiological studies – others? We rarely have whole population, Rely on sample Who is picked? chance?

20 sided dice Rolling dice = outcome e.g. Diarrhoea 10 rolls =exposed subjects 6 to 20 = diarrhoea in exposed 10 rolls unexposed. 16 to 20 = diarrhoea in unexposed

a b c d Total a+c b+d Diarrhoea No diarrhoea Exposed Unexposed Exposed (10 rolls) 1 roll = 1 participant Unexposed (10 rolls) ≥6 (diarrhoea) <6 (no diarrhoea) ≥16 (diarrhoea) <16 a b c d Diarrhoea No diarrhoea Exposed a b Unexposed c d Total a+c b+d

What is Risk/odds ratio? Assume dice fair calculate ‘true’ risk ratio and odds ratio of the outcome?

What is the true odds ratio? Risk in exposed? Risk in unexposed? Odds in exposed? Odds in unexposed? 8

Risk in exposed?

=15/20

Risk in unexposed?

=5/20

Odds among exposed? 1/3 ½ ¾ 4/3 3

Odds among unexposed? 1/3 ½ ¼ 3 4

True odds ratio? 3 4 5 6 8 9

What is odds ratio? Odds in exposed is 15/5 = 3 Odds in unexposed is 5/15 = 1/3 Odds ratio = 3/(1/3) = 9 Risk ratio is three, but bounded by prevalence in unexposed (denominator), which is ¼. Reciprocal of this (4) is upper limit of risk ratio.

Aggregate outcomes Rolling dice = defines outcome e.g. Diarrhoea 10 students exposed; 10 unexposed. Roll dice 20 times each; ≥6 diarrhoea in exposed; ≥16 diarrhoea in unexposed Simulated epidemiological study, n=10 exposed, n=10 unexposed. For exposed, >5 = disease; for unexposed >15 = disease; otherwise no-disease.

Every pair: Two by two table Diarrhoea No diarrhoea Exposed a b Unexposed c d a+c b+d Get students to prepare two by two table to summarise results. 18

Odds ratio odds of exposure in cases odds of exposure in controls = a*d -------- b*c Also odds ratio, if time.

95% confidence interval for odds ratio se(log OR) = √[1/a+1/b+1/c+1/d] Error factor (EF) = exp[1.96 x s.e.(log OR)] 95%CI = OR / EF to OR x EF Calculate ‘z’ statistic =log (OR)/se(log OR) Use tables to convert to p-value This can be difficult, no need to calculate p-value. 20

My example Diarrhoea No diarrhoea Exposed 6 4 Unexposed 2 8 12 Example for students to work through, if having trouble. OR =(6/4)/(2/8)=(6*8) /(2*4)=48/8 = 6 SE (log(OR)) = √[1/6+1/4+1/2+1/8] =1.02 Error factor = exp[1.96 x s.e.(1.02)] =7.39 95% CI = 6/7.39 to 6*7.39 = 0.81 to 44.4 z-statistic = log(OR)/se(log(OR) =1.76 P =0.08 True odds ratio is... 9 Does calculated 95% odds ratio contain true value?

No significant diff. What does this mean? What is the “null” hypothesis? What is the “alternate” hypothesis?

No significant diff. Null hypothesis (no difference) That P(Diarrhoea | Exposed) = 0.5 P(Diarrhoea | Unexposed) = 0.5 OR P(D|E) - P(D|no E) = 0 Alternate hypothesis (yes, a difference) P(D|E)- P(D|no E) = delta (δ)

Which of the following are not affected by the study sample size? 95% confidence interval P-value Bias Random error

Notice that the point estimates for larger study samples are getting closer to the true value, and the 95% CI is narrower. P(type-2 error) is lower if sample size ↑ True odds ratio (9) This are the odds ratios from the student dice rolling exercises. 19/20 95% confidence intervals should include the true effect (OR=9) Doesn’t contain true value (95% of intervals should!) see definition of confidence interval Type-2 errors: cross null value of 1 (3/10 or 30%). All other groups made correct inference from study data (rejected null hypothesis)

Errors in hypothesis testing Test result (P or 95% CI) Null Hypothesis (No difference) Accept Null (P < 0.05 or 95% CI includes 1) Reject Null (P ≥ 0.05 or 95% CI excludes 1) True (exposure doesn’t cause disease) OK Type-1 error False (exposure causes disease) Type-2 error This is the so called “Neyman-Pearson” approach to hypothesis testing. We fix the probability of making a type 1 error (concluding an effect is present, when there is none), and type 2 error (concluding no effect is present, when the exposure truly does influence outcome), and calculate the sample size required, based on this.

v/2 u (~10%) v/2 (~5%) H0: exp doesn’t cause disease HAlt: exp. associated with disease 0.75 Distribution of risk of disease if no effect of exposure (null hypothesis) Distribution of risk of disease if exposure associated with disease (alternative hypothesis) When we perform a study, our results come from a hypothesised distribution. The null in our sample, represents 0.5. Our estimate for the exposed, is most likely to land at 0.75, but as you have seen, this may be different, due to chance. We have a small chance of getting a result that suggests our exposure has no effect on the outcome. The width of the probability curves can be reduced by increasing the numbers in the study. To accomplish this we fix the shaded and solid areas, estimate the likely treatment effect (0.75) and then use a calculation to find what width of curve is produced by a sample size that we recruit. v/2 v/2 0.50 0.75 u (~10%) v/2 (~5%)

Sample size n=sample in each group π0=risk in unexposed (1/4) π1=risk in exposed (3/4) u=1 sided AUC of normal dist. Corresponding to 100%-power (eg. 10%; u=1.28) v=2 sided z level corresponding to % of AUC of normal dist. for two sided significance level required (5%, v=1.96) Ugly formula for sample size. Put in for interest, not examinable, but principles are.

Our example... n = 18.8 per group (38 total) We were unlucky not to find a significant difference between the two samples

Simpler formula... n=16 per group or 32 total Simplification, pi hat = average of null (pi 0) and alternative (pi 1). 16*0.6*0.4/(0.25)^2 = 16

Sample size vs. power The statistical power (1-probability of type 2 error) increases as sample size increases. Most investigators consider 90% power (0.9) standard for most study designs.

5 different effect measures We have 10 repetitions of the study; How many of these 95% confidence intervals would you expect to contain the true value? How many studies draw the incorrect conclusion of no difference between the treatment and no treatment groups using p-values?

Two approaches Yes/No→ ‘p-value’ What is the true difference? →’95% confidence interval’ 33

Confidence Interval What is the true difference? 95% confidence interval is not “95% probability that the true effect estimate lies within the confidence interval” “If we repeated the study over and over, and calculate 95% confidence intervals on each, 19/20 times, these intervals will contain the true effect estimate” Population Confidence interval Sample 34

95% CI “a series of values compatible with the true value”

Confidence Interval If 95% confidence interval does not contain 0 (difference between means) or 1 (ratio effect measures) the null hypothesis can be rejected. Increasing popularity over p-values...

P-value Propose ‘null’ hypothesis: no effect of eating contaminated food on risk of diarrhoea Is observed difference True vs. spurious (sampling error, chance)? Gives Yes or No answer P-value = P(observed results or more extreme | null true) If P-value low (P < 0.05): Chance is unlikely to explain result Reject null hypothesis 37

Problems with P-values NOT- probability that the null hypothesis is true Wrong conditional interpretation. Affected by both sample size and effect size Sample size is by design… Meta-analyses Is the effect clinically significant? Are the null and alternate hypotheses sufficiently different?

Problems with P-values Imagine study; All given two treatments. At follow up which drug do you prefer? Total Preference Treatment: placebo % Preference Two sided p-value 20 15:5 75% 0.04 200 115:86 57.50% 2000 1046:954 52.30% 2 000 000 1001445: 998555 50.07%

Which of the following statements about p-values are true? A) Probability of the observed results (or more extreme) if we assume there is no association B) Probability null hypothesis is true C) Unaffected by the sample size D) Used to adjudicate the presence of confounding.

Imagine…. Redesigned the dice rolling expt. The risks are the same in both cases and controls of getting disease (1 to 10 = disease; 11 to 20 = no disease). What is the chance of a type-1 (false positive) error? Is the type-1 error rate improved by increasing the sample size?

Summary Confidence interval – range of values within which we are reasonably confident that the population measure lies P-value – strength of evidence against the null hypothesis that the true difference in the population is zero.

Type-2 error is best described as: Not seeing an effect that truly is there. Detecting an effect that really isn’t there Bias Increases with increasing sample size

Type-1 error is: Detecting an effect that really isn’t there Not detecting an effect that really is there Bias Not fixed.

The probability of a type-1 error is usually... 20% 5% 1% 10%

a) It is a true effect of the exposure on the outcome. During the analysis of the results of this epidemiological study, you derive a crude odds ratio with a 95% which excludes the null value of 1. Which of the following could not explain the result you have found?   a) It is a true effect of the exposure on the outcome. b) It is a type-2 error c) It is a type-1 error d) It is due to information bias e) It is due to the presence of unadjusted confounding factors.

Which of the following measures of disease frequency is able to be estimated from a cross-sectional study? Incidence rate Cumulative incidence Prevalence Risk ratio