Random error, Confidence intervals and p-values Simon Thornley Simon Thornley

Overview Simulated, repeated, epidemiological study to understand Random error confidence intervals p-values

What is random error? Only one type of error in epidemiological studies – Others? We rarely have whole population, so rely on sample Who is picked? Element of chance? Assumption of representative study?

10 x 20 sided dice Rolling dice = outcome e.g. Diarrhoea 10 rolls exposed; 10 rolls unexposed. Roll dice 20 times each; >5 diarrhoea in exposed; >15 diarrhoea in unexposed

Exposed (10 rolls) 1 roll = 1 participant Unexposed (10 rolls) ≥5 (diarrhoea)<5 (no diarrhoea) ≥15 (diarrhoea) <15 (no diarrhoea) abcd DiarrhoeaNo diarrhoea Exposed ab Unexposed cd a+cb+d

What is Risk/odds ratio? Assuming the die are fair, calculate true risk ratio and odds ratio of the outcome?

What is the true odds ratio? Assume die are unbiased Risk in exposed? Risk in unexposed? Odds in exposed? Odds in unexposed?

What is Risk/odds ratio? Risk in exposed is 15/20 = 3/4 Odds in exposed is 15/5 = 3 Risk in unexposed is 5/20 = ¼ Odds in unexposed is 5/15 = 1/3 Risk ratio or relative risk = ¾ / ¼ = 12/4 = 3 Odds ratio = 3/(1/3) = 9 *Note – what is theoretical upper limit of RR

Aggregate outcomes Rolling dice = outcome e.g. Diarrhoea 10 students exposed; 5 unexposed. Roll dice 40 times each; >5 diarrhoea in exposed; >15 diarrhoea in unexposed

Every pair: Two by two table DiarrhoeaNo diarrhoea Exposed ab Unexposed cd a+cb+d

In pairs Calculate risk ratio, odds ratio and 95% confidence interval for odds ratio Incidence in exposed Relative risk = Incidence in unexposed a/(a+b) = c/(c+d)

Odds ratio odds of exposure in cases odds ratio= odds of exposure in controls =a*d b*c

95% confidence interval for odds ratio se(log OR) = √[1/a+1/b+1/c+1/d] Error factor= exp[1.96 x s.e.(log OR)] 95%CI= OR / EF to OR x EF Calculate ‘z’ statistic =log (OR)/se(log OR) Use tables to convert to p-value

My example DiarrhoeaNo diarrhoea Exposed64 Unexposed OR =(6/4)/(2/8)=(6*8) /(2*4)=48/8=6 RR=(6/[6+4])/(2/[8+2])=0.6/0.2 =3 SE (log(OR)) = √[1/6+1/4+1/2+1/8] =1.02 Error factor = exp[1.96 x s.e.(1.02)]= % CI = 6/7.39 to 6*7.39 = 0.81 to 44.4 z-statistic = log(OR)/se(log(OR) =1.76; P=0.08 True odds ratio is... 9 Does calculated 95% odds ratio contain true value?

In pairs Calculate risk ratio, odds ratio and 95% confidence interval for odds ratio Incidence in exposed Relative risk = Incidence in unexposed a/(a+b) = c/(c+d)

No significant diff. What does this mean? What is the “null” hypothesis? What is the “alternate” hypothesis?

No significant diff. Null hypothesis That P(Diarrhoea|Exposed) = P(Diarrhoea|Unexposed) = 0.5 OR P(D|E)-P(D|no E)=0 Alternate hypothesis P(D|E)-P(D|no E)=delta

Errors in hypothesis testing Test result Null Hypothesis (No difference) Accept NullReject Null TrueOKType 1 error FalseType 2 errorOK

Sample size n=sample in each group π 0 =risk in unexposed (1/4) π 1 =risk in exposed (3/4) u=1 sided AUC of normal dist. Corresponding to 100%-power (eg. 10%; u=1.28) v=2 sided z level corresponding to % of AUC of normal dist for two sided significance level required (5%, v=1.96)

0.50 H 0 : π 1 - π null = u (~10%)v/2 (~5%) v/2 H Alt : π 1 - π null =0.25

Overlap

Increase sample by 2

Increase sample by 10

Our example... n=18.8 per group or 38 total We were unlucky not to find a significant difference between the two samples

Simpler formula... n=16 per group or 32 total

Sample size vs power

What about loss to follow up? In most studies, we anticipate that some people won’t contribute data due to loss to follow up. High risk groups include people with addictions, smokers, weight loss etc Adjustment factor for x% loss to follow up = 100/(100-x) Eg. x=20%; adjustment factor = 1.25

5 different effect measures We have 10 repetitions of the study; How many of these 95% confidence intervals would you expect to contain the true value? How many studies draw the incorrect conclusion of no difference between the treatment and no treatment groups using p-values?

2 different approaches Yes/No→ ‘p-value’ What is the true difference? →’95% confidence interval’

Confidence Interval What is the true difference cf. probability we are making the right decision (p-value) 95% confidence interval is not –“95% probability that the true effect estimate lies within the confidence interval” –“If we repeated the experiment over and over, then, on average, and calculate 95% confidence intervals on all of them, 19/20 will contain the true effect estimate” Population Sample Confidence interval

Confidence Interval If 95% confidence interval does not contain 0 (difference between means) or 1 (ratio effect measures) the null hypothesis can be rejected. Increasing popularity over p-values...

P-value Is observed difference –True vs. sampling error (chance)… –Can be used to give ‘Yes’ / ‘No’ answer ‘null’ hypothesis – e.g. no effect of parental smoking on smoking initiation P-value= probability of getting results by chance alone (no difference), given/if null hypothesis is true. NOT- common misinterpretation – probability that the null hypothesis is true. Does not take into account statistical power; ignores type 2 error (accepting Null when Null is false). If P-value low (e.g. p<0.05): –Chance is improbable explanation for result –Reject the null hypothesis i.e. Exposure doesn’t affect risk of diarrhoea.

Problems with P-values TotalPreference Treatment: placebo % PreferenceTwo sided p- value 2015:575% : % : % : %0.04 Imagine study; All given two treatments. At follow up which drug do you prefer?

Summary Confidence interval – range of values within which we are reasonably confident that the population measure lies P-value – strength of evidence against the null hypothesis that the true difference in the population is zero.