Confidence Intervals and Hypothesis Testing Mark Dancox Public Health Intelligence Course – Day 3

In this session… Confidence Intervals - How to calculate - How to interpret Hypothesis testing Sample size and small numbers

Measuring uncertainty Most measures used to assess health are subject to chance variation. ᶿ ᶿ Source: APHO Health Profiles 2009, using ONS teenage conception data How likely is it that these differences are due to chance? East

Confidence Intervals Summary statistics are point estimates based on samples Confidence Intervals quantify the degree of uncertainty in these estimates Quoted as a lower limit and an upper limit which provide a range of values within which the population value is likely to lie

Source: APHO Health Profiles, ONS data, 95% confidence intervals ᶿ ᶿ East

Calculating Confidence Intervals General form of any 95% C.I.: Point Estimate ± 1.96*(Estimated SE) For 99% CI’s we use 2.57 For 90% CI’s we use 1.64

Standard Error Summary statistics – such as the mean- are based on samples Different samples from the same population give rise to different values This variation is quantified by the standard error

An example….

An example….continued….

Standard Errors Normal distribution Poisson distribution Binomial Distribution

Example If the mean weight (kg) for a given sample of 43 men aged 55 is 81.4kg and the standard deviation is 12.7 kg…. Then a 95% confidence interval is… [77.6, 85.2]

Interpretation A 95% Confidence Interval is a random interval, such that in related sampling 95 out of every 100 intervals succeed in covering the parameter Loose interpretation – “95% chance true value inside interval” 5% of cases (X X) True Value X)

Interpretation of confidence intervals Non overlapping intervals indicative of real differences Overlapping intervals need to be considered with caution Need to be careful about using confidence intervals as a means of testing. The smaller the sample size, the wider the confidence interval

Which areas are significantly higher than England?

The width of a confidence interval is affected by: Size of the sample Variability in phenomenon being measured Required level of confidence, e.g. 95%, 99%

In this session… Confidence Intervals - How to calculate - How to interpret Hypothesis testing Sample size and small numbers

Hypothesis testing & Small Numbers Outline of hypothesis testing P-values Sample sizes

Hypothesis Testing Inferences about a population are often based upon a sample. Want to be able to use sample statistics to draw conclusions about the underlying population values Hypothesis testing provides some criteria for reaching these conclusions

Types of hypotheses Null Hypothesis (H 0 ) – The hypothesis under consideration – “TB incidence in Area 1 is no different from England” Alternative Hypothesis (H a ) – The hypothesis we accept if we reject the null hypothesis – “TB incidence in Area 1 is different from England”

General principles Formulate null (H 0 ) and alternative (H a ) hypotheses Choose test statistic – preferably one whose distribution is known Decide rule for choosing between the null and alternative hypotheses – This involves assuming an underlying distribution Calculate test statistic and compare against the decision rule. Check assumptions

Illustration of acceptance region

P-values Criteria to judge statistical significance of results. Quoted as values between 0 and 1 Probability of result, assuming H o true Values less than 0.05 (or 0.01) indicates an observation unlikely under the assumption that H O is true E.g. A p-value of 0.05 could be interpreted as 5% probability this observation was just due to chance.

Illustration of P-value under H o

Significance Levels Used as the criteria to accept or reject Null Reject the null hypothesis if P-value < 0.05 (or 0.01) These values should be chosen a priori

Example: TB Incidence Formulate null (H 0 ) and alternative (H a ) hypotheses Choose test statistic – preferably one whose distribution is known Decide rule for choosing between the null and alternative hypotheses – This involves assuming an underlying distribution Calculate test statistic and compare against the decision rule. Check assumptions

Example TB Incidence Null Hypothesis (H 0 ) – The hypothesis under consideration – “TB incidence in Area 1 is no different from England” Alternative Hypothesis (H a ) – The hypothesis we accept if we reject the null hypothesis – “TB incidence in Area 1 is different from England”

Example: TB Incidence Area 1 TB INCIDENCE Null: “TB Incidence in Area 1 no different from England”

In this session… Confidence Intervals - How to calculate - How to interpret Hypothesis testing Sample size and small numbers

Sample size Results may indicate no difference between groups This may be because there is truly no difference between groups or because there was an insufficiently large sample size for this to be detected

Determining Sample Size Choice of sample size depends on: – Anticipated size of effect/ required precision – Variability in measurement – Power (the probability of detecting a difference if it exists) – Significance levels

‘Power’ is related to Type 1 and Type 2 error

Small samples The smaller the sample, the higher degree of uncertainty in results. Increased variability in small samples Confidence Intervals for estimates are wider Low numbers may affect the calculation of directly standardised rates (for instance) Distribution assumptions may be affected.

Dealing with small numbers Can combine several years of data – Mortality pooled over several years for rare conditions Combine counts across categories of data – Low cell counts in cross-classifications of the data Exact methods may be needed. Disclosure control needed?

Finding out more The Association of Public Health Observatories has produced a useful briefing on confidence intervals

Finding out more Lots of useful information can be found at the HealthKnowledge website…

Finding out more Some further references of interest: – Bland, M. Introduction to Medical Statistics. Third Edition. Oxford University Press, – Hennekens CH, Buring JE. Epidemiology in Medicine, Lippincott Williams & Wilkins, – Larson, H.J. Introduction to Probability Theory and Statistical Inference. Third Edition. Wiley, 1982

Power…if time…