1. Statistics in Screening/Diagnosis
Annie Herbert
Research & Development Department
Salford Royal Hospitals NHS Foundation Trust

2. Outline Intro: Design, recording results Sensitivity Specificity
Continuous variables: ROC curves
Predictive values
Likelihood ratio
Bias

3. Introduction: diagnostic test

4. Introduction: assessing a diagnostic test
All participants
Index test
Reference test
2 x 2 Table of results + - 90 10 60
240
2 x 2 Table of results
Reference test
Index test

5. Introduction: 2x2 table of results
‘TRUTH’
(by Reference test
– Gold standard)
Total + -
TEST RESULT
(by Index test)
True
Positive
False
Total said to have disease
False Negative
True Negative
Total said not to have disease
Total with disease
Total without disease

6. Hypothetical example – Breast cancer screening study
Gold standard:
Mammography
Cheaper/more convenient option:
GP examination

7. Breast cancer screening results
‘TRUTH’
(by mammography)
Total + -
TEST RESULT (by GP Exam) 95 45
140 5
855
860
100
900
1000

8. Sensitivity - Definition
What proportion of people who have the condition are identified as positive by the test?
If a test has a high sensitivity, most people with the condition are picked up by the test.
+ve test
Have condition

9. Sensitivity - Calculation
‘TRUTH’
(by gold standard) + -
TEST RESULT a b c d
Total
a+c
b+d
Sensitivity = a/(a+c)

10. Sensitivity - Example Mammography + - GP Exam 95 45 5 855 Total 100
900
Sensitivity = 95/100 = 0.95
I.e. 95% of patients diagnosed as having breast cancer by the mammogram are picked up by GP examination.

11. Specificity - Definition
What proportion of people who don’t have the condition are identified as negative by the test?
If a test has a high specificity, most people without the condition are ruled out by the test.
-ve test
Don’t have condition

12. Specificity - Calculation
‘TRUTH’
(by gold standard) + -
TEST RESULT a b c d
Total
a+c
b+d
Specificity = d/(b+d)

13. Specificity - Example Mammography + - GP Exam 95 45 5 855 Total 100
900
Specificity = 855/900 = 0.95
I.e. 95% of patients diagnosed as not having breast cancer by the mammogram are ruled out by GP examination.

14. Sensitivity & Specificity - Notes
It is essential to have a confirmed ‘true’ diagnosis (+ve or -ve). The accuracies of sensitivity and specificity are only as good as that of the gold standard.
Sensitivity and specificity are estimated from a sample, and so should be accompanied by confidence intervals to convey amount of uncertainty.
(StatsDirect: Analysis -> Proportions -> Single)

15. Tests based on continuous variables (1)
One or more continuous variables can be a marker for a condition, where a very low/high level indicates a low/high likelihood of having the condition.
A cut-off level can be determined where having higher/lower than that cut-off indicates a positive test result.
Different cut-off points will give different sensitivity/specificity values.

16. Tests based on continuous variables (2)
E.g. Creatinekinase in patients with unstable angina or acute myocardial infarction
Data of Frances Boa, from ‘An introduction to Medical Statistics’ by Martin Bland

17. Tests based on continuous variables (3)
Truth + -
Test 27 54 39
Total 93
Cut-off level at 80
Specificity
= 39/93
= 0.42
Sensitivity
= 27/27
= 1.0

18. Tests based on continuous variables (4)
Truth + -
Test 26 35 1 58
Total 27 93
Cut-off level at 100
Specificity
= 58/93
= 0.62
Sensitivity
= 26/27
= 0.96

19. The trade-off
Plot sensitivity against (1-specificity) to get the ROC (‘receiver operating characteristic’) curve.
Ideally want high sensitivity and high specificity (but increase in one is at expense of the other).
Also requires some clinical judgement,
e.g. Likely considered better to send women without breast cancer to have a mammogram than give those with breast cancer the all clear.
Check sensitivity and specificity values in a new sample.

20. ROC curve Sensitivity = 1.0 Specificity = 1.0
The diagonal line represents
sensitivity = specificity,
i.e. taking the test is as good as flipping a coin.
ROC plot for MI data
0.00
0.25
0.50
0.75
1.00
Sensitivity
1-Specificity
Sensitivity = 0.0
Specificity = 0.0

21. Optimum cut-off MI data: ‘Optimum’ cut-off point selected = 302
Sensitivity (95% CI) = 0.93 (0.76 to 0.99)
Specificity (95% CI) = (0.91 to 0.99)
Note: ‘optimum’ assumes sensitivity and specificity
of equal concern.

22. Area under the ROC curve
Area under the ROC curve can be between 0 (sensitivity and specificity always 0.0) and 1 (sensitivity and specificity always 1.0).
Can be useful for comparing two tests.
MI data: Area under curve is an estimate of ‘probability that creatinekinase of random person with MI will be higher than for random person with angina’.

23. The difference between sensitivity & specificity and predictive values…
Sensitivity & Specificity: How good is the test at making the right diagnosis?
Predictive Values: Once diagnosis has been made, how reliable is it?

24. Positive Predictive Value - Definition
Proportion of those with a positive test result that actually have the condition.
If a test has a high positive predictive value, if someone tests positive for the condition, there is a high probability that they have it.
Have condition
Test positive

25. Positive Predictive Value - Calculation
‘TRUTH’
Total + -
TEST RESULT a b
a+b c d
c+d
PPV
= a/(a+b)

26. Positive Predictive Value - Example
‘TRUTH’
(by mammogram)
Total + -
TEST RESULT (by GP Exam) 95 45
140 5
855
860
PPV
= 95/140
= 0.68
I.e. 68% of patients who test positive for breast cancer by GP examination could be expected to test positive by mammogram.

27. Negative Predictive Value - Definition
Proportion of those with a negative test result that really don’t have the condition.
If a test has a high negative predictive value, if someone tests negative for the condition, there is a high probability that they don’t have it.
Don’t have condition
Test negative

28. Negative Predictive Value - Calculation
‘TRUTH’
Total + -
TEST RESULT a b
a+b c d
c+d
NPV
= d/(c+d)

29. Negative Predictive Value - Example
‘TRUTH’
(by mammogram)
Total + -
TEST RESULT (by GP Exam) 95 45
140 5
855
860
NPV
= 855/860
= 0.99
I.e. 99% of patients who test negative for breast cancer by GP examination would be expected to test negative by mammogram.

30. Prevalence
What proportion of people in a cohort have the disease? E.g. “The prevalence of breast cancer in females over 40 years of age is approximately 1.5%”.
‘Prevalence’ is not the same as ‘incidence’.
Sensitivity & specificity values are unaffected by prevalence, though predictive values are.

31. E.g. Test with 95% sensitivity and 95% specificity:
MeReC Briefing: supplement to issue 30

32. Example: Self administered cognitive screening test (TYM) for detection of Alzheimer’s disease: cross sectional study, Brown et al, June 2009
“A score of 42/50 had a sensitivity of 93% and specificity of 86% in the diagnosis of Alzheimer’s disease. The TYM was more sensitive in detection of Alzheimer’s disease than the mini-mental examination, detecting 93% of patients compared with 52% for the mini-mental state examination. The negative and positive predictive values of the TYM with the cut off of 42 were 99% and 42% with a prevalence of Alzheimer’s disease of 10%.”

33. Likelihood Ratio - Definition
How many times more (or less) a patient with the condition is likely to have that particular result than a patient without the disease.
Can be used to calculate the probability of individual patient having condition based on test results.

34. Use of Fagan's nomogram for calculating post-test probabilities:
Deeks, J. J et al. BMJ 2004;329:
Copyright ©2004 BMJ Publishing Group Ltd.

35. Bias in studies: Is the reference appropriate?
Was the same reference used for all patients (verification bias)?
Were assessors blind to case details?
Was it a ‘diagnostic case-control study’?
Which population was the test tested in?

36. Summary
All patients must have both new test & reference test (gold standard).
Report 2x2 table and give sensitivity, specificity with precision.
A good screening test is not necessarily a good diagnostic test.
Test cut-offs in an independent sample.
Predictive values vary according to prevalence.
Consider all potential sources of bias.

37. Recommended Texts BMJ Statistics Notes: 1) Sensitivity & Specificity
2) Predictive Values
3) ROC Curves
4) Likelihood Ratios
Assessing bias
How to read a paper: Papers that report diagnostic or screening tests by Trisha Greenhalgh; BMJ pg. 540.