1. How to review a diagnostic study CCBS Study Afternoon 12 August 2008 francesca.chappell@ed.ac.uk

2. What is a diagnostic study? Accuracy of an index test compared to reference standard Accuracy very often given as sensitivity and specificity Disease positive Disease negative Test positive TP = no. of true positives FP = no. of false positives Test negative FN = no. of false negatives TN = no. of true negatives Sensitivity = TP/No. of disease +ve patients Specificity = TN/No. of disease –ve patients TP + FN = no. of disease +ve patients FP + TN = no. of disease -ve patients

3. Factors affecting sensitivity and specificity patients sensitivity and specificity intrinsic performance of test Bias due to methods of conducting tests Bias due to (de)selection of patients Bias due to misanalysis of data

4. How patient selection goes wrong Ideal patients = people who would get the test in real life to establish diagnosis patients selected because they do or do not have the disease Recruitment stage patients selected for convenience Patients included in reference standard testing depending on result of index test, i.e. Verification bias Testing stage Patients included in the analysis

5. How testing goes wrong ProblemAffect on sens and spec Lack of blindingOverestimation Workup bias – patients not getting the same tests (cf verification bias) Depends Incorporation bias – reference standard uses info from index test Overestimation Inaccurate reference standardDepends Disease progression – time between tests allows disease to change Underestimation Knowing / not knowing clinical dataDepends

6. How the analysis goes wrong (1) PosNeg PosTPFP NegFNTN Index test results Reference standard results PosNeg PosTPFP ??? NegFNTN Reference standard results Index test results

7. How analysis goes wrong (2) ProblemConsequence Not correcting for verification bias overestimate sensitivity, underestimate specificity Not correcting for inaccurate reference standard Overestimation or underestimation of both sensitivity and specificity The methods for dealing with these are complicated.

8. Another problem “However, evaluations were hampered because many reports lacked information on key elements of design, conduct and analysis of diagnostic studies.” Bossuyt et al. The STARD Statement http://www.stard-statement.org/

9. Practicalities of reviewing (1) Abstract inclusion/exclusion criteria Full-text inclusion/exclusion criteria

10. Practicalities of reviewing (2) QUADAS checklist for diagnostic studiesData extraction QUADAS: See Whiting P et al. BMC Med Res Methodol 2003;3:25 http://www.biomedcentral.com/14 71-2288/3/25

11. Document everything

12. Meta-analysis Aim: to summarise the two-dimensional data

13. http://www2.napier.ac.uk/depts/fhls/diagmeta/ FREE SOFTWARE!

14. Example 1

15. Example 2

16. References Loong T-W. Understanding sensitivity and specificity with the right side of the brain. BMJ 2003;327:716-719. STARD Statement http://www.stard-statement.org/ QUADAS checklist Whiting P et al. BMC Med Res Methodol 2003;3:25 (Also available in NCC HTA monograph) The Magnificent ROC http://www.anaesthetist.com/mnm/stats/roc/Findex.htm Summary ROC curves http://srdta.cochrane.org/Files/Website/Presentations/ 4_AdvancedAnalysis_SaoPaulo.pdf DiagMeta website http://www2.napier.ac.uk/depts/fhls/diagmeta/ francesca.chappell@ed.ac.uk