1. Toward Automated Pre-Biopsy Thyroid Cancer Risk Estimation in Ultrasound
Predictive Analytics for Complex Conditions
S36
Alfiia Galimzianova, Sean M. Siebert, Aya Kamaya, Terry S. Desser and Daniel L. Rubin
Stanford University

2. Disclosure
Grant funding from GE Medical Systems under Stanford- GE Blue Sky Initiative
No other relationships with commercial interests
AMIA | amia.org

3. Motivation (1): Thyroid Nodules
Thyroid nodules are very common
Palpable in 4-7% of adults
Visible in up to 67% on ultrasound exams
Commonly detected on routine ultrasound (US)
Diagnostic concern: cancer
AMIA | amia.org

4. Motivation (2): Cancerous Nodules
Thyroid cancer:
Rare; 11th most common cancer in the United States
5-10% of thyroid nodules
Common incidental finding: occult papillary CA in 36% of autopsies
5-year survival of early cancers near 100% for most common types
Aggressive cancers extremely rare
However, recognizing these on US is challenging; no reliable diagnostic features
Howlader N et al. (2016) SEER Cancer Statistics Review, , National Cancer Institute. Bethesda, MD,
AMIA | amia.org

5. Motivation (3): Thyroid Cancer Diagnosis
Incidence of thyroid cancer has tripled
Attributable to increasing use of US
But no change in mortality!
This is overdiagnosis!
Morris et al. (2016) Changing Trends in the Incidence of Thyroid Cancer in the United States. JAMA Otolaryngol Head Neck Surg.;142(7):709–711. doi: /jamaoto
AMIA | amia.org

6. Motivation (4): Thyroid CA Overdiagnosis
Overdiagnosis: Dx of disease that would have remained occult if not for detection
Leads to aggressive treatment for indolent disease
Morbidity of thyroid surgery
Medical costs of thyroid CA exceed $1.6 billion/yr in U.S. and expected to more than double by 2030

7. Thyroid nodules pose a diagnostic challenge
67% of population has US-detectable thyroid nodules
Majority of nodules are benign
Definitive diagnosis involves biopsy or surgery (invasive, costly, morbidity)
As US proliferates, more nodules are being detected with many unnecessary biopsies
Need pre-biopsy malignancy risk evaluation tools
AMIA | amia.org

8. Background: Current Procedure
Diagnostic imaging
Cancer risk estimation
Suspicion
Follow-up
palpation
incidental imaging finding
ultrasound exam
nodule localization
appearance description
measurements
benign vs malignant pattern
total score favoring benign or malignant
Pathological diagnosis
biopsy
surgery
Treatment
AMIA | amia.org

9. Background: Cancer Risk Estimation
Classification systems based on qualitative image features:
Pattern-based classifiers
Consensus-based complex patterns of benign and malignant nodules
Limitation: Covers only a subset of the variety of lesion appearances
Scoring-based classifiers
Evidence-based features and their weights resulting in total malignancy score
Limitation: Oversimplified rules for scoring and management decision, limited number of scores
Limitation common to both: Inter-reader variation
AMIA | amia.org

10. Background: Cancer Risk Estimation
Classification systems based on qualitative image features :
Pattern-based classifiers
US images
Features
Pattern matching
ATA 2015
Punctate echogenic foci
Predominantly solid
Hypoechoic
Lobulated
Solid hypoechoic nodule or solid hypoechoic component of a partially cystic nodule with one or more of the following features: irregular margins (infiltrative, microlobulated), microcalcifications, taller than wide shape, rim calcifications with small extrusive soft tissue component, evidence of extrathyroidal extention.
High suspicion
Intermediate
Low suspicion
Very low
Benign
AMIA | amia.org

11. Background: Cancer Risk Estimation
Classification systems based on qualitative image features:
Pattern-based classifiers
Scoring-based classifiers
US images
Features
Pattern matching
ATA 2015
High suspicion: Solid hypoechoic component of a partially cystic nodule with irregular margins (microlobulated) and microcalcifications
Punctate echogenic foci
Predominantly solid
Hypoechoic
Lobulated
US images
Features
Classification
ACR 2017
Malignancy score
Punctate echogenic foci
Predominantly solid
Hypoechoic
Lobulated ∑ 3 1 2 1 2 3 4 5
points
AMIA | amia.org

12. Hypothesis
Computer-derived image features on thyroid US can distinguish benign and malignant nodules
Advantages:
Automated assessment
Reproducible
Practical to integrate into workflow
AMIA | amia.org

13. Goal: Computerized Decision Support
Strategy: Machine learning classification
US images w/diagnoses
Quantitative Image Features
Classification model
Training
US images
Quantitative Image Features
Classification
Malignancy score
ElasticNet
95%
Intensity x N
Texture x M
Shape x K
Edge x L
Test
AMIA | amia.org

14. Two Classification Approaches
Types of image features:
Radiologist-observed qualitative features
Tailored learning of the complex feature combinations given representative training data
Quantitative features extracted from images
Learning both features and the rules to discriminate benign and malignant nodules
AMIA | amia.org

15. Materials: Qualitative Image Features
ACR TI-RADS* nomenclature descriptors of nodule features:
Composition
{Solid, Predominantly solid, Predominantly cystic, Cystic, Spongiform}
Echogenicity
{Hyperechoic, Isoechoic, Hypoechoic, Very hypoechoic}
Taller-than-wide shape
Margins
Border: {smooth, irregular, lobulated, ill-defined}
Halo
Extrathyroidal extension
Echogenic foci
{Punctate echogenic foci, Macrocalcifications, Peripheral calcifications, Comet-tail artifacts}
*Grant et al. (2015). Thyroid Ultrasound Reporting Lexicon: White Paper of the ACR Thyroid Imaging, Reporting and Data System (TIRADS) Committee. Journal of the American College of Radiology,12(12),
AMIA | amia.org

16. Materials: Datasets Nodules from 93 patients:
47 pathology-confirmed malignant
46 pathology-confirmed benign
Imaged in transverse and longitudinal planes on US
ACR TI-RADS qualitative descriptors
Nodule ROI delineations by expert radiologist
AMIA | amia.org

17. Materials: Expert Annotation
ePad© annotation:
Done by expert radiologist, blindly to diagnosis
Nodule delineation and semantic annotation
Longitudinal
Transverse
AMIA | amia.org

18. Methods: Expert Risk Scoring
Classification systems for cancer risk:
ACR TI-RADS, Tessler et al., JACR 2017
19 features (15 nonzero weighted)
Zayadeen et al., AJR 2016
14 features
Kwak et al., KJR 2013
7 features
Russ et al., Eur J Endocrinol. 2013
11 features
Kwak et al., Radiology 2011
8 featured
Park et al., Thyroid 2009
13 features
AMIA | amia.org

19. Methods: Quantitative Features
Intensity
{statistics (x18), histogram (x33), deciles (x9), peak, contrast, edge contrast}–62 features
Margin curvature
{5-kernel Local Area Integral Invariant statistics (x6)}–30 features
Edge sharpness
Sigmoid fit window features: {statistics (x8), histogram (x33), deciles (x9)}—50 features,
Sigmoid fit scale features: {statistics (x8), histogram (x33), deciles (x9)}—50 features
Texture
{GLCM (x16), GLRLM (x11)}–27 features
Total of 219 features
AMIA | amia.org

20. Experimental design Implemented classifiers: Details:
Proposed method: Elastic net classifier with quantitative features (”ML-Quantitative”)
Baseline method: Qualitative features with elastic net classifier (”ML- Qualitative”)
Comparision methods: Six qualitative feature-based scoring systems (TI-RADS variants)
Details:
Ground truth: Biopsy-proven malignancy/benignity
Evaluation:
Leave-one-out validation (x93 independently trained models)
AUC-ROC, potentially spared benign biopsy rate analysis
AMIA | amia.org

21. Results: Qualitative Image Features
Distribution of qualitative features across cases
Typically benign, not biopsied
AMIA | amia.org

22. Results: Qualitative Image Features
Use of qualitative features by classification systems
AMIA | amia.org

23. Results: Scoring Performance
Classifier
AUROC
CI 95%
ML-Quantitative
0.83
(0.72,0.94)
ML-Qualitative
0.78
(0.67,0.89)
Kwak et al. 2013
0.81
(0.71,0.91)
Zayadeen et al. 2016
0.77
(0.66,0.88)
Russ et al. 2013
0.75
(0.64,0.86)
ACR TI-RADS 2017
0.73
(0.63,0.84)
Park et al. 2009
(0.61,0.84)
Kwak et al. 2011
0.72
(0.62,0.82)
True positive rate
Machine learning classifiers
ML-Computational
ML-Semantic
Expert classifiers
False positive rate
AMIA | amia.org

24. Results: Scoring Performance
All malignancies biopsied,
57% benign biopsies
our motivation
Biopsied malignancy rate
True positive rate
Machine learning classifiers
ML-Computational
ML-Semantic
Expert classifiers
Unnecessary biopsy rate
False positive rate
AMIA | amia.org

25. Results: Biopsy Decision
Sparing benign findings (TN) at the cost of missing malignancies (FN)
Spared benign findings by
decision rules of classification systems
TNs at given level of FN
Classifiers
FN=0
FN=1
FN=2
ML-Quantitative 20 26 28
ML-Qualitative 1 4
ACR TI-RADS 2017 8
Zayadeen et al. 2016 2 18
Kwak et al. 2013 14
Russ et al. 2013 7 10
Kwak et al. 2011 11
Park et al. 2009
Spared benignities
Missed malignancies
Risk Classification system
TN at FNA decision
FN at FNA
decision
ACR TI-RADS 2017 15 2
Zayadeen et al. 2016 18 7
Kwak et al. 2013 26 4
Russ et al. 2013 1
Kwak et al. 2011 3
Park et al. 2009 35 20
highest number of spared benign nodules
at minimal missed malignancy rate
TN – true negatives (benign nodules labeled as benign), FN – false negatives (malignant nodules labeled as malignant)
AMIA | amia.org

26. Results: Feature Importance
Qualitative features in
classification systems
Qualitative features in
machine learning classifier
Quantitative features in
machine learning classifier
20 features:
13 malignant
7 benign
17 features:
11 malignant
6 benign
102 features
65 malignant
37 benign
AMIA | amia.org

27. Summary
Machine learning-based malignancy classifiers have a potential to be used as thyroid cancer risk estimation tools:
Comparable performance to best current classification system- based methods (AUC 83% vs 81%)
Highest number of benign nodules spared from biopsy at minimal missed malignancy rate (20 at 0 vs 18 at 1)
Provides rich set of features to describe both benign and malignant appearance
No inter/intra-rater variability at low cost
AMIA | amia.org

28. Acknowledgements This research was supported in part by grants from:
GE Medical Systems (Blue Sky Initiative at Stanford University)
National Cancer Institute, National Institutes of Health, U01CA142555, 1U01CA190214, and 1U01CA187947
AMIA | amia.org

29. Thank you!