1. Classification of Emergency Department CT Imaging Reports using Natural Language Processing and Machine Learning Efsun Sarioglu, Kabir Yadav, Meaghan Smith, Hyeong-Ah Choi This project supported by the NIH National Center for Research Resources (UL1RR031988 and KL2RR031987)

2. Background, Objective & Methods  Use of electronic medical record data for clinical research and quality improvement requires free-text data interpretation for outcomes of interest.  Natural language processing has shown promise for this purpose  To demonstrate real-world performance of a hybrid NLP-machine learning system for automated classification of radiology reports

3. Approach Overview  Multicenter review of consecutive CT reports obtained for facial trauma using a trained reference standard  Medical Language Extraction and Encoding (MedLEE)  WEKA 3.7.5  Salford Systems CART 6.6

4. Results  Total reports: 3710  Positive cases: 460 (12.4%)  Manual coding had high reliability  Kappa=0.97 [95% CI 0.94-0.99]

5. CART Decision Trees (50:50) Raw Text (8-node) NLP (9-node)

6. Classification Performance Raw Text NLP Precisio n 0.9490.968 Recall 0.9320.964 F-score 0.9400.966  Unexpectedly high performance of machine learning without NLP  Comparable to inter- rater performance and prior studies of inter- physician agreement  Comparable to prior real-world and simulation studies

7. Concluding Remarks  How’s it novel?  One of only a handful of real-world NLP studies using validated reference standard  Translating existing NLP and machine learning technologies to support CER  Next step: validation  Test approach using other clinical cases  Evaluate different features or classification algorithms