1. Phenotyping youth depression
using free text from electronic medical records
Geraci J1,2,3, Wilansky P1, de Luca V1, Roy A1, Kennedy JL1, Strauss J1,3
1 Centre for Addiction and Mental Health, Toronto, Canada
2Queen’s University, Dept. of Pathology and Molecular Medicine, Kingston, Canada
3Shannon Centennial Informatics Lab, Centre for Addiction and Mental Health, Toronto, Canada
ABSTRACT
RESULTS
2015 – 2017 Project Timeline
This investigation of machine learning applied to psychiatric diagnosis phenotyping for research recruitment, using 861 labeled electronic health record (EMR) documents. We were specifically interested in building a model that could accurately identify individuals who were appropriate candidates for clinical study of adolescent depression. The goal was a model to identify individuals who meet inclusion criteria as well as exclude unsuitable patients who met our exclusion criteria. Methods involved deidentifying the EMR documents, having two psychiatrists to label a set of EMR documents (from which the 861 came), applying a brute force search, and training a deep neural network. Utilizing a cross validation evaluation, we discovered a model that had a specificity of 97% and a sensitivity of 45% and another model that had a specificity of 53% and a sensitivity of 89%. We combined these two models into one for the purpose of supplying a list of most suitable candidates to support recruitment efforts (sensitivity 93.5% and specificity 68%).
Table 1 The performance of DL0 with five fold cross-validation. It performs very well in
rejecting unsuitable patients accurately, but it does not perform well with predicting
suitable participants (the True 1's).
Sensitivity % ; Specificity 97 %
Predicted 0’s
Predicted 1’s
True 0’s 
639 18
True 1’s  56 45
BACKGROUND
- Traditional methods fail to identify up to 60% of possible research participants.
- Structured diagnostic codes are sometimes available, but often are missing.
- Natural language processing (NLP) and machine learning (ML) methods have been used to extract clinical information from EMRs’ unstructured text.
NLP has been used extensively to improve phenotyping by EMR
colorectal cancer screening RA
ADE’s
Depression dx
NLP improves detection by 1/3
NLP improves ROC curve
NLP + ML yielded F1=89.6% in one study
Becauase discrete diagnostic codes had poor completion rates in our EMR, we used NLP and ML on unstructured EMR text notes to impose our diagnostic inclusion criteria, specifically DSM-IV TR depression diagnoses, to enable recruitment for a genomic investigation of depressed teens. This poster summarizes the NLP and ML procedures and results. The core purpose of this REB approved investigation is to present a model that identifies youth with a depression diagnosis and lacking multiple exclusion comorbidities -- a model examined via cross-validation and an independent test data set, from the vantage of deep neural networks.
Table 2 The performance of DL1 with a five fold cross-validation. In contrast to model DL0, this model
is quite good at accurately predicting participants (True 1's) but is poor at rejecting inappropriate patients.
Sensitivity 89 % ; Specificity 53 %
Issues,
20%
Requests,
80%
Predicted 0’s
Predicted 1’s
True 0’s  47 53
True 1’s  11 90
2015
2015
DISCUSSION
We assembled recommendation algorithms - first training two deep neural networks, one that accurately identifies patients who are should be excluded, and another that accurately recognizes those who meet inclusion criteria. The two deep neural network models were combined into a single procedure. This was validated on an independent test set after tuning each component with a 5-fold  cross validation protocol.  
Limitations
We were not able to use discrete diagnostic codes, however, published studies support the idea that NLP /ML methods improve the performance of information extraction.
The brute force method was not practical owing to inconsistent performance. The suspected reason for the fluctuations in output we suspect is that the content of the EMR documents could vary substantially. Some documents have an obvious diagnosis, while others have sufficiently useful narrative, and still others would be too ambiguous for the brute force approach to capture reliable information.
Our modest corpus size is the greatest contributor to the low sensitivity of our results.
METHODS
Data extraction
De-identification
File preparation
Annotation by two psychiatrists (JS, AR)
TF-IDF = document term matrix
Brute force analysis
Supervised learning by deep neural network
+ five-fold cross validation
DL1
DL0
Future Directions
Larger corpus
More powerful version of neural networks – recursive deep networks.
Other ML techniques – e.g. gradient boosting
Supported by
McLaughlin Accerlator Grant in Genomic Medicine (PW, JS)
Ontario Shores Foundation for Mental Health; CAMH - Campbell Family Mental Health Research Institute
CAMH – Information Management Group
CONTACT: