Using a Natural Language Processing Toolkit to Classify Patient Charts by Psychiatric Diagnosis

Background/Significance

Natural language processing (NLP) is a powerful method to extract information from unstructured electronic health record (EHR) data. In psychiatry, NLP is especially promising, as diagnostic criteria rely heavily on patient report and are often contained in the narrative text of a clinical encounter (1). However, the required informatics expertise can be a barrier to clinician-initiated machine learning (ML) and NLP research. The University of North Carolina (UNC) Translational and Clinical Sciences Institute (TraCS) developed an NLP software toolkit (Clinical Annotation Research Kit, CLARK) to overcome these barriers and support ML and NLP projects led by users with a range of informatics knowledge (2). This study analyzed the toolkit’s performance when classifying patient charts by psychiatric diagnosis.

Methods

Our team manually reviewed and labeled the EHR of 652 adult solid organ transplant recipients based on the presence or absence of a depressive and/or substance use diagnosis (SUD) prior to transplant to generate a gold standard label. Labeled records were randomly split into training and evaluation sets. We used CLARK to generate, train, and evaluate classification models for the two diagnosis groups by combining user-generated regular expressions and the gold standard labeled records.

Results

The dataset contained 312 (48%) depression cases and 135 (21%) SUD cases by manual review, which were evenly distributed between training and evaluation sets. The CLARK-generated model accurately categorized 69% of depression cases, with an area under the curve (AUC) of 0.69, and 84% of SUD cases, with an AUC of 0.73.

Discussion

This study describes the first application of the machine learning toolkit, CLARK, to classify unstructured EHR data by psychiatric diagnosis. The SUD model performance is in line with other NLP studies diagnosing SUDs (3)(4). The depression model performed less well, which may be due in part to the fact that clinical notes for the medically ill often contain references to fatigue, low appetite, weight change, and poor concentration; all of which are also DSM-5 criteria for major depressive disorder (5).

Conclusion

NLP toolkits can use unstructured EHR data to aid in the classification of patient charts by psychiatric diagnosis. This approach may be especially helpful when combined with a confidence threshold for clinician or researcher review, especially in CL psychiatry where clinical and research collaborations with other specialties can often require manipulation of large EHR datasets. This type of toolkit could improve access to ML and NLP applications for clinicians.

References

1. 1. Le Glaz A, et al. Machine Learning and Natural Language Processing in Mental Health: Systematic Review. J Med Internet Res. 2021 May 4;23(5):e15708.

2. 2. Pfaff ER, et al. Clinical Annotation Research Kit (CLARK): Computable Phenotyping Using Machine Learning. JMIR Med Inform. 2020 Jan 24;8(1):e16042.

3. 3. To D, et al. Validation of an alcohol misuse classifier in hospitalized patients. Alcohol. 2020 May;84:49–55.

4. 4. Wang Y, et al. Automated Extraction of Substance Use Information from Clinical Texts. AMIA Annu Symp Proc. 2015;2015:2121–30.

5. 5. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. American Psychiatric Association; 2013.