Dataset Information

Comparison of Natural Language Processing Rules-based and Machine-learning Systems to Identify Lumbar Spine Imaging Findings Related to Low Back Pain.

ABSTRACT: RATIONALE AND OBJECTIVES:To evaluate a natural language processing (NLP) system built with open-source tools for identification of lumbar spine imaging findings related to low back pain on magnetic resonance and x-ray radiology reports from four health systems. MATERIALS AND METHODS:We used a limited data set (de-identified except for dates) sampled from lumbar spine imaging reports of a prospectively assembled cohort of adults. From N?=?178,333 reports, we randomly selected N?=?871 to form a reference-standard dataset, consisting of N?=?413 x-ray reports and N?=?458 MR reports. Using standardized criteria, four spine experts annotated the presence of 26 findings, where 71 reports were annotated by all four experts and 800 were each annotated by two experts. We calculated inter-rater agreement and finding prevalence from annotated data. We randomly split the annotated data into development (80%) and testing (20%) sets. We developed an NLP system from both rule-based and machine-learned models. We validated the system using accuracy metrics such as sensitivity, specificity, and area under the receiver operating characteristic curve (AUC). RESULTS:The multirater annotated dataset achieved inter-rater agreement of Cohen's kappa?>?0.60 (substantial agreement) for 25 of 26 findings, with finding prevalence ranging from 3% to 89%. In the testing sample, rule-based and machine-learned predictions both had comparable average specificity (0.97 and 0.95, respectively). The machine-learned approach had a higher average sensitivity (0.94, compared to 0.83 for rules-based), and a higher overall AUC (0.98, compared to 0.90 for rules-based). CONCLUSIONS:Our NLP system performed well in identifying the 26 lumbar spine findings, as benchmarked by reference-standard annotation by medical experts. Machine-learned models provided substantial gains in model sensitivity with slight loss of specificity, and overall higher AUC.

SUBMITTER: Tan WK

PROVIDER: S-EPMC6162177 | biostudies-literature | 2018 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Comparison of Natural Language Processing Rules-based and Machine-learning Systems to Identify Lumbar Spine Imaging Findings Related to Low Back Pain.

Tan W Katherine WK Hassanpour Saeed S Heagerty Patrick J PJ Rundell Sean D SD Suri Pradeep P Huhdanpaa Hannu T HT James Kathryn K Carrell David S DS Langlotz Curtis P CP Organ Nancy L NL Meier Eric N EN Sherman Karen J KJ Kallmes David F DF Luetmer Patrick H PH Griffith Brent B Nerenz David R DR Jarvik Jeffrey G JG

Academic radiology 20180328 11

<h4>Rationale and objectives</h4>To evaluate a natural language processing (NLP) system built with open-source tools for identification of lumbar spine imaging findings related to low back pain on magnetic resonance and x-ray radiology reports from four health systems.<h4>Materials and methods</h4>We used a limited data set (de-identified except for dates) sampled from lumbar spine imaging reports of a prospectively assembled cohort of adults. From N = 178,333 reports, we randomly selected N = 8 ...[more]

PMID: 29605561

Similar Datasets

Project description:PurposeTo characterize the inter- and intraobserver variability of qualitative, non-disk contour degenerative findings of the lumbar spine at magnetic resonance (MR) imaging.Materials and methodsThe case accrual method used to perform this institutional review board-approved, HIPAA-compliant retrospective study was the random selection of 111 interpretable MR examination cases of subjects from the Spine Patient Outcomes Research Trial. The subjects were aged 18-87 years (mean, 53 years +/- 16 [standard deviation]). Four independent readers rated the cases according to defined criteria. A subsample of 40 MR examination cases was selected for reevaluation at least 1 month later. The following findings were assessed: spondylolisthesis, disk degeneration, marrow endplate abnormality (Modic changes), posterior anular hyperintense zone (HIZ), and facet arthropathy. Inter- and intraobserver agreement in rating the data was summarized by using weighted kappa statistics.ResultsInterobserver agreement was good (kappa = 0.66) in rating disk degeneration and moderate in rating spondylolisthesis (kappa = 0.55), Modic changes (kappa = 0.59), facet arthropathy (kappa = 0.54), and posterior HIZ (kappa = 0.44). Interobserver agreement in rating the extent of Modic changes was moderate: kappa Values were 0.43 for determining superior anteroposterior extent, 0.47 for determining superior craniocaudal extent, 0.57 for determining inferior anteroposterior extent, and 0.48 for determining inferior craniocaudal extent. Intraobserver agreement was good in rating spondylolisthesis (kappa = 0.66), disk degeneration (kappa = 0.74), Modic changes (kappa = 0.64), facet arthropathy (kappa = 0.69), and posterior HIZ (kappa = 0.67). Intraobserver agreement in rating the extent of Modic changes was moderate, with kappa values of 0.54 for superior anteroposterior, 0.60 for inferior anteroposterior, 0.50 for superior craniocaudal, and 0.60 for inferior craniocaudal extent determinations.ConclusionThe interpretation of general lumbar spine MR characteristics has sufficient reliability to warrant the further evaluation of these features as potential prognostic indicators.

Project description:Accurate, automated extraction of clinical stroke information from unstructured text has several important applications. ICD-9/10 codes can misclassify ischemic stroke events and do not distinguish acuity or location. Expeditious, accurate data extraction could provide considerable improvement in identifying stroke in large datasets, triaging critical clinical reports, and quality improvement efforts. In this study, we developed and report a comprehensive framework studying the performance of simple and complex stroke-specific Natural Language Processing (NLP) and Machine Learning (ML) methods to determine presence, location, and acuity of ischemic stroke from radiographic text. We collected 60,564 Computed Tomography and Magnetic Resonance Imaging Radiology reports from 17,864 patients from two large academic medical centers. We used standard techniques to featurize unstructured text and developed neurovascular specific word GloVe embeddings. We trained various binary classification algorithms to identify stroke presence, location, and acuity using 75% of 1,359 expert-labeled reports. We validated our methods internally on the remaining 25% of reports and externally on 500 radiology reports from an entirely separate academic institution. In our internal population, GloVe word embeddings paired with deep learning (Recurrent Neural Networks) had the best discrimination of all methods for our three tasks (AUCs of 0.96, 0.98, 0.93 respectively). Simpler NLP approaches (Bag of Words) performed best with interpretable algorithms (Logistic Regression) for identifying ischemic stroke (AUC of 0.95), MCA location (AUC 0.96), and acuity (AUC of 0.90). Similarly, GloVe and Recurrent Neural Networks (AUC 0.92, 0.89, 0.93) generalized better in our external test set than BOW and Logistic Regression for stroke presence, location and acuity, respectively (AUC 0.89, 0.86, 0.80). Our study demonstrates a comprehensive assessment of NLP techniques for unstructured radiographic text. Our findings are suggestive that NLP/ML methods can be used to discriminate stroke features from large data cohorts for both clinical and research-related investigations.

Dataset Information

Comparison of Natural Language Processing Rules-based and Machine-learning Systems to Identify Lumbar Spine Imaging Findings Related to Low Back Pain.

Publications

Comparison of Natural Language Processing Rules-based and Machine-learning Systems to Identify Lumbar Spine Imaging Findings Related to Low Back Pain.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets