Agreement between neuroimages and reports for natural language processing-based detection of silent brain infarcts and white matter disease.
Ontology highlight
ABSTRACT: Background
There are numerous barriers to identifying patients with silent brain infarcts (SBIs) and white matter disease (WMD) in routine clinical care. A natural language processing (NLP) algorithm may identify patients from neuroimaging reports, but it is unclear if these reports contain reliable information on these findings.Methods
Four radiology residents reviewed 1000 neuroimaging reports (RI) of patients age > 50 years without clinical histories of stroke, TIA, or dementia for the presence, acuity, and location of SBIs, and the presence and severity of WMD. Four neuroradiologists directly reviewed a subsample of 182 images (DR). An NLP algorithm was developed to identify findings in reports. We assessed interrater reliability for DR and RI, and agreement between these two and with NLP.Results
For DR, interrater reliability was moderate for the presence of SBIs (k = 0.58, 95 % CI 0.46-0.69) and WMD (k = 0.49, 95 % CI 0.35-0.63), and moderate to substantial for characteristics of SBI and WMD. Agreement between DR and RI was substantial for the presence of SBIs and WMD, and fair to substantial for characteristics of SBIs and WMD. Agreement between NLP and DR was substantial for the presence of SBIs (k = 0.64, 95 % CI 0.53-0.76) and moderate (k = 0.52, 95 % CI 0.39-0.65) for the presence of WMD.Conclusions
Neuroimaging reports in routine care capture the presence of SBIs and WMD. An NLP can identify these findings (comparable to direct imaging review) and can likely be used for cohort identification.
SUBMITTER: Leung LY
PROVIDER: S-EPMC8111708 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BackgroundSilent brain infarction (SBI) is defined as the presence of 1 or more brain lesions, presumed to be because of vascular occlusion, found by neuroimaging (magnetic resonance imaging or computed tomography) in patients without clinical manifestations of stroke. It is more common than stroke and can be detected in 20% of healthy elderly people. Early detection of SBI may mitigate the risk of stroke by offering preventative treatment plans. Natural language processing (NLP) techniques offer an opportunity to systematically identify SBI cases from electronic health records (EHRs) by extracting, normalizing, and classifying SBI-related incidental findings interpreted by radiologists from neuroimaging reports.ObjectiveThis study aimed to develop NLP systems to determine individuals with incidentally discovered SBIs from neuroimaging reports at 2 sites: Mayo Clinic and Tufts Medical Center.MethodsBoth rule-based and machine learning approaches were adopted in developing the NLP system. The rule-based system was implemented using the open source NLP pipeline MedTagger, developed by Mayo Clinic. Features for rule-based systems, including significant words and patterns related to SBI, were generated using pointwise mutual information. The machine learning models adopted convolutional neural network (CNN), random forest, support vector machine, and logistic regression. The performance of the NLP algorithm was compared with a manually created gold standard. The gold standard dataset includes 1000 radiology reports randomly retrieved from the 2 study sites (Mayo and Tufts) corresponding to patients with no prior or current diagnosis of stroke or dementia. 400 out of the 1000 reports were randomly sampled and double read to determine interannotator agreements. The gold standard dataset was equally split to 3 subsets for training, developing, and testing.ResultsAmong the 400 reports selected to determine interannotator agreement, 5 reports were removed due to invalid scan types. The interannotator agreements across Mayo and Tufts neuroimaging reports were 0.87 and 0.91, respectively. The rule-based system yielded the best performance of predicting SBI with an accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of 0.991, 0.925, 1.000, 1.000, and 0.990, respectively. The CNN achieved the best score on predicting white matter disease (WMD) with an accuracy, sensitivity, specificity, PPV, and NPV of 0.994, 0.994, 0.994, 0.994, and 0.994, respectively.ConclusionsWe adopted a standardized data abstraction and modeling process to developed NLP techniques (rule-based and machine learning) to detect incidental SBIs and WMDs from annotated neuroimaging reports. Validation statistics suggested a high feasibility of detecting SBIs and WMDs from EHRs using NLP.
| S-EPMC6524454 | biostudies-literature
Project description: Not available
| S-EPMC4909551 | biostudies-literature
Project description:This study aimed to develop Natural Language Processing (NLP) approaches to supplement manual outcome validation, specifically to validate pneumonia cases from chest radiograph reports.We trained one NLP system, ONYX, using radiograph reports from children and adults that were previously manually reviewed. We then assessed its validity on a test set of 5000 reports. We aimed to substantially decrease manual review, not replace it entirely, and so, we classified reports as follows: (1) consistent with pneumonia; (2) inconsistent with pneumonia; or (3) requiring manual review because of complex features. We developed processes tailored either to optimize accuracy or to minimize manual review. Using logistic regression, we jointly modeled sensitivity and specificity of ONYX in relation to patient age, comorbidity, and care setting. We estimated positive and negative predictive value (PPV and NPV) assuming pneumonia prevalence in the source data.Tailored for accuracy, ONYX identified 25% of reports as requiring manual review (34% of true pneumonias and 18% of non-pneumonias). For the remainder, ONYX's sensitivity was 92% (95% CI 90-93%), specificity 87% (86-88%), PPV 74% (72-76%), and NPV 96% (96-97%). Tailored to minimize manual review, ONYX classified 12% as needing manual review. For the remainder, ONYX had sensitivity 75% (72-77%), specificity 95% (94-96%), PPV 86% (83-88%), and NPV 91% (90-91%).For pneumonia validation, ONYX can replace almost 90% of manual review while maintaining low to moderate misclassification rates. It can be tailored for different outcomes and study needs and thus warrants exploration in other settings.
| S-EPMC3811072 | biostudies-literature
Project description:A small number of population-based studies reported an association between migraine with aura and risk of silent brain infarcts and white matter hyperintensities in females. We investigated these relations in a population-based sample of female twins. We contacted female twins ages 30-60 years identified through the population-based Danish Twin Registry. Based on questionnaire responses, twins were invited to participate in a telephone-based interview conducted by physicians. Headache diagnoses were established according to the International Headache Society criteria. Cases with migraine with aura, their co-twins, and unrelated migraine-free twins (controls) were invited to a brain magnetic resonance imaging scan performed at a single centre. Brain scans were assessed for the presence of infarcts, and white matter hyperintensities (visual rating scales and volumetric analyses) blinded to headache diagnoses. Comparisons were based on 172 cases, 34 co-twins, and 139 control subjects. Compared with control subjects, cases did not differ with regard to frequency of silent brain infarcts (four cases versus one control), periventricular white matter hyperintensity scores [adjusted mean difference (95% confidence interval): -0.1 (-0.5 to 0.2)] or deep white matter hyperintensity scores [adjusted mean difference (95% confidence interval): 0.1 (-0.8 to 1.1)] assessed by Scheltens' scale. Cases had a slightly higher total white matter hyperintensity volume compared with controls [adjusted mean difference (95% confidence interval): 0.17 (-0.08 to 0.41) cm(3)] and a similar difference was present in analyses restricted to twin pairs discordant for migraine with aura [adjusted mean difference 0.21 (-0.20 to 0.63)], but these differences did not reach statistical significance. We found no evidence of an association between silent brain infarcts, white matter hyperintensities, and migraine with aura.
| S-EPMC4939694 | biostudies-literature
Project description:AimsStress echocardiography (SE) findings and interpretations are commonly documented in free-text reports. Reusing SE results requires laborious manual reviews. This study aimed to develop and validate an automated method for abstracting SE reports in a large cohort.Methods and resultsThis study included adult patients who had SE within 30 days of their emergency department visit for suspected acute coronary syndrome in a large integrated healthcare system. An automated natural language processing (NLP) algorithm was developed to abstract SE reports and classify overall SE results into normal, non-diagnostic, infarction, and ischaemia categories. Randomly selected reports (n = 140) were double-blindly reviewed by cardiologists to perform criterion validity of the NLP algorithm. Construct validity was tested on the entire cohort using abstracted SE data and additional clinical variables. The NLP algorithm abstracted 6346 consecutive SE reports. Cardiologists had good agreements on the overall SE results on the 140 reports: Kappa (0.83) and intraclass correlation coefficient (0.89). The NLP algorithm achieved 98.6% specificity and negative predictive value, 95.7% sensitivity, positive predictive value, and F-score on the overall SE results and near-perfect scores on ischaemia findings. The 30-day acute myocardial infarction or death outcomes were highest among patients with ischaemia (5.0%), followed by infarction (1.4%), non-diagnostic (0.8%), and normal (0.3%) results. We found substantial variations in the format and quality of SE reports, even within the same institution.ConclusionsNatural language processing is an accurate and efficient method for abstracting unstructured SE reports. This approach creates new opportunities for research, public health measures, and care improvement.
| S-EPMC9779789 | biostudies-literature
Project description:Silent lacunar infarcts, which are present in over 20% of healthy elderly individuals, are associated with subtle deficits in cognitive functions. However, it remains largely unclear how these silent brain infarcts lead to cognitive deficits and even dementia. Here, we used diffusion tensor imaging tractography and graph theory to examine the topological organization of white matter networks in 27 patients with silent lacunar infarcts in the basal ganglia territory and 30 healthy controls. A whole-brain white matter network was constructed for each subject, where the graph nodes represented brain regions and the edges represented interregional white matter tracts. Compared with the controls, the patients exhibited a significant reduction in local efficiency and global efficiency. In addition, a total of eighteen brain regions showed significantly reduced nodal efficiency in patients. Intriguingly, nodal efficiency-behavior associations were significantly different between the two groups. The present findings provide new aspects into our understanding of silent infarcts that even small lesions in subcortical brain regions may affect large-scale cortical white matter network, as such may be the link between subcortical silent infarcts and the associated cognitive impairments. Our findings highlight the need for network-level neuroimaging assessment and more medical care for individuals with silent subcortical infarcts.
| S-EPMC4640338 | biostudies-literature
Project description:BackgroundNatural language processing (NLP) has a significant role in advancing healthcare and has been found to be key in extracting structured information from radiology reports. Understanding recent developments in NLP application to radiology is of significance but recent reviews on this are limited. This study systematically assesses and quantifies recent literature in NLP applied to radiology reports.MethodsWe conduct an automated literature search yielding 4836 results using automated filtering, metadata enriching steps and citation search combined with manual review. Our analysis is based on 21 variables including radiology characteristics, NLP methodology, performance, study, and clinical application characteristics.ResultsWe present a comprehensive analysis of the 164 publications retrieved with publications in 2019 almost triple those in 2015. Each publication is categorised into one of 6 clinical application categories. Deep learning use increases in the period but conventional machine learning approaches are still prevalent. Deep learning remains challenged when data is scarce and there is little evidence of adoption into clinical practice. Despite 17% of studies reporting greater than 0.85 F1 scores, it is hard to comparatively evaluate these approaches given that most of them use different datasets. Only 14 studies made their data and 15 their code available with 10 externally validating results.ConclusionsAutomated understanding of clinical narratives of the radiology reports has the potential to enhance the healthcare process and we show that research in this field continues to grow. Reproducibility and explainability of models are important if the domain is to move applications into clinical use. More could be done to share code enabling validation of methods on different institutional data and to reduce heterogeneity in reporting of study properties allowing inter-study comparisons. Our results have significance for researchers in the field providing a systematic synthesis of existing work to build on, identify gaps, opportunities for collaboration and avoid duplication.
| S-EPMC8176715 | biostudies-literature
Project description:BackgroundFindings and interpretations of myocardial perfusion imaging (MPI) studies are documented in free-text MPI reports. MPI results are essential for research, but manual review is prohibitively time consuming. This study aimed to develop and validate an automated method to abstract MPI reports.MethodsWe developed a natural language processing (NLP) algorithm to abstract MPI reports. Randomly selected reports were double-blindly reviewed by two cardiologists to validate the NLP algorithm. Secondary analyses were performed to describe patient outcomes based on abstracted-MPI results on 16,957 MPI tests from adult patients evaluated for suspected ACS.ResultsThe NLP algorithm achieved high sensitivity (96.7%) and specificity (98.9%) on the MPI categorical results and had a similar degree of agreement compared to the physician reviewers. Patients with abnormal MPI results had higher rates of 30-day acute myocardial infarction or death compared to patients with normal results. We identified issues related to the quality of the reports that not only affect communication with referring physicians but also challenges for automated abstraction.ConclusionNLP is an accurate and efficient strategy to abstract results from the free-text MPI reports. Our findings will facilitate future research to understand the benefits of MPI studies but requires validation in other settings.
| S-EPMC8096860 | biostudies-literature
Project description:Background and purposeAccurate prehospital diagnosis of stroke by emergency medical services (EMS) can increase treatments rates, mitigate disability, and reduce stroke deaths. We aimed to develop a model that utilizes natural language processing of EMS reports and machine learning to improve prehospital stroke identification.MethodsWe conducted a retrospective study of patients transported by the Chicago EMS to 17 regional primary and comprehensive stroke centers. Patients who were suspected of stroke by the EMS or had hospital-diagnosed stroke were included in our cohort. Text within EMS reports were converted to unigram features, which were given as input to a support-vector machine classifier that was trained on 70% of the cohort and tested on the remaining 30%. Outcomes included final diagnosis of stroke versus nonstroke, large vessel occlusion, severe stroke (National Institutes of Health Stroke Scale score >5), and comprehensive stroke center-eligible stroke (large vessel occlusion or hemorrhagic stroke).ResultsOf 965 patients, 580 (60%) had confirmed acute stroke. In a test set of 289 patients, the text-based model predicted stroke nominally better than models based on the Cincinnati Prehospital Stroke Scale (c-statistic: 0.73 versus 0.67, P=0.165) and was superior to the 3-Item Stroke Scale (c-statistic: 0.73 versus 0.53, P<0.001) scores. Improvements in discrimination were also observed for the other outcomes.ConclusionsWe derived a model that utilizes clinical text from paramedic reports to identify stroke. Our results require validation but have the potential of improving prehospital routing protocols.
| S-EPMC8514137 | biostudies-literature
Project description:PurposeTo develop natural language processing (NLP) to identify incidental lung nodules (ILNs) in radiology reports for assessment of management recommendations.Methods and materialsWe searched the electronic health records for patients who underwent chest CT during 2014 and 2017, before and after implementation of a department-wide dictation macro of the Fleischner Society recommendations. We randomly selected 950 unstructured chest CT reports and reviewed manually for ILNs. An NLP tool was trained and validated against the manually reviewed set, for the task of automated detection of ILNs with exclusion of previously known or definitively benign nodules. For ILNs found in the training and validation sets, we assessed whether reported management recommendations agreed with Fleischner Society guidelines. The guideline concordance of management recommendations was compared between 2014 and 2017.ResultsThe NLP tool identified ILNs with sensitivity and specificity of 91.1% and 82.2%, respectively, in the validation set. Positive and negative predictive values were 59.7% and 97.0%. In reports of ILNs in the training and validation sets before versus after introduction of a Fleischner reporting macro, there was no difference in the proportion of reports with ILNs (108 of 500 [21.6%] versus 101 of 450 [22.4%]; P = .8), or in the proportion of reports with ILNs containing follow-up recommendations (75 of 108 [69.4%] versus 80 of 101 [79.2%]; P = .2]. Rates of recommendation guideline concordance were not significantly different before and after implementation of the standardized macro (52 of 75 [69.3%] versus 60 of 80 [75.0%]; P = .43).ConclusionNLP reliably automates identification of ILNs in unstructured reports, pertinent to quality improvement efforts for ILN management.
| S-EPMC7686874 | biostudies-literature