Dataset Information

Identifying Patients With Delirium Based on Unstructured Clinical Notes: Observational Study.

ABSTRACT:

Background

Delirium in hospitalized patients is a syndrome of acute brain dysfunction. Diagnostic (International Classification of Diseases [ICD]) codes are often used in studies using electronic health records (EHRs), but they are inaccurate.

Objective

We sought to develop a more accurate method using natural language processing (NLP) to detect delirium episodes on the basis of unstructured clinical notes.

Methods

We collected 1.5 million notes from >10,000 patients from among 9 hospitals. Seven experts iteratively labeled 200,471 sentences. Using these, we trained three NLP classifiers: Support Vector Machine, Recurrent Neural Networks, and Transformer. Testing was performed using an external data set. We also evaluated associations with delirium billing (ICD) codes, medications, orders for restraints and sitters, direct assessments (Confusion Assessment Method [CAM] scores), and in-hospital mortality. F1 scores, confusion matrices, and areas under the receiver operating characteristic curve (AUCs) were used to compare NLP models. We used the φ coefficient to measure associations with other delirium indicators.

Results

The transformer NLP performed best on the following parameters: micro F1=0.978, macro F1=0.918, positive AUC=0.984, and negative AUC=0.992. NLP detections exhibited higher correlations (φ) than ICD codes with deliriogenic medications (0.194 vs 0.073 for ICD codes), restraints and sitter orders (0.358 vs 0.177), mortality (0.216 vs 0.000), and CAM scores (0.256 vs -0.028).

Conclusions

Clinical notes are an attractive alternative to ICD codes for EHR delirium studies but require automated methods. Our NLP model detects delirium with high accuracy, similar to manual chart review. Our NLP approach can provide more accurate determination of delirium for large-scale EHR-based studies regarding delirium, quality improvement, and clinical trails.

SUBMITTER: Ge W

PROVIDER: S-EPMC9270709 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundNext-generation sequencing provides comprehensive information about individuals' genetic makeup and is commonplace in oncology clinical practice. However, the utility of genetic information in the clinical decision-making process has not been examined extensively from a real-world, data-driven perspective. Through mining real-world data (RWD) from clinical notes, we could extract patients' genetic information and further associate treatment decisions with genetic information.MethodsWe proposed a real-world evidence (RWE) study framework that incorporates context-based natural language processing (NLP) methods and data quality examination before final association analysis. The framework was demonstrated in a Foundation-tested women cancer cohort (N?=?196). Upon retrieval of patients' genetic information using NLP system, we assessed the completeness of genetic data captured in unstructured clinical notes according to a genetic data-model. We examined the distribution of different topics regarding BRCA1/2 throughout patients' treatment process, and then analyzed the association between BRCA1/2 mutation status and the discussion/prescription of targeted therapy.ResultsWe identified seven topics in the clinical context of genetic mentions including: Information, Evaluation, Insurance, Order, Negative, Positive, and Variants of unknown significance. Our rule-based system achieved a precision of 0.87, recall of 0.93 and F-measure of 0.91. Our machine learning system achieved a precision of 0.901, recall of 0.899 and F-measure of 0.9 for four-topic classification and a precision of 0.833, recall of 0.823 and F-measure of 0.82 for seven-topic classification. We found in result-containing sentences, the capture of BRCA1/2 mutation information was 75%, but detailed variant information (e.g. variant types) is largely missing. Using cleaned RWD, significant associations were found between BRCA1/2 positive mutation and targeted therapies.ConclusionsIn conclusion, we demonstrated a framework to generate RWE using RWD from different clinical sources. Rule-based NLP system achieved the best performance for resolving contextual variability when extracting RWD from unstructured clinical notes. Data quality issues such as incompleteness and discrepancies exist thus manual data cleaning is needed before further analysis can be performed. Finally, we were able to use cleaned RWD to evaluate the real-world utility of genetic information to initiate a prescription of targeted therapy.

Project description:ObjectivesTo develop and validate a simple clinical prediction rule, based on variables easily measurable at admission, to identify patients at high risk of developing delirium during their hospital stay on an internal medicine ward.DesignProspective study of two cohorts of patients admitted between 1 May and 30 June 2008 (derivation cohort), and between 1 May and 30 June 2009 (validation cohort).SettingA tertiary hospital in Donostia-Gipuzkoa (Spain).ParticipantsIn total 397 patients participated in the study. The mean age and incidence of delirium were 75.9 years and 13%, respectively, in the derivation cohort, and 75.8 years and 25% in the validation cohort.Main outcome measuresThe predictive variables analysed and finally included in the rule were: being aged 85 years old or older, being dependent in five or more activities of daily living, and taking two or more psychotropic drugs (antipsychotics, benzodiazepines, antidepressants, anticonvulsant and/or antidementia drugs). The variable of interest was delirium as defined by the short Confusion Assessment Method, which assesses four characteristics: acute onset and fluctuating course, inattention, disorganised thinking and altered level of consciousness.ResultsWe developed a rule in which the individual risk of delirium is obtained by adding one point for each criterion met (age?85, high level of dependence, and being on psychotropic medication). The result is considered positive if the score is ?1. The rule accuracy was: sensitivity=93.4% (95% CI 85.5% to 97.2%), specificity=60.6% (95% CI 54.1% to 66.8%), positive predictive value=44.4% (95% CI 36.9% to 52.1%) and negative predictive value=96.5% (95% CI: 92% to 98.5%). The area under the receiver operator characteristic (ROC) curve was 0.85 for the validation cohort.ConclusionsThe presence or absence of any of the three predictive factors (age?85, high level of dependence and psychotropic medication) allowed us to classify patients on internal medicine wards according to the risk of developing delirium. The simplicity of the variables in our clinical prediction rule means that the data collection required is feasible in busy medicine units.