Dataset Information

Comparison of 2 Natural Language Processing Methods for Identification of Bleeding Among Critically Ill Patients.

ABSTRACT:

Importance

To improve patient safety, health care systems need reliable methods to detect adverse events in large patient populations. Events are often described in clinical notes, rather than structured data, which make them difficult to identify on a large scale.

Objective

To develop and compare 2 natural language processing methods, a rules-based approach and a machine learning (ML) approach, for identifying bleeding events in clinical notes.

Design, setting, and participants

This diagnostic study used deidentified notes from the Medical Information Mart for Intensive Care, which spans 2001 to 2012. A training set of 990 notes and a test set of 660 notes were randomly selected. Physicians classified each note as present or absent for a clinically relevant bleeding event during the hospitalization. A bleeding dictionary was developed for the rules-based approach; bleeding mentions were then aggregated to arrive at a classification for each note. Three ML models (support vector machine, extra trees, and convolutional neural network) were developed and trained using the 990-note training set. Another instance of each ML model was also trained on a sample of 450 notes, with equal numbers of bleeding-present and bleeding-absent notes. The notes were represented using term frequency-inverse document frequency vectors and global vectors for word representation.

Main outcomes and measures

The main outcomes were accuracy, sensitivity, specificity, positive predictive value, and negative predictive value for each model. Following training, the models were tested on the test set and sensitivities were compared using a McNemar test.

Results

The 990-note training set represented 769 patients (296 [38.5%] female; mean [SD] age, 67.42 [14.7] years). The 660-note test set represented 527 patients (211 [40.0%] female; mean [SD] age, 67.86 [14.7] years). Bleeding was present in 146 notes (22.1%). The extra trees down-sampled model and rules-based approaches were similarly sensitive (93.8% vs 91.1%; difference, 2.7%; 95% CI, -3.8% to 7.9%; P?=?.44). The positive predictive value for the extra trees model, however, was 48.6%. The rules-based model had the best performance overall, with 84.6% specificity, 62.7% positive predictive value, and 97.1% negative predictive value.

Conclusions and relevance

Bleeding is a common complication in health care, and these results demonstrate an automated and scalable detection method. The rules-based natural language processing approach, compared with ML, had the best performance in identifying bleeding, with high sensitivity and negative predictive value.

SUBMITTER: Taggart M

PROVIDER: S-EPMC6324448 | biostudies-literature | 2018 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Comparison of 2 Natural Language Processing Methods for Identification of Bleeding Among Critically Ill Patients.

Taggart Maxwell M Chapman Wendy W WW Steinberg Benjamin A BA Ruckel Shane S Pregenzer-Wenzler Arianna A Du Yishuai Y Ferraro Jeffrey J Bucher Brian T BT Lloyd-Jones Donald M DM Rondina Matthew T MT Shah Rashmee U RU

JAMA network open 20181005 6

<h4>Importance</h4>To improve patient safety, health care systems need reliable methods to detect adverse events in large patient populations. Events are often described in clinical notes, rather than structured data, which make them difficult to identify on a large scale.<h4>Objective</h4>To develop and compare 2 natural language processing methods, a rules-based approach and a machine learning (ML) approach, for identifying bleeding events in clinical notes.<h4>Design, setting, and participant ...[more]

PMID: 30646240

Similar Datasets

Project description:BackgroundAcute gastrointestinal bleeding (GIB) may be a severe condition in immunocompromised patients and may require intensive care unit (ICU) admission. We aimed to describe the clinical spectrum of critically ill immunocompromised patients with GIB and identify risk factors associated with mortality and severe GIB defined by hemorrhagic shock, hyperlactatemia and/or the transfusion of more than 5 red blood cells units. Finally, we compared this cohort with a control population of non-immunocompromised admitted in ICU for GIB.ResultsRetrospective study in 3 centers including immunocompromised patients with GIB admitted in ICU from January, 1st 2010 to December, 31rd 2019. Risk factors for mortality and severe GIB were assessed by logistic regression. Immunocompromised patients were matched with a control group of patients admitted in ICU with GIB. A total of 292 patients were analyzed in the study, including 141 immunocompromised patients (compared to a control group of 151 patients). Among immunocompromised patients, upper GIB was more frequent (73%) than lower GIB (27%). By multivariate analysis, severe GIB was associated with male gender (OR 4.48, CI95% 1.75-11.42, p = 0.00), upper GIB (OR 2.88, CI95% 1.11-7.46, p = 0.03) and digestive malignant infiltration (OR 5.85, CI95% 1.45-23.56, p = 0.01). Conversely, proton pump inhibitor treatment before hospitalization was significantly associated with decreased risk of severe GIB (OR 0.25, IC95% 0.10-0.65, p < 0.01). Fifty-four patients (38%) died within 90 days. By multivariate analysis, mortality was associated with hemorrhagic shock (OR 2.91, IC95% 1.33-6.38, p = 0 .01), upper GIB (OR 4.33, CI95% 1.50-12.47, p = 0.01), and long-term corticosteroid therapy before admission (OR 2.98, CI95% 1.32-6.71, p = 0.01). Albuminemia (per 5 g/l increase) was associated with lower mortality (OR 0.54, IC95% 0.35-0.84, p = 0.01). After matching with a control group of non-immunocompromised patients, severity of bleeding was increased in immunocompromised patients, but mortality was not different between the 2 groups.ConclusionMortality is high in immunocompromised patients with GIB in ICU, especially in patients receiving long term corticosteroids. Mortality of GIB is not different from mortality of non-immunocompromised patients in ICU. The prophylactic administration of proton pump inhibitors should be considered in this population.

Project description:BackgroundDelirium is an acute neurocognitive disorder that affects up to half of older hospitalized medical patients and can lead to dementia, longer hospital stays, increased health costs, and death. Although delirium can be prevented and treated, it is difficult to identify and predict.ObjectiveThis study aimed to improve machine learning models that retrospectively identify the presence of delirium during hospital stays (eg, to measure the effectiveness of delirium prevention interventions) by using the natural language processing (NLP) technique of sentiment analysis (in this case a feature that identifies sentiment toward, or away from, a delirium diagnosis).MethodsUsing data from the General Medicine Inpatient Initiative, a Canadian hospital data and analytics network, a detailed manual review of medical records was conducted from nearly 4000 admissions at 6 Toronto area hospitals. Furthermore, 25.74% (994/3862) of the eligible hospital admissions were labeled as having delirium. Using the data set collected from this study, we developed machine learning models with, and without, the benefit of NLP methods applied to diagnostic imaging reports, and we asked the question "can NLP improve machine learning identification of delirium?"ResultsAmong the eligible 3862 hospital admissions, 994 (25.74%) admissions were labeled as having delirium. Identification and calibration of the models were satisfactory. The accuracy and area under the receiver operating characteristic curve of the main model with NLP in the independent testing data set were 0.807 and 0.930, respectively. The accuracy and area under the receiver operating characteristic curve of the main model without NLP in the independent testing data set were 0.811 and 0.869, respectively. Model performance was also found to be stable over the 5-year period used in the experiment, with identification for a likely future holdout test set being no worse than identification for retrospective holdout test sets.ConclusionsOur machine learning model that included NLP (ie, sentiment analysis in medical image description text mining) produced valid identification of delirium with the sentiment analysis, providing significant additional benefit over the model without NLP.

Project description:BackgroundHealth researchers are increasingly using natural language processing (NLP) to study various mental health conditions using both social media and electronic health records (EHRs). There is currently no published synthesis that relates specifically to the use of NLP methods for bipolar disorder, and this scoping review was conducted to synthesize valuable insights that have been presented in the literature.ObjectiveThis scoping review explored how NLP methods have been used in research to better understand bipolar disorder and identify opportunities for further use of these methods.MethodsA systematic, computerized search of index and free-text terms related to bipolar disorder and NLP was conducted using 5 databases and 1 anthology: MEDLINE, PsycINFO, Academic Search Ultimate, Scopus, Web of Science Core Collection, and the ACL Anthology.ResultsOf 507 identified studies, a total of 35 (6.9%) studies met the inclusion criteria. A narrative synthesis was used to describe the data, and the studies were grouped into four objectives: prediction and classification (n=25), characterization of the language of bipolar disorder (n=13), use of EHRs to measure health outcomes (n=3), and use of EHRs for phenotyping (n=2). Ethical considerations were reported in 60% (21/35) of the studies.ConclusionsThe current literature demonstrates how language analysis can be used to assist in and improve the provision of care for people living with bipolar disorder. Individuals with bipolar disorder and the medical community could benefit from research that uses NLP to investigate risk-taking, web-based services, social and occupational functioning, and the representation of gender in bipolar disorder populations on the web. Future research that implements NLP methods to study bipolar disorder should be governed by ethical principles, and any decisions regarding the collection and sharing of data sets should ultimately be made on a case-by-case basis, considering the risk to the data participants and whether their privacy can be ensured.

Project description:Objective: Bleeding can be a severe complication of critical illness, but its true epidemiologic impact on children has seldom been studied. Our objective is to describe the epidemiology of bleeding in critically ill children, using a validated clinical tool, as well as the hemostatic interventions and clinical outcomes associated with bleeding. Design: Prospective observational cohort study. Setting: Tertiary pediatric critical care unit Patients: All consecutive patients (1 month to 18 years of age) admitted to a tertiary pediatric critical care unit Measurements and Main Results: Bleeding events were categorized as minimal, moderate, severe, or fatal, according to the Bleeding Assessment Scale in Critically Ill Children. We collected demographics and severity at admission, as evaluated by the Pediatric Index of Mortality. We used regression models to compare the severity of bleeding with outcomes adjusting for age, surgery, and severity. Over 12 months, 902 critically ill patients were enrolled. The median age was 64 months (IQR 17; 159), the median admission predicted risk of mortality was 0.5% (IQR 0.2; 1.4), and 24% were post-surgical. Eighteen percent of patients experienced at least one bleeding event. The highest severity of bleeding was minimal for 7.9% of patients, moderate for 5.8%, severe for 3.8%, and fatal for 0.1%. Adjusting for age, severity at admission, medical diagnosis, type of surgery, and duration of surgery, bleeding severity was independently associated with fewer ventilator-free days (p < 0.001) and fewer PICU-free days (p < 0.001). Adjusting for the same variables, bleeding severity was independently associated with an increased risk of mortality (adjusted odds ratio for each bleeding category 2.4, 95% CI 1.5; 3.7, p < 0.001). Conclusion: Our data indicate bleeding occurs in nearly one-fifth of all critically ill children, and that higher severity of bleeding was independently associated with worse clinical outcome. Further multicenter studies are required to better understand the impact of bleeding in critically ill children.

Project description:BackgroundPeripheral artery disease (PAD) is underrecognized, undertreated, and understudied: each of these endeavors requires efficient and accurate identification of patients with PAD. Currently, PAD patient identification relies on diagnosis/procedure codes or lists of patients diagnosed or treated by specific providers in specific locations and ways. The goal of this research was to leverage natural language processing to more accurately identify patients with PAD in an electronic health record system compared with a structured data-based approach.MethodsThe clinical notes from a cohort of 6861 patients in our health system whose PAD status had previously been adjudicated were used to train, test, and validate a natural language processing model using 10-fold cross-validation. The performance of this model was described using the area under the receiver operating characteristic and average precision curves; its performance was quantitatively compared with an administrative data-based least absolute shrinkage and selection operator (LASSO) approach using the DeLong test.ResultsThe median (SD) of the area under the receiver operating characteristic curve for the natural language processing model was 0.888 (0.009) versus 0.801 (0.017) for the LASSO-based approach alone (DeLong P<0.0001). The median (SD) of the area under the precision curve was 0.909 (0.008) versus 0.816 (0.012) for the structured data-based approach. When sensitivity was set at 90%, the precision for LASSO was 65% and the machine learning approach was 74%, while the specificity for LASSO was 41% and for the machine learning approach was 62%.ConclusionsUsing a natural language processing approach in addition to partial cohort preprocessing with a LASSO-based model, we were able to meaningfully improve our ability to identify patients with PAD compared with an approach using structured data alone. This model has potential applications to both interventions targeted at improving patient care as well as efficient, large-scale PAD research. Graphic Abstract: A graphic abstract is available for this article.

Dataset Information

Comparison of 2 Natural Language Processing Methods for Identification of Bleeding Among Critically Ill Patients.

Importance

Objective

Design, setting, and participants

Main outcomes and measures

Results

Conclusions and relevance

Publications

Comparison of 2 Natural Language Processing Methods for Identification of Bleeding Among Critically Ill Patients.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets