Dataset Information

Validation of an Automatic Tagging System for Identifying Respiratory and Hemodynamic Deterioration Events in the Intensive Care Unit.

ABSTRACT:

Objective

Predictive models for critical events in the intensive care unit (ICU) might help providers anticipate patient deterioration. At the heart of predictive model development lies the ability to accurately label significant events, thereby facilitating the use of machine learning and similar strategies. We conducted this study to establish the validity of an automated system for tagging respiratory and hemodynamic deterioration by comparing automatic tags to tagging by expert reviewers.

Methods

This retrospective cohort study included 72,650 unique patient stays collected from Electronic Medical Records of the University of Massachusetts' eICU. An enriched subgroup of stays was manually tagged by expert reviewers. The tags generated by the reviewers were compared to those generated by an automated system.

Results

The automated system was able to rapidly and efficiently tag the complete database utilizing available clinical data. The overall agreement rate between the automated system and the clinicians for respiratory and hemodynamic deterioration tags was 89.4% and 87.1%, respectively. The automatic system did not add substantial variability beyond that seen among the reviewers.

Conclusions

We demonstrated that a simple rule-based tagging system could provide a rapid and accurate tool for mass tagging of a compound database. These types of tagging systems may replace human reviewers and save considerable resources when trying to create a validated, labeled database used to train artificial intelligence algorithms. The ability to harness the power of artificial intelligence depends on efficient clinical validation of targeted conditions; hence, these systems and the methodology used to validate them are crucial.

SUBMITTER: Jeddah D

PROVIDER: S-EPMC8369051 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundEarly prediction model of hemodynamic instability has the potential to improve the critical care, whereas limited external validation on the generalizability. We aimed to independently validate the Hemodynamic Stability Index (HSI), a multi-parameter machine learning model, in predicting hemodynamic instability in Asian patients.MethodHemodynamic instability was marked by using inotropic, vasopressor, significant fluid therapy, and/or blood transfusions. This retrospective study included among 15,967 ICU patients who aged 20 years or older (not included 20 years) and stayed in ICU for more than 6 h admitted to Taipei Veteran General Hospital (TPEVGH) between January 1, 2010, and March 31, 2020, of whom hemodynamic instability occurred in 3053 patients (prevalence = 19%). These patients in unstable group received at least one intervention during their ICU stays, and the HSI score of both stable and unstable group was calculated in every hour before intervention. The model performance was assessed using the area under the receiver operating characteristic curve (AUROC) and was compared to single indicators like systolic blood pressure (SBP) and shock index. The hemodynamic instability alarm was set by selecting optimal threshold with high sensitivity, acceptable specificity, and lead time before intervention was calculated to indicate when patients were firstly identified as high risk of hemodynamic instability.ResultsThe AUROC of HSI was 0.76 (95% CI, 0.75-0.77), which performed significantly better than shock Index (0.7; 95% CI, 0.69-0.71) and SBP (0.69; 95% CI, 0.68-0.70). By selecting 0.7 as a threshold, HSI predicted 72% of all 3053 patients who received hemodynamic interventions with 67% in specificity. Time-varying results also showed that HSI score significantly outperformed single indicators even up to 24 h before intervention. And 95% unstable patients can be identified more than 5 h in advance.ConclusionsThe HSI has acceptable discrimination but underestimates the risk of stable patients in predicting the onset of hemodynamic instability in an external cohort.

Project description:BackgroundTimely recognition of hemodynamic instability in critically ill patients enables increased vigilance and early treatment opportunities. We develop the Hemodynamic Stability Index (HSI), which highlights situational awareness of possible hemodynamic instability occurring at the bedside and to prompt assessment for potential hemodynamic interventions.MethodsWe used an ensemble of decision trees to obtain a real-time risk score that predicts the initiation of hemodynamic interventions an hour into the future. We developed the model using the eICU Research Institute (eRI) database, based on adult ICU admissions from 2012 to 2016. A total of 208,375 ICU stays met the inclusion criteria, with 32,896 patients (prevalence = 18%) experiencing at least one instability event where they received one of the interventions during their stay. Predictors included vital signs, laboratory measurements, and ventilation settings.ResultsHSI showed significantly better performance compared to single parameters like systolic blood pressure and shock index (heart rate/systolic blood pressure) and showed good generalization across patient subgroups. HSI AUC was 0.82 and predicted 52% of all hemodynamic interventions with a lead time of 1-h with a specificity of 92%. In addition to predicting future hemodynamic interventions, our model provides confidence intervals and a ranked list of clinical features that contribute to each prediction. Importantly, HSI can use a sparse set of physiologic variables and abstains from making a prediction when the confidence is below an acceptable threshold.ConclusionsThe HSI algorithm provides a single score that summarizes hemodynamic status in real time using multiple physiologic parameters in patient monitors and electronic medical records (EMR). Importantly, HSI is designed for real-world deployment, demonstrating generalizability, strong performance under different data availability conditions, and providing model explanation in the form of feature importance and prediction confidence.

Project description:BackgroundTransitions of care between providers are vulnerable periods in healthcare delivery that expose patients to preventable errors and adverse events. Patient discharge from the intensive care unit (ICU) to a medical or surgical hospital ward is one of the most challenging and high risk transitions of care. Approximately 1 in 12 patients discharged will be readmitted to ICU or die before leaving the hospital. Many more patients are exposed to unnecessary healthcare, adverse events and/or are disappointed with the quality of their care. Our objective is to conduct a scoping review by systematically searching the literature to identify ICU discharge planning tools and their supporting evidence-base including barriers and facilitators to their use.Methods and analysisSystematic searching of the published health literature will be conducted to identify the existing ICU discharge planning tools and supporting evidence. Literature (research and non-research) reporting on the tools used to facilitate decision making and/or communication at ICU discharge with patients of any age will be included. Outcomes will include adverse events and provider and patient/family-reported outcomes. Two investigators will independently review the abstracts (screen 1) to identify those meeting the inclusion criteria and then independently assess the full text articles (screen 2) to determine if they meet the inclusion criteria. Data collection will include information on citations and identified tools. A quality assessment will be performed on original research studies. A descriptive summary will be developed for each tool.Ethics and disseminationOur scoping review will synthesise the literature for ICU discharge planning tools and identify the opportunities for knowledge to action and gaps in evidence where primary evidence is necessary. This will serve as the foundational element in a multistep research programme to standardise and improve the quality of care provided to patients during ICU discharge. Ethics approval is not required for this study.