Dataset Information

Developing an Interpretable Machine Learning Model to Predict in-Hospital Mortality in Sepsis Patients: A Retrospective Temporal Validation Study.

ABSTRACT:

Background

Risk stratification plays an essential role in the decision making for sepsis management, as existing approaches can hardly satisfy the need to assess this heterogeneous population. We aimed to develop and validate a machine learning model to predict in-hospital mortality in critically ill patients with sepsis.

Methods

Adult patients fulfilling the definition of Sepsis-3 were included at a large tertiary medical center. Relevant clinical features were extracted within the first 24 h in ICU, re-classified into different genres, and utilized for model development under three strategies: "Basic + Lab", "Basic + Intervention", and "Whole" feature sets. Extreme gradient boosting (XGBoost) was compared with logistic regression (LR) and established severity scores. Temporal validation was conducted using admissions from 2017 to 2019.

Results

The final cohort included 24,272 patients, of which 4013 patients formed the test cohort for temporal validation. The trained and fine-tuned XGBoost model with the whole feature set showed the best discriminatory ability in the test cohort with AUROC as 0.85, significantly higher than the XGBoost "Basic + Lab" model (0.83), the LR "Whole" model (0.82), SOFA (0.63), SAPS-II (0.73), and LODS score (0.74). The performance in varying subgroups remained robust, and predictors, such as increased urine output and supplemental oxygen therapy, were crucially correlated with improved survival when interpretability was explored.

Conclusions

We developed and validated a novel XGBoost-based model and demonstrated significantly improved performance to LR and other scores in predicting the mortality risks of sepsis patients in the hospital using features in the first 24 h.

SUBMITTER: Li S

PROVIDER: S-EPMC9917524 | biostudies-literature | 2023 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Developing an Interpretable Machine Learning Model to Predict in-Hospital Mortality in Sepsis Patients: A Retrospective Temporal Validation Study.

Li Shuhe S Dou Ruoxu R Song Xiaodong X Lui Ka Yin KY Xu Jinghong J Guo Zilu Z Hu Xiaoguang X Guan Xiangdong X Cai Changjie C

Journal of clinical medicine 20230124 3

<h4>Background</h4>Risk stratification plays an essential role in the decision making for sepsis management, as existing approaches can hardly satisfy the need to assess this heterogeneous population. We aimed to develop and validate a machine learning model to predict in-hospital mortality in critically ill patients with sepsis.<h4>Methods</h4>Adult patients fulfilling the definition of Sepsis-3 were included at a large tertiary medical center. Relevant clinical features were extracted within t ...[more]

PMID: 36769564

Similar Datasets

Project description:BackgroundEarly and reliable identification of patients with sepsis who are at high risk of mortality is important to improve clinical outcomes. However, 3 major barriers to artificial intelligence (AI) models, including the lack of interpretability, the difficulty in generalizability, and the risk of automation bias, hinder the widespread adoption of AI models for use in clinical practice.ObjectiveThis study aimed to develop and validate (internally and externally) a conformal predictor of sepsis mortality risk in patients who are critically ill, leveraging AI-assisted prediction modeling. The proposed approach enables explaining the model output and assessing its confidence level.MethodsWe retrospectively extracted data on adult patients with sepsis from a database collected in a teaching hospital at Beth Israel Deaconess Medical Center for model training and internal validation. A large multicenter critical care database from the Philips eICU Research Institute was used for external validation. A total of 103 clinical features were extracted from the first day after admission. We developed an AI model using gradient-boosting machines to predict the mortality risk of sepsis and used Mondrian conformal prediction to estimate the prediction uncertainty. The Shapley additive explanation method was used to explain the model.ResultsA total of 16,746 (80%) patients from Beth Israel Deaconess Medical Center were used to train the model. When tested on the internal validation population of 4187 (20%) patients, the model achieved an area under the receiver operating characteristic curve of 0.858 (95% CI 0.845-0.871), which was reduced to 0.800 (95% CI 0.789-0.811) when externally validated on 10,362 patients from the Philips eICU database. At a specified confidence level of 90% for the internal validation cohort the percentage of error predictions (n=438) out of all predictions (n=4187) was 10.5%, with 1229 (29.4%) predictions requiring clinician review. In contrast, the AI model without conformal prediction made 1449 (34.6%) errors. When externally validated, more predictions (n=4004, 38.6%) were flagged for clinician review due to interdatabase heterogeneity. Nevertheless, the model still produced significantly lower error rates compared to the point predictions by AI (n=1221, 11.8% vs n=4540, 43.8%). The most important predictors identified in this predictive model were Acute Physiology Score III, age, urine output, vasopressors, and pulmonary infection. Clinically relevant risk factors contributing to a single patient were also examined to show how the risk arose.ConclusionsBy combining model explanation and conformal prediction, AI-based systems can be better translated into medical practice for clinical decision-making.

Project description:ObjectiveSepsis-induced coagulopathy (SIC) is extremely common in individuals with sepsis, significantly associated with poor outcomes. This study attempted to develop an interpretable and generalizable machine learning (ML) model for early predicting the risk of 28-day death in patients with SIC.MethodsIn this retrospective cohort study, we extracted SIC patients from the Medical Information Mart for Intensive Care III (MIMIC-III), MIMIC-IV, and eICU-CRD database according to Toshiaki Iba's scale. And the overlapping in the MIMIC-IV was excluded for this study. Afterward, only the MIMIC-III cohort was randomly divided into the training set, and the internal validation set according to the ratio of 7:3, while the MIMIC-IV and eICU-CRD databases were considered the external validation sets. The predictive factors for 28-day mortality of SIC patients were determined using recursive feature elimination combined with tenfold cross-validation (RFECV). Then, we constructed models using ML algorithms. Multiple metrics were used for evaluation of performance of the models, including the area under the receiver operating characteristic curve (AUROC), area under the precision recall curve (AUPRC), accuracy, sensitivity, specificity, negative predictive value, positive predictive value, recall, and F1 score. Finally, Shapley Additive Explanations (SHAP), Local Interpretable Model-Agnostic Explanations (LIME) were employed to provide a reasonable interpretation for the prediction results.ResultsA total of 3280, 2798, and 1668 SIC patients were screened from MIMIC-III, MIMIC-IV, and eICU-CRD databases, respectively. Seventeen features were selected to construct ML prediction models. XGBoost had the best performance in predicting the 28-day mortality of SIC patients, with AUC of 0.828, 0.913 and 0.923, the AUPRC of 0.807, 0.796 and 0.921, the accuracy of 0.785, 0.885 and 0.891, the F1 scores were 0.63, 0.69 and 0.70 in MIMIC-III (internal validation set), MIMIC-IV, and eICU-CRD databases. The importance ranking and SHAP analyses showed that initial SOFA score, red blood cell distribution width (RDW), and age were the top three critical features in the XGBoost model.ConclusionsWe developed an optimal and explainable ML model to predict the risk of 28-day death of SIC patients 28-day death risk. Compared with conventional scoring systems, the XGBoost model performed better. The model established will have the potential to improve the level of clinical practice for SIC patients.

Project description:OBJECTIVES:A growing number of patients survive sepsis hospitalizations each year and are at high risk for readmission. However, little is known about temporal trends in hospital-based acute care (emergency department treat-and-release visits and hospital readmission) after sepsis. Our primary objective was to measure temporal trends in sepsis survivorship and hospital-based acute care use in sepsis survivors. In addition, because readmissions after pneumonia are subject to penalty under the national readmission reduction program, we examined whether readmission rates declined after sepsis hospitalizations related to pneumonia. DESIGN AND SETTING:Retrospective, observational cohort study conducted within an academic healthcare system from 2010 to 2015. PATIENTS:We used three validated, claims-based approaches to identify 17,256 sepsis or severe sepsis hospitalizations to examine trends in hospital-based acute care after sepsis. INTERVENTIONS:None. MEASUREMENTS AND MAIN RESULTS:From 2010 to 2015, sepsis as a proportion of medical and surgical admissions increased from 3.9% to 9.4%, whereas in-hospital mortality rate for sepsis hospitalizations declined from 24.1% to 14.8%. As a result, the proportion of medical and surgical discharges at-risk for hospital readmission after sepsis increased from 2.7% to 7.8%. Over 6 years, 30-day hospital readmission rates declined modestly, from 26.4% in 2010 to 23.1% in 2015, driven largely by a decline in readmission rates among survivors of nonsevere sepsis, and nonpneumonia sepsis specifically, as the readmission rate of severe sepsis survivors was stable. The modest decline in 30-day readmission rates was offset by an increase in emergency department treat-and-release visits, from 2.8% in 2010 to a peak of 5.4% in 2014. CONCLUSIONS:Owing to increasing incidence and declining mortality, the number of sepsis survivors at risk for hospital readmission rose significantly between 2010 and 2015. The 30-day hospital readmission rates for sepsis declined modestly but were offset by a rise in emergency department treat-and-release visits.

Project description:BackgroundEarly and accurate identification of septic patients at high risk for ICU mortality can help clinicians make optimal clinical decisions and improve the patients' outcomes. This study aimed to develop and validate (internally and externally) a mortality prediction score for sepsis following admission in the ICU.MethodsWe extracted data retrospectively regarding adult septic patients from one teaching hospital in Wenzhou, China and a large multi-center critical care database from the USA. Demographic data, vital signs, laboratory values, comorbidities, and clinical outcomes were collected. The primary outcome was ICU mortality. Through multivariable logistic regression, a mortality prediction score for sepsis was developed and validated.ResultsFour thousand two hundred and thirty six patients in the development cohort and 8359 patients in three validation cohorts. The Prediction of Sepsis Mortality in ICU (POSMI) score included age ≥ 50 years, temperature < 37 °C, Respiratory rate > 35 breaths/min, MAP ≤ 50 mmHg, SpO2 < 90%, albumin ≤ 2 g/dL, bilirubin ≥ 0.8 mg/dL, lactate ≥ 4.2 mmol/L, BUN ≥ 21 mg/dL, mechanical ventilation, hepatic failure and metastatic cancer. In addition, the area under the receiver operating characteristic curve (AUC) for the development cohort was 0.831 (95% CI, 0.813-0.850) while the AUCs ranged from 0.798 to 0.829 in the three validation cohorts. Moreover, the POSMI score had a higher AUC than both the SOFA and APACHE IV scores. Notably, the Hosmer-Lemeshow (H-L) goodness-of-fit test results and calibration curves suggested good calibration in the development and validation cohorts. Additionally, the POSMI score still exhibited excellent discrimination and calibration following sensitivity analysis. With regard to clinical usefulness, the decision curve analysis (DCA) of POSMI showed a higher net benefit than SOFA and APACHE IV in the development cohort.ConclusionPOSMI was validated to be an effective tool for predicting mortality in ICU patients with sepsis.

Project description:Machine learning (ML), one aspect of artificial intelligence (AI), involves computer algorithms that train themselves. They have been widely applied in the healthcare domain. However, many trained ML algorithms operate as 'black boxes', producing a prediction from input data without a clear explanation of their workings. Non-transparent predictions are of limited utility in many clinical domains, where decisions must be justifiable. Here, we apply class-contrastive counterfactual reasoning to ML to demonstrate how specific changes in inputs lead to different predictions of mortality in people with severe mental illness (SMI), a major public health challenge. We produce predictions accompanied by visual and textual explanations as to how the prediction would have differed given specific changes to the input. We apply it to routinely collected data from a mental health secondary care provider in patients with schizophrenia. Using a data structuring framework informed by clinical knowledge, we captured information on physical health, mental health, and social predisposing factors. We then trained an ML algorithm and other statistical learning techniques to predict the risk of death. The ML algorithm predicted mortality with an area under receiver operating characteristic curve (AUROC) of 0.80 (95% confidence intervals [0.78, 0.82]). We used class-contrastive analysis to produce explanations for the model predictions. We outline the scenarios in which class-contrastive analysis is likely to be successful in producing explanations for model predictions. Our aim is not to advocate for a particular model but show an application of the class-contrastive analysis technique to electronic healthcare record data for a disease of public health significance. In patients with schizophrenia, our work suggests that use or prescription of medications like antidepressants was associated with lower risk of death. Abuse of alcohol/drugs and a diagnosis of delirium were associated with higher risk of death. Our ML models highlight the role of co-morbidities in determining mortality in patients with schizophrenia and the need to manage co-morbidities in these patients. We hope that some of these bio-social factors can be targeted therapeutically by either patient-level or service-level interventions. Our approach combines clinical knowledge, health data, and statistical learning, to make predictions interpretable to clinicians using class-contrastive reasoning. This is a step towards interpretable AI in the management of patients with schizophrenia and potentially other diseases.

Project description:BackgroundAcute kidney injury has been described as a common complication in patients hospitalized with COVID-19, which may lead to the need for kidney replacement therapy (KRT) in its most severe forms. Our group developed and validated the MMCD score in Brazilian COVID-19 patients to predict KRT, which showed excellent performance using data from 2020. This study aimed to validate the MMCD score in a large cohort of patients hospitalized with COVID-19 in a different pandemic phase and assess its performance to predict in-hospital mortality.MethodsThis study is part of the "Brazilian COVID-19 Registry", a retrospective observational cohort of consecutive patients hospitalized for laboratory-confirmed COVID-19 in 25 Brazilian hospitals between March 2021 and August 2022. The primary outcome was KRT during hospitalization and the secondary was in-hospital mortality. We also searched literature for other prediction models for KRT, to assess the results in our database. Performance was assessed using area under the receiving operator characteristic curve (AUROC) and the Brier score.ResultsA total of 9422 patients were included, 53.8% were men, with a median age of 59 (IQR 48-70) years old. The incidence of KRT was 8.8% and in-hospital mortality was 18.1%. The MMCD score had excellent discrimination and overall performance to predict KRT (AUROC: 0.916 [95% CI 0.909-0.924]; Brier score = 0.057). Despite the excellent discrimination and overall performance (AUROC: 0.922 [95% CI 0.914-0.929]; Brier score = 0.100), the calibration was not satisfactory concerning in-hospital mortality. A random forest model was applied in the database, with inferior performance to predict KRT requirement (AUROC: 0.71 [95% CI 0.69-0.73]).ConclusionThe MMCD score is not appropriate for in-hospital mortality but demonstrates an excellent predictive ability to predict KRT in COVID-19 patients. The instrument is low cost, objective, fast and accurate, and can contribute to supporting clinical decisions in the efficient allocation of assistance resources in patients with COVID-19.

Project description:While deep neural networks (DNNs) and other machine learning models often have higher accuracy than simpler models like logistic regression (LR), they are often considered to be "black box" models and this lack of interpretability and transparency is considered a challenge for clinical adoption. In healthcare, intelligible models not only help clinicians to understand the problem and create more targeted action plans, but also help to gain the clinicians' trust. One method of overcoming the limited interpretability of more complex models is to use Generalized Additive Models (GAMs). Standard GAMs simply model the target response as a sum of univariate models. Inspired by GAMs, the same idea can be applied to neural networks through an architecture referred to as Generalized Additive Models with Neural Networks (GAM-NNs). In this manuscript, we present the development and validation of a model applying the concept of GAM-NNs to allow for interpretability by visualizing the learned feature patterns related to risk of in-hospital mortality for patients undergoing surgery under general anesthesia. The data consists of 59,985 patients with a feature set of 46 features extracted at the end of surgery to which we added previously not included features: total anesthesia case time (1 feature); the time in minutes spent with mean arterial pressure (MAP) below 40, 45, 50, 55, 60, and 65 mmHg during surgery (6 features); and Healthcare Cost and Utilization Project (HCUP) Code Descriptions of the Primary current procedure terminology (CPT) codes (33 features) for a total of 86 features. All data were randomly split into 80% for training (n = 47,988) and 20% for testing (n = 11,997) prior to model development. Model performance was compared to a standard LR model using the same features as the GAM-NN. The data consisted of 59,985 surgical records, and the occurrence of in-hospital mortality was 0.81% in the training set and 0.72% in the testing set. The GAM-NN model with HCUP features had the highest area under the curve (AUC) 0.921 (0.895-0.95). Overall, both GAM-NN models had higher AUCs than LR models, however, had lower average precisions. The LR model without HCUP features had the highest average precision 0.217 (0.136-0.31). To assess the interpretability of the GAM-NNs, we then visualized the learned contributions of the GAM-NNs and compared against the learned contributions of the LRs for the models with HCUP features. Overall, we were able to demonstrate that our proposed generalized additive neural network (GAM-NN) architecture is able to (1) leverage a neural network's ability to learn nonlinear patterns in the data, which is more clinically intuitive, (2) be interpreted easily, making it more clinically useful, and (3) maintain model performance as compared to previously published DNNs.

Project description:ObjectiveDetermine if deep learning detects sepsis earlier and more accurately than other models. To evaluate model performance using implementation-oriented metrics that simulate clinical practice.Materials and methodsWe trained internally and temporally validated a deep learning model (multi-output Gaussian process and recurrent neural network [MGP-RNN]) to detect sepsis using encounters from adult hospitalized patients at a large tertiary academic center. Sepsis was defined as the presence of 2 or more systemic inflammatory response syndrome (SIRS) criteria, a blood culture order, and at least one element of end-organ failure. The training dataset included demographics, comorbidities, vital signs, medication administrations, and labs from October 1, 2014 to December 1, 2015, while the temporal validation dataset was from March 1, 2018 to August 31, 2018. Comparisons were made to 3 machine learning methods, random forest (RF), Cox regression (CR), and penalized logistic regression (PLR), and 3 clinical scores used to detect sepsis, SIRS, quick Sequential Organ Failure Assessment (qSOFA), and National Early Warning Score (NEWS). Traditional discrimination statistics such as the C-statistic as well as metrics aligned with operational implementation were assessed.ResultsThe training set and internal validation included 42 979 encounters, while the temporal validation set included 39 786 encounters. The C-statistic for predicting sepsis within 4 h of onset was 0.88 for the MGP-RNN compared to 0.836 for RF, 0.849 for CR, 0.822 for PLR, 0.756 for SIRS, 0.619 for NEWS, and 0.481 for qSOFA. MGP-RNN detected sepsis a median of 5 h in advance. Temporal validation assessment continued to show the MGP-RNN outperform all 7 clinical risk score and machine learning comparisons.ConclusionsWe developed and validated a novel deep learning model to detect sepsis. Using our data elements and feature set, our modeling approach outperformed other machine learning methods and clinical scores.

Dataset Information

Developing an Interpretable Machine Learning Model to Predict in-Hospital Mortality in Sepsis Patients: A Retrospective Temporal Validation Study.

Background

Methods

Results

Conclusions

Publications

Developing an Interpretable Machine Learning Model to Predict in-Hospital Mortality in Sepsis Patients: A Retrospective Temporal Validation Study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets