Development and validation of a machine learning model to predict mortality risk in patients with COVID-19.
Ontology highlight
ABSTRACT:
SUBMITTER: Stachel A
PROVIDER: S-EPMC8108129 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:This study was designed to develop machine-learning models to predict COVID-19 mortality and identify its key features based on clinical characteristics and laboratory tests. For this, deep-learning (DL) and machine-learning (ML) models were developed using receiver operating characteristic (ROC) area under the curve (AUC) and F1 score optimization of 87 parameters. Of the two, the DL model exhibited better performance (AUC 0.8721, accuracy 0.84, and F1 score 0.76). However, we also blended DL with ML, and the ensemble model performed the best (AUC 0.8811, accuracy 0.85, and F1 score 0.77). The DL model is generally unable to extract feature importance; however, we succeeded by using the Shapley Additive exPlanations method for each model. This study demonstrated both the applicability of DL and ML models for classifying COVID-19 mortality using hospital-structured data and that the ensemble model had the best predictive ability.
| S-EPMC9221552 | biostudies-literature
Project description:ImportanceMachine-learning algorithms offer better predictive accuracy than traditional prognostic models but are too complex and opaque for clinical use.ObjectiveTo compare different machine learning methods in predicting overall mortality in cirrhosis and to use machine learning to select easily scored clinical variables for a novel cirrhosis prognostic model.Design, setting, and participantsThis prognostic study used a retrospective cohort of adult patients with cirrhosis or its complications seen in 130 hospitals and affiliated ambulatory clinics in the integrated, national Veterans Affairs health care system from October 1, 2011, to September 30, 2015. Patients were followed up through December 31, 2018. Data were analyzed from October 1, 2017, to May 31, 2020.ExposuresPotential predictors included demographic characteristics; liver disease etiology, severity, and complications; use of health care resources; comorbid conditions; and comprehensive laboratory and medication data. Patients were randomly selected for model development (66.7%) and validation (33.3%). Three different statistical and machine learning methods were evaluated: gradient descent boosting, logistic regression with least absolute shrinkage and selection operator (LASSO) regularization, and logistic regression with LASSO constrained to select no more than 10 predictors (partial pathway model). Predictor inclusion and model performance were evaluated in a 5-fold cross-validation. Last, the predictors identified in the most parsimonious (the partial path) model were refit using maximum-likelihood estimation (Cirrhosis Mortality Model [CiMM]), and its predictive performance was compared with that of the widely used Model for End Stage Liver Disease with sodium (MELD-Na) score.Main outcomes and measuresAll-cause mortality.ResultsOf the 107 939 patients with cirrhosis (mean [SD] age, 62.7 [9.6] years; 96.6% male; 66.3% white, 18.4% African American), the annual mortality rate ranged from 8.8% to 15.3%. In total, 32.7% of patients died within 3 years, and 46.2% died within 5 years after the index date. Models predicting 1-year mortality had good discrimination for the gradient descent boosting (area under the receiver operating characteristics curve [AUC], 0.81; 95% CI, 0.80-0.82), logistic regression with LASSO regularization (AUC, 0.78; 95% CI, 0.77-0.79), and the partial path logistic model (AUC, 0.78; 95% CI, 0.76-0.78). All models showed good calibration. The final CiMM model with machine learning-derived clinical variables offered significantly better discrimination than the MELD-Na score, with AUCs of 0.78 (95% CI, 0.77-0.79) vs 0.67 (95% CI, 0.66-0.68) for 1-year mortality, respectively (DeLong z = 17.00; P < .001).Conclusions and relevanceIn this study, simple machine learning techniques performed as well as the more advanced ensemble gradient boosting. Using the clinical variables identified from simple machine learning in a cirrhosis mortality model produced a new score more transparent than machine learning and more predictive than the MELD-Na score.
| S-EPMC7610191 | biostudies-literature
Project description:BackgroundDuring a pandemic, it is important for clinicians to stratify patients and decide who receives limited medical resources. Machine learning models have been proposed to accurately predict COVID-19 disease severity. Previous studies have typically tested only one machine learning algorithm and limited performance evaluation to area under the curve analysis. To obtain the best results possible, it may be important to test different machine learning algorithms to find the best prediction model.ObjectiveIn this study, we aimed to use automated machine learning (autoML) to train various machine learning algorithms. We selected the model that best predicted patients' chances of surviving a SARS-CoV-2 infection. In addition, we identified which variables (ie, vital signs, biomarkers, comorbidities, etc) were the most influential in generating an accurate model.MethodsData were retrospectively collected from all patients who tested positive for COVID-19 at our institution between March 1 and July 3, 2020. We collected 48 variables from each patient within 36 hours before or after the index time (ie, real-time polymerase chain reaction positivity). Patients were followed for 30 days or until death. Patients' data were used to build 20 machine learning models with various algorithms via autoML. The performance of machine learning models was measured by analyzing the area under the precision-recall curve (AUPCR). Subsequently, we established model interpretability via Shapley additive explanation and partial dependence plots to identify and rank variables that drove model predictions. Afterward, we conducted dimensionality reduction to extract the 10 most influential variables. AutoML models were retrained by only using these 10 variables, and the output models were evaluated against the model that used 48 variables.ResultsData from 4313 patients were used to develop the models. The best model that was generated by using autoML and 48 variables was the stacked ensemble model (AUPRC=0.807). The two best independent models were the gradient boost machine and extreme gradient boost models, which had an AUPRC of 0.803 and 0.793, respectively. The deep learning model (AUPRC=0.73) was substantially inferior to the other models. The 10 most influential variables for generating high-performing models were systolic and diastolic blood pressure, age, pulse oximetry level, blood urea nitrogen level, lactate dehydrogenase level, D-dimer level, troponin level, respiratory rate, and Charlson comorbidity score. After the autoML models were retrained with these 10 variables, the stacked ensemble model still had the best performance (AUPRC=0.791).ConclusionsWe used autoML to develop high-performing models that predicted the survival of patients with COVID-19. In addition, we identified important variables that correlated with mortality. This is proof of concept that autoML is an efficient, effective, and informative method for generating machine learning-based clinical decision support tools.
| S-EPMC7919846 | biostudies-literature
Project description:BackgroundCOVID-19 has infected millions of people worldwide and is responsible for several hundred thousand fatalities. The COVID-19 pandemic has necessitated thoughtful resource allocation and early identification of high-risk patients. However, effective methods to meet these needs are lacking.ObjectiveThe aims of this study were to analyze the electronic health records (EHRs) of patients who tested positive for COVID-19 and were admitted to hospitals in the Mount Sinai Health System in New York City; to develop machine learning models for making predictions about the hospital course of the patients over clinically meaningful time horizons based on patient characteristics at admission; and to assess the performance of these models at multiple hospitals and time points.MethodsWe used Extreme Gradient Boosting (XGBoost) and baseline comparator models to predict in-hospital mortality and critical events at time windows of 3, 5, 7, and 10 days from admission. Our study population included harmonized EHR data from five hospitals in New York City for 4098 COVID-19-positive patients admitted from March 15 to May 22, 2020. The models were first trained on patients from a single hospital (n=1514) before or on May 1, externally validated on patients from four other hospitals (n=2201) before or on May 1, and prospectively validated on all patients after May 1 (n=383). Finally, we established model interpretability to identify and rank variables that drive model predictions.ResultsUpon cross-validation, the XGBoost classifier outperformed baseline models, with an area under the receiver operating characteristic curve (AUC-ROC) for mortality of 0.89 at 3 days, 0.85 at 5 and 7 days, and 0.84 at 10 days. XGBoost also performed well for critical event prediction, with an AUC-ROC of 0.80 at 3 days, 0.79 at 5 days, 0.80 at 7 days, and 0.81 at 10 days. In external validation, XGBoost achieved an AUC-ROC of 0.88 at 3 days, 0.86 at 5 days, 0.86 at 7 days, and 0.84 at 10 days for mortality prediction. Similarly, the unimputed XGBoost model achieved an AUC-ROC of 0.78 at 3 days, 0.79 at 5 days, 0.80 at 7 days, and 0.81 at 10 days. Trends in performance on prospective validation sets were similar. At 7 days, acute kidney injury on admission, elevated LDH, tachypnea, and hyperglycemia were the strongest drivers of critical event prediction, while higher age, anion gap, and C-reactive protein were the strongest drivers of mortality prediction.ConclusionsWe externally and prospectively trained and validated machine learning models for mortality and critical events for patients with COVID-19 at different time horizons. These models identified at-risk patients and uncovered underlying relationships that predicted outcomes.
| S-EPMC7652593 | biostudies-literature
Project description:BackgroundEchocardiography (echo) based machine learning (ML) models may be useful in identifying patients at high-risk of all-cause mortality.MethodsWe developed ML models (ResNet deep learning using echo videos and CatBoost gradient boosting using echo measurements) to predict 1-year, 3-year, and 5-year mortality. Models were trained on the Mackay dataset, Taiwan (6083 echos, 3626 patients) and validated in the Alberta HEART dataset, Canada (997 echos, 595 patients). We examined the performance of the models overall, and in subgroups (healthy controls, at risk of heart failure (HF), HF with reduced ejection fraction (HFrEF) and HF with preserved ejection fraction (HFpEF)). We compared the models' performance to the MAGGIC risk score, and examined the correlation between the models' predicted probability of death and baseline quality of life as measured by the Kansas City Cardiomyopathy Questionnaire (KCCQ).FindingsMortality rates at 1-, 3- and 5-years were 14.9%, 28.6%, and 42.5% in the Mackay cohort, and 3.0%, 10.3%, and 18.7%, in the Alberta HEART cohort. The ResNet and CatBoost models achieved area under the receiver-operating curve (AUROC) between 85% and 92% in internal validation. In external validation, the AUROCs for the ResNet (82%, 82%, and 78%) were significantly better than CatBoost (78%, 73%, and 75%), for 1-, 3- and 5-year mortality prediction respectively, with better or comparable performance to the MAGGIC score. ResNet models predicted higher probability of death in the HFpEF and HFrEF (30%-50%) subgroups than in controls and at risk patients (5%-20%). The predicted probabilities of death correlated with KCCQ scores (all p < 0.05).InterpretationEcho-based ML models to predict mortality had good internal and external validity, were generalizable, correlated with patients' quality of life, and are comparable to an established HF risk score. These models can be leveraged for automated risk stratification at point-of-care.FundingFunding for Alberta HEART was provided by an Alberta Innovates - Health Solutions Interdisciplinary Team Grant no. AHFMRITG 200801018. P.K. holds a Canadian Institutes of Health Research (CIHR) Sex and Gender Science Chair and a Heart & Stroke Foundation Chair in Cardiovascular Research. A.V. and V.S. received funding from the Mitacs Globalink Research Internship.
| S-EPMC10006431 | biostudies-literature
Project description:Coronavirus disease 2019 (COVID-19) is a novel disease resulting from infection with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), which has quickly risen since the beginning of 2020 to become a global pandemic. As a result of the rapid growth of COVID-19, hospitals are tasked with managing an increasing volume of these cases with neither a known effective therapy, an existing vaccine, nor well-established guidelines for clinical management. The need for actionable knowledge amidst the COVID-19 pandemic is dire and yet, given the urgency of this illness and the speed with which the healthcare workforce must devise useful policies for its management, there is insufficient time to await the conclusions of detailed, controlled, prospective clinical research. Thus, we present a retrospective study evaluating laboratory data and mortality from patients with positive RT-PCR assay results for SARS-CoV-2. The objective of this study is to identify prognostic serum biomarkers in patients at greatest risk of mortality. To this end, we develop a machine learning model using five serum chemistry laboratory parameters (c-reactive protein, blood urea nitrogen, serum calcium, serum albumin, and lactic acid) from 398 patients (43 expired and 355 non-expired) for the prediction of death up to 48?h prior to patient expiration. The resulting support vector machine model achieved 91% sensitivity and 91% specificity (AUC 0.93) for predicting patient expiration status on held-out testing data. Finally, we examine the impact of each feature and feature combination in light of different model predictions, highlighting important patterns of laboratory values that impact outcomes in SARS-CoV-2 infection.
| S-EPMC7567420 | biostudies-literature
Project description:BackgroundA real-time model for predicting short-term mortality in critically ill patients is needed to identify patients at imminent risk. However, the performance of the model needs to be validated in various clinical settings and ethnicities before its clinical application. In this study, we aim to develop an ensemble machine learning model using routinely measured clinical variables at a single academic institution in South Korea.MethodsWe developed an ensemble model using deep learning and light gradient boosting machine models. Internal validation was performed using the last two years of the internal cohort dataset, collected from a single academic hospital in South Korea between 2007 and 2021. External validation was performed using the full Medical Information Mart for Intensive Care (MIMIC), eICU-Collaborative Research Database (eICU-CRD), and Amsterdam University Medical Center database (AmsterdamUMCdb) data. The area under the receiver operating characteristic curve (AUROC) was calculated and compared to that for the National Early Warning Score (NEWS).ResultsThe developed model (iMORS) demonstrated high predictive performance with an internal AUROC of 0.964 (95% confidence interval [CI] 0.963-0.965) and external AUROCs of 0.890 (95% CI 0.889-0.891) for MIMIC, 0.886 (95% CI 0.885-0.887) for eICU-CRD, and 0.870 (95% CI 0.868-0.873) for AmsterdamUMCdb. The model outperformed the NEWS with higher AUROCs in the internal and external validation (0.866 for the internal, 0.746 for MIMIC, 0.798 for eICU-CRD, and 0.819 for AmsterdamUMCdb; p < 0.001).ConclusionsOur real-time machine learning model to predict short-term mortality in critically ill patients showed excellent performance in both internal and external validations. This model could be a useful decision-support tool in the intensive care units to assist clinicians.
| S-EPMC10938661 | biostudies-literature
Project description:ImportanceThere is an absence of mortality risk assessment tools in first-episode psychosis (FEP) that could enable personalized interventions.ObjectiveTo examine the feasibility of machine learning (ML) in discerning mortality risk in FEP and to assess whether such risk predictions can inform pharmacotherapy choices.Design, setting, and participantsIn this prognostic study, Swedish nationwide cohort data (from July 1, 2006, to December 31, 2021) were harnessed for model development and validation. Finnish cohort data (from January 1, 1998, to December 31, 2017) were used for external validation. Data analyses were completed between December 2022 and December 2023.Main outcomes and measuresFifty-one nationwide register variables, encompassing demographics and clinical and work-related histories, were subjected to ML to predict future mortality risk. The ML model's performance was evaluated by calculating the area under the receiver operating characteristic curve (AUROC). The comparative effectiveness of pharmacotherapies in patients was assessed and was stratified by the ML model to those with predicted high mortality risk (vs low risk), using the between-individual hazard ratio (HR). The 5 most important variables were then identified and a model was retrained using these variables in the discovery sample.ResultsThis study included 24 052 Swedish participants (20 000 in the discovery sample and 4052 in the validation sample) and 1490 Finnish participants (in the validation sample). Swedish participants had a mean (SD) age of 29.1 (8.1) years, 62.1% were men, and 418 died with 2 years. Finnish participants had a mean (SD) age of 29.7 (8.0) years, 61.7% were men, and 31 died within 2 years. The discovery sample achieved an AUROC of 0.71 (95% CI, 0.68-0.74) for 2-year mortality prediction. Using the 5 most important variables (ie, the top 10% [substance use comorbidities, first hospitalization duration due to FEP, male sex, prior somatic hospitalizations, and age]), the final model resulted in an AUROC of 0.70 (95% CI, 0.63-0.76) in the Swedish sample and 0.67 (95% CI, 0.56-0.78) in the Finnish sample. Individuals with predicted high mortality risk had an elevated 15-year risk in the Swedish sample (HR, 3.77 [95% CI, 2.92-4.88]) and an elevated 20-year risk in the Finnish sample (HR, 3.72 [95% CI, 2.67-5.18]). For those with predicted high mortality risk, long-acting injectable antipsychotics (HR, 0.45 [95% CI, 0.23-0.88]) and mood stabilizers (HR, 0.64 [95% CI, 0.46-0.90]) were associated with decreased mortality risk. Conversely, for those predicted to survive, only oral aripiprazole (HR, 0.38 [95% CI, 0.20-0.69]) and risperidone (HR, 0.38 [95% CI, 0.18-0.82]) were associated with decreased mortality risk.Conclusions and relevanceIn this prognostic study, an ML-based model was developed and validated to predict mortality risk in FEP. These findings may help to develop personalized interventions to mitigate mortality risk in FEP.
| S-EPMC10949098 | biostudies-literature
Project description:Early prediction of patient mortality risks during a pandemic can decrease mortality by assuring efficient resource allocation and treatment planning. This study aimed to develop and compare prognosis prediction machine learning models based on invasive laboratory and noninvasive clinical and demographic data from patients' day of admission. Three Support Vector Machine (SVM) models were developed and compared using invasive, non-invasive, and both groups. The results suggested that non-invasive features could provide mortality predictions that are similar to the invasive and roughly on par with the joint model. Feature inspection results from SVM-RFE and sparsity analysis displayed that, compared with the invasive model, the non-invasive model can provide better performances with a fewer number of features, pointing to the presence of high predictive information contents in several non-invasive features, including SPO2, age, and cardiovascular disorders. Furthermore, while the invasive model was able to provide better mortality predictions for the imminent future, non-invasive features displayed better performance for more distant expiration intervals. Early mortality prediction using non-invasive models can give us insights as to where and with whom to intervene. Combined with novel technologies, such as wireless wearable devices, these models can create powerful frameworks for various medical assignments and patient triage.
| S-EPMC8253432 | biostudies-literature
Project description:Predicting risk factors for death in leptospirosis is challenging, and identifying high-risk patients is crucial as it might expedite the start of life-saving supportive care. Admission data of 295 leptospirosis patients were enrolled, and a machine-learning approach was used to fit models in a derivation cohort. The comparison of accuracy metrics was performed with two previous models-SPIRO score and quick SOFA score. A Lasso regression analysis was the selected model, demonstrating the best accuracy to predict mortality in leptospirosis [area under the curve (AUC-ROC) = 0.776]. A score-based prediction was carried out with the coefficients of this model and named LeptoScore. Then, to simplify the predictive tool, a new score was built by attributing points to the predictors with importance values higher than 1. The simplified score, named QuickLepto, has five variables (age > 40 years; lethargy; pulmonary symptom; mean arterial pressure < 80 mmHg and hematocrit < 30%) and good predictive accuracy (AUC-ROC = 0.788). LeptoScore and QuickLepto had better accuracy to predict mortality in patients with leptospirosis when compared to SPIRO score (AUC-ROC = 0.500) and quick SOFA score (AUC-ROC = 0.782). The main result is a new scoring system, the QuickLepto, that is a simple and useful tool to predict death in leptospirosis patients at hospital admission.
| S-EPMC10024714 | biostudies-literature