Multi-center validation of machine learning model for preoperative prediction of postoperative mortality.
Ontology highlight
ABSTRACT:
SUBMITTER: Lee SW
PROVIDER: S-EPMC9276734 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BACKGROUND:Rapid, preoperative identification of patients with the highest risk for medical complications is necessary to ensure that limited infrastructure and human resources are directed towards those most likely to benefit. Existing risk scores either lack specificity at the patient level or utilise the American Society of Anesthesiologists (ASA) physical status classification, which requires a clinician to review the chart. METHODS:We report on the use of machine learning algorithms, specifically random forests, to create a fully automated score that predicts postoperative in-hospital mortality based solely on structured data available at the time of surgery. Electronic health record data from 53 097 surgical patients (2.01% mortality rate) who underwent general anaesthesia between April 1, 2013 and December 10, 2018 in a large US academic medical centre were used to extract 58 preoperative features. RESULTS:Using a random forest classifier we found that automatically obtained preoperative features (area under the curve [AUC] of 0.932, 95% confidence interval [CI] 0.910-0.951) outperforms Preoperative Score to Predict Postoperative Mortality (POSPOM) scores (AUC of 0.660, 95% CI 0.598-0.722), Charlson comorbidity scores (AUC of 0.742, 95% CI 0.658-0.812), and ASA physical status (AUC of 0.866, 95% CI 0.829-0.897). Including the ASA physical status with the preoperative features achieves an AUC of 0.936 (95% CI 0.917-0.955). CONCLUSIONS:This automated score outperforms the ASA physical status score, the Charlson comorbidity score, and the POSPOM score for predicting in-hospital mortality. Additionally, we integrate this score with a previously published postoperative score to demonstrate the extent to which patient risk changes during the perioperative period.
| S-EPMC6883494 | biostudies-literature
Project description:BackgroundAccurate, pragmatic risk stratification for postoperative delirium (POD) is necessary to target preventative resources toward high-risk patients. Machine learning (ML) offers a novel approach to leveraging electronic health record (EHR) data for POD prediction. We sought to develop and internally validate a ML-derived POD risk prediction model using preoperative risk features, and to compare its performance to models developed with traditional logistic regression.MethodsThis was a retrospective analysis of preoperative EHR data from 24,885 adults undergoing a procedure requiring anesthesia care, recovering in the main post-anesthesia care unit, and staying in the hospital at least overnight between December 2016 and December 2019 at either of two hospitals in a tertiary care health system. One hundred fifteen preoperative risk features including demographics, comorbidities, nursing assessments, surgery type, and other preoperative EHR data were used to predict postoperative delirium (POD), defined as any instance of Nursing Delirium Screening Scale ≥2 or positive Confusion Assessment Method for the Intensive Care Unit within the first 7 postoperative days. Two ML models (Neural Network and XGBoost), two traditional logistic regression models ("clinician-guided" and "ML hybrid"), and a previously described delirium risk stratification tool (AWOL-S) were evaluated using the area under the receiver operating characteristic curve (AUC-ROC), sensitivity, specificity, positive likelihood ratio, and positive predictive value. Model calibration was assessed with a calibration curve. Patients with no POD assessments charted or at least 20% of input variables missing were excluded.ResultsPOD incidence was 5.3%. The AUC-ROC for Neural Net was 0.841 [95% CI 0. 816-0.863] and for XGBoost was 0.851 [95% CI 0.827-0.874], which was significantly better than the clinician-guided (AUC-ROC 0.763 [0.734-0.793], p < 0.001) and ML hybrid (AUC-ROC 0.824 [0.800-0.849], p < 0.001) regression models and AWOL-S (AUC-ROC 0.762 [95% CI 0.713-0.812], p < 0.001). Neural Net, XGBoost, and ML hybrid models demonstrated excellent calibration, while calibration of the clinician-guided and AWOL-S models was moderate; they tended to overestimate delirium risk in those already at highest risk.ConclusionUsing pragmatically collected EHR data, two ML models predicted POD in a broad perioperative population with high discrimination. Optimal application of the models would provide automated, real-time delirium risk stratification to improve perioperative management of surgical patients at risk for POD.
| S-EPMC8722098 | biostudies-literature
Project description:BackgroundFungal keratitis (FK) is a leading cause of corneal blindness in developing countries due to poor clinical recognition and laboratory identification. Here, we aimed to identify the distinct clinical signature of FK and develop a diagnostic model to differentiate FK from other types of infectious keratitis.MethodsWe reviewed the electronic health records (EHRs) of all patients with suspected infectious keratitis in Beijing Tongren Hospital from January 2011 to December 2021. Twelve clinical signs of slit-lamp images were assessed by Lasso regression analysis and collinear variables were excluded. Three models based on binary logistic regression, random forest classification, and decision tree classification were trained for FK diagnosis and employed for internal validation. Independent external validation of the models was performed in a cohort of 420 patients from seven different ophthalmic centers to evaluate the accuracy, specificity, and sensitivity in real world.FindingsThree diagnostic models of FK based on binary logistic regression, random forest classification, and decision tree classification were established and internal validation were achieved with the mean AUC of 0.916, 0.920, and 0.859, respectively. The models were well-calibrated by external validation using a prospective cohort including 210 FK and 210 non-FK patients from seven eye centers across China. The diagnostic model with the binary logistic regression algorithm classified the external validation dataset with a sensitivity of 0.907 (0.774, 1.000), specificity 0.899 (0.750, 1.000), accuracy 0.905 (0.805, 1.000), and AUC 0.903 (0.808, 0.998).InterpretationOur model enables rapid identification of FK, which will help ophthalmologists to establish a preliminary diagnosis and to improve the diagnostic accuracy in clinic.FundingThe Open Research Fund from the National Key Research and Development Program of China (2021YFC2301000) and the Open Research Fund from Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, Beijing Tongren Hospital, Beihang University &Capital Medical University (BHTR-KFJJ-202001) supported this study.
| S-EPMC9869416 | biostudies-literature
Project description:Introduction: The incidence of postoperative sepsis is continually increased, while few studies have specifically focused on the risk factors and clinical outcomes associated with the development of sepsis after surgical procedures. The present study aimed to develop a mathematical model for predicting the in-hospital mortality among patients with postoperative sepsis. Materials and Methods: Surgical patients in Medical Information Mart for Intensive Care (MIMIC-III) database who simultaneously fulfilled Sepsis 3.0 and Agency for Healthcare Research and Quality (AHRQ) criteria at ICU admission were incorporated. We employed both extreme gradient boosting (XGBoost) and stepwise logistic regression model to predict the in-hospital mortality among patients with postoperative sepsis. Consequently, the model performance was assessed from the angles of discrimination and calibration. Results: We included 3,713 patients who fulfilled our inclusion criteria, in which 397 (10.7%) patients died during hospitalization, and 3,316 (89.3%) patients survived through discharge. Fluid-electrolyte disturbance, coagulopathy, renal replacement therapy (RRT), urine output, and cardiovascular surgery were important features related to the in-hospital mortality. The XGBoost model had a better performance in both discriminatory ability (c-statistics, 0.835 vs. 0.737 and 0.621, respectively; AUPRC, 0.418 vs. 0.280 and 0.237, respectively) and goodness of fit (visualized by calibration curve) compared to the stepwise logistic regression model and baseline model. Conclusion: XGBoost model has a better performance in predicting hospital mortality among patients with postoperative sepsis in comparison to the stepwise logistic regression model. Machine learning-based algorithm might have significant application in the development of early warning system for septic patients following major operations.
| S-EPMC7438711 | biostudies-literature
Project description:Ascertaining which patients are at highest risk of poor postoperative outcomes could improve care and enhance safety. This study aimed to construct and validate a propensity index for 30-day postoperative mortality. A retrospective cohort study was conducted at Hospital de Clínicas de Porto Alegre, Brazil, over a period of 3 years. A dataset of 13524 patients was used to develop the model and another dataset of 7254 was used to validate it. The primary outcome was 30-day in-hospital mortality. Overall mortality in the development dataset was 2.31% [n = 311; 95% confidence interval: 2.06-2.56%]. Four variables were significantly associated with outcome: age, ASA class, nature of surgery (urgent/emergency vs elective), and surgical severity (major/intermediate/minor). The index with this set of variables to predict mortality in the validation sample (n = 7253) gave an AUROC = 0.9137, 85.2% sensitivity, and 81.7% specificity. This sensitivity cut-off yielded four classes of death probability: class I, <2%; class II, 2-5%; class III, 5-10%; class IV, >10%. Model application showed that, amongst patients in risk class IV, the odds of death were approximately fivefold higher (odds ratio 5.43, 95% confidence interval: 2.82-10.46) in those admitted to intensive care after a period on the regular ward than in those sent to the intensive care unit directly after surgery. The SAMPE (Anaesthesia and Perioperative Medicine Service) model accurately predicted 30-day postoperative mortality. This model allows identification of high-risk patients and could be used as a practical tool for care stratification and rational postoperative allocation of critical care resources.
| S-EPMC5662221 | biostudies-other
Project description:ImportanceA variety of perioperative risk factors are associated with postoperative mortality risk. However, the relative contribution of routinely collected intraoperative clinical parameters to short-term and long-term mortality remains understudied.ObjectiveTo examine the performance of multiple machine learning models with data from different perioperative periods to predict 30-day, 1-year, and 5-year mortality and investigate factors that contribute to these predictions.Design, setting, and participantsIn this prognostic study using prospectively collected data, risk prediction models were developed for short-term and long-term mortality after cardiac surgery. Included participants were adult patients undergoing a first-time valve operation, coronary artery bypass grafting, or a combination of both between 1997 and 2017 in a single center, the University Medical Centre Groningen in the Netherlands. Mortality data were obtained in November 2017. Data analysis took place between February 2020 and August 2021.ExposureCardiac surgery.Main outcomes and measuresPostoperative mortality rates at 30 days, 1 year, and 5 years were the primary outcomes. The area under the receiver operating characteristic curve (AUROC) was used to assess discrimination. The contribution of all preoperative, intraoperative hemodynamic and temperature, and postoperative factors to mortality was investigated using Shapley additive explanations (SHAP) values.ResultsData from 9415 patients who underwent cardiac surgery (median [IQR] age, 68 [60-74] years; 2554 [27.1%] women) were included. Overall mortality rates at 30 days, 1 year, and 5 years were 268 patients (2.8%), 420 patients (4.5%), and 612 patients (6.5%), respectively. Models including preoperative, intraoperative, and postoperative data achieved AUROC values of 0.82 (95% CI, 0.78-0.86), 0.81 (95% CI, 0.77-0.85), and 0.80 (95% CI, 0.75-0.84) for 30-day, 1-year, and 5-year mortality, respectively. Models including only postoperative data performed similarly (30 days: 0.78 [95% CI, 0.73-0.82]; 1 year: 0.79 [95% CI, 0.74-0.83]; 5 years: 0.77 [95% CI, 0.73-0.82]). However, models based on all perioperative data provided less clinically usable predictions, with lower detection rates; for example, postoperative models identified a high-risk group with a 2.8-fold increase in risk for 5-year mortality (4.1 [95% CI, 3.3-5.1]) vs an increase of 11.3 (95% CI, 6.8-18.7) for the high-risk group identified by the full perioperative model. Postoperative markers associated with metabolic dysfunction and decreased kidney function were the main factors contributing to mortality risk.Conclusions and relevanceThis study found that the addition of continuous intraoperative hemodynamic and temperature data to postoperative data was not associated with improved machine learning-based identification of patients at increased risk of short-term and long-term mortality after cardiac operations.
| S-EPMC9606847 | biostudies-literature
Project description:BackgroundPostoperative delirium is a common complication that is distressing. This study aimed to demonstrate a prediction model for delirium.MethodsAmong 203,374undergoing non-cardiac surgery between January 2011 and June 2019 at Samsung Medical Center, 2,865 (1.4%) were diagnosed with postoperative delirium. After comparing performances of machine learning algorithms, we chose variables for a prediction model based on an extreme gradient boosting algorithm. Using the top five variables, we generated a prediction model for delirium and conducted an external validation. The Kaplan-Meier and Cox survival analyses were used to analyse the difference of delirium occurrence in patients classified as a prediction model.ResultsThe top five variables selected for the postoperative delirium prediction model were age, operation duration, physical status classification, male sex, and surgical risk. An optimal probability threshold in this model was estimated to be 0.02. The area under the receiver operating characteristic (AUROC) curve was 0.870 with a 95% confidence interval of 0.855-0.885, and the sensitivity and specificity of the model were 0.76 and 0.84, respectively. In an external validation, the AUROC was 0.867 (0.845-0.877). In the survival analysis, delirium occurred more frequently in the group of patients predicted as delirium using an internal validation dataset (p < 0.001).ConclusionBased on machine learning techniques, we analyzed a prediction model of delirium in patients who underwent non-cardiac surgery. Screening for delirium based on the prediction model could improve postoperative care. The working model is provided online and is available for further verification among other populations.Trial registrationKCT 0006363.
| S-EPMC10161528 | biostudies-literature
Project description:In this study, we aimed to develop and validate a machine learning-based mortality prediction model for hospitalized heat-related illness patients. After 2393 hospitalized patients were extracted from a multicentered heat-related illness registry in Japan, subjects were divided into the training set for development (n = 1516, data from 2014, 2017-2019) and the test set (n = 877, data from 2020) for validation. Twenty-four variables including characteristics of patients, vital signs, and laboratory test data at hospital arrival were trained as predictor features for machine learning. The outcome was death during hospital stay. In validation, the developed machine learning models (logistic regression, support vector machine, random forest, XGBoost) demonstrated favorable performance for outcome prediction with significantly increased values of the area under the precision-recall curve (AUPR) of 0.415 [95% confidence interval (CI) 0.336-0.494], 0.395 [CI 0.318-0.472], 0.426 [CI 0.346-0.506], and 0.528 [CI 0.442-0.614], respectively, compared to that of the conventional acute physiology and chronic health evaluation (APACHE)-II score of 0.287 [CI 0.222-0.351] as a reference standard. The area under the receiver operating characteristic curve (AUROC) values were also high over 0.92 in all models, although there were no statistical differences compared to APACHE-II. This is the first demonstration of the potential of machine learning-based mortality prediction models for heat-related illnesses.
| S-EPMC8096946 | biostudies-literature
Project description:ObjectiveTo compare the performance of machine learning models against the traditionally derived Combined Assessment of Risk Encountered in Surgery (CARES) model and the American Society of Anaesthesiologists-Physical Status (ASA-PS) in the prediction of 30-day postsurgical mortality and need for intensive care unit (ICU) stay >24 hours.BackgroundPrediction of surgical risk preoperatively is important for clinical shared decision-making and planning of health resources such as ICU beds. The current growth of electronic medical records coupled with machine learning presents an opportunity to improve the performance of established risk models.MethodsAll patients aged 18 years and above who underwent noncardiac and nonneurological surgery at Singapore General Hospital (SGH) between 1 January 2012 and 31 October 2016 were included. Patient demographics, comorbidities, preoperative laboratory results, and surgery details were obtained from their electronic medical records. Seventy percent of the observations were randomly selected for training, leaving 30% for testing. Baseline models were CARES and ASA-PS. Candidate models were trained using random forest, adaptive boosting, gradient boosting, and support vector machine. Models were evaluated on area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC).ResultsA total of 90,785 patients were included, of whom 539 (0.6%) died within 30 days and 1264 (1.4%) required ICU admission >24 hours postoperatively. Baseline models achieved high AUROCs despite poor sensitivities by predicting all negative in a predominantly negative dataset. Gradient boosting was the best performing model with AUPRCs of 0.23 and 0.38 for mortality and ICU admission outcomes respectively.ConclusionsMachine learning can be used to improve surgical risk prediction compared to traditional risk calculators. AUPRC should be used to evaluate model predictive performance instead of AUROC when the dataset is imbalanced.
| S-EPMC7668340 | biostudies-literature
Project description:BackgroundWe aimed to develop and validate an automated machine learning (autoML) prediction model for cardiac surgery-associated acute kidney injury (CSA-AKI).MethodsUsing 69 preoperative variables, we developed several models to predict post-operative AKI in adult patients undergoing cardiac surgery. Models included autoML and non-autoML types, including decision tree (DT), random forest (RF), extreme gradient boosting (XGBoost), and artificial neural network (ANN), as well as a logistic regression prediction model. We then compared model performance using area under the receiver operating characteristic curve (AUROC) and assessed model calibration using Brier score on the independent testing dataset.ResultsThe incidence of CSA-AKI was 36%. Stacked ensemble autoML had the highest predictive performance among autoML models, and was chosen for comparison with other non-autoML and multivariable logistic regression models. The autoML had the highest AUROC (0.79), followed by RF (0.78), XGBoost (0.77), multivariable logistic regression (0.77), ANN (0.75), and DT (0.64). The autoML had comparable AUROC with RF and outperformed the other models. The autoML was well-calibrated. The Brier score for autoML, RF, DT, XGBoost, ANN, and multivariable logistic regression was 0.18, 0.18, 0.21, 0.19, 0.19, and 0.18, respectively. We applied SHAP and LIME algorithms to our autoML prediction model to extract an explanation of the variables that drive patient-specific predictions of CSA-AKI.ConclusionWe were able to present a preoperative autoML prediction model for CSA-AKI that provided high predictive performance that was comparable to RF and superior to other ML and multivariable logistic regression models. The novel approaches of the proposed explainable preoperative autoML prediction model for CSA-AKI may guide clinicians in advancing individualized medicine plans for patients under cardiac surgery.
| S-EPMC9656700 | biostudies-literature