Dataset Information

Development and Validation of an Explainable Machine Learning Model for Major Complications After Cytoreductive Surgery.

ABSTRACT:

Importance

Cytoreductive surgery (CRS) is one of the most complex operations in surgical oncology with significant morbidity, and improved risk prediction tools are critically needed. Machine learning models can potentially overcome the limitations of traditional multiple logistic regression (MLR) models and provide accurate risk estimates.

Objective

To develop and validate an explainable machine learning model for predicting major postoperative complications in patients undergoing CRS.

Design, setting, and participants

This prognostic study used patient data from tertiary care hospitals with expertise in CRS included in the US Hyperthermic Intraperitoneal Chemotherapy Collaborative Database between 1998 and 2018. Information from 147 variables was extracted to predict the risk of a major complication. An ensemble-based machine learning (gradient-boosting) model was optimized on 80% of the sample with subsequent validation on a 20% holdout data set. The machine learning model was compared with traditional MLR models. The artificial intelligence SHAP (Shapley additive explanations) method was used for interpretation of patient- and cohort-level risk estimates and interactions to define novel surgical risk phenotypes. Data were analyzed between November 2019 and August 2021.

Exposures

Cytoreductive surgery.

Main outcomes and measures

Area under the receiver operating characteristics (AUROC); area under the precision recall curve (AUPRC).

Results

Data from a total 2372 patients were included in model development (mean age, 55 years [range, 11-95 years]; 1366 [57.6%] women). The optimized machine learning model achieved high discrimination (AUROC: mean cross-validation, 0.75 [range, 0.73-0.81]; test, 0.74) and precision (AUPRC: mean cross-validation, 0.50 [range, 0.46-0.58]; test, 0.42). Compared with the optimized machine learning model, the published MLR model performed worse (test AUROC and AUPRC: 0.54 and 0.18, respectively). Higher volume of estimated blood loss, having pelvic peritonectomy, and longer operative time were the top 3 contributors to the high likelihood of major complications. SHAP dependence plots demonstrated insightful nonlinear interactive associations between predictors and major complications. For instance, high estimated blood loss (ie, above 500 mL) was only detrimental when operative time exceeded 9 hours. Unsupervised clustering of patients based on similarity of sources of risk allowed identification of 6 distinct surgical risk phenotypes.

Conclusions and relevance

In this prognostic study using data from patients undergoing CRS, an optimized machine learning model demonstrated a superior ability to predict individual- and cohort-level risk of major complications vs traditional methods. Using the SHAP method, 6 distinct surgical phenotypes were identified based on sources of risk of major complications.

SUBMITTER: Deng H

PROVIDER: S-EPMC9133947 | biostudies-literature | 2022 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Development and Validation of an Explainable Machine Learning Model for Major Complications After Cytoreductive Surgery.

Deng Huiyu H Eftekhari Zahra Z Carlin Cameron C Veerapong Jula J Fournier Keith F KF Johnston Fabian M FM Dineen Sean P SP Powers Benjamin D BD Hendrix Ryan R Lambert Laura A LA Abbott Daniel E DE Vande Walle Kara K Grotz Travis E TE Patel Sameer H SH Clarke Callisia N CN Staley Charles A CA Abdel-Misih Sherif S Cloyd Jordan M JM Lee Byrne B Fong Yuman Y Raoof Mustafa M

JAMA network open 20220502 5

<h4>Importance</h4>Cytoreductive surgery (CRS) is one of the most complex operations in surgical oncology with significant morbidity, and improved risk prediction tools are critically needed. Machine learning models can potentially overcome the limitations of traditional multiple logistic regression (MLR) models and provide accurate risk estimates.<h4>Objective</h4>To develop and validate an explainable machine learning model for predicting major postoperative complications in patients undergoin ...[more]

PMID: 35612856

Similar Datasets

Project description:BackgroundMyocardial injury after noncardiac surgery (MINS) is an easily overlooked complication but closely related to postoperative cardiovascular adverse outcomes; therefore, the early diagnosis and prediction are particularly important.ObjectiveWe aimed to develop and validate an explainable machine learning (ML) model for predicting MINS among older patients undergoing noncardiac surgery.MethodsThe retrospective cohort study included older patients who had noncardiac surgery from 1 northern center and 1 southern center in China. The data sets from center 1 were divided into a training set and an internal validation set. The data set from center 2 was used as an external validation set. Before modeling, the least absolute shrinkage and selection operator and recursive feature elimination methods were used to reduce dimensions of data and select key features from all variables. Prediction models were developed based on the extracted features using several ML algorithms, including category boosting, random forest, logistic regression, naïve Bayes, light gradient boosting machine, extreme gradient boosting, support vector machine, and decision tree. Prediction performance was assessed by the area under the receiver operating characteristic (AUROC) curve as the main evaluation metric to select the best algorithms. The model performance was verified by internal and external validation data sets with the best algorithm and compared to the Revised Cardiac Risk Index. The Shapley Additive Explanations (SHAP) method was applied to calculate values for each feature, representing the contribution to the predicted risk of complication, and generate personalized explanations.ResultsA total of 19,463 eligible patients were included; among those, 12,464 patients in center 1 were included as the training set; 4754 patients in center 1 were included as the internal validation set; and 2245 in center 2 were included as the external validation set. The best-performing model for prediction was the CatBoost algorithm, achieving the highest AUROC of 0.805 (95% CI 0.778-0.831) in the training set, validating with an AUROC of 0.780 in the internal validation set and 0.70 in external validation set. Additionally, CatBoost demonstrated superior performance compared to the Revised Cardiac Risk Index (AUROC 0.636; P<.001). The SHAP values indicated the ranking of the level of importance of each variable, with preoperative serum creatinine concentration, red blood cell distribution width, and age accounting for the top three. The results from the SHAP method can predict events with positive values or nonevents with negative values, providing an explicit explanation of individualized risk predictions.ConclusionsThe ML models can provide a personalized and fairly accurate risk prediction of MINS, and the explainable perspective can help identify potentially modifiable sources of risk at the patient level.

Project description:ObjectiveWe aimed to develop and validate a predictive machine learning (ML) model for cardiac surgery associated with acute kidney injury (CSA-AKI) based on a multicenter randomized control trial (RCT) and a Medical Information Mart for Intensive Care-IV (MIMIC-IV) dataset.MethodsThis was a subanalysis from a completed RCT approved by the Ethics Committee of Fuwai Hospital in Beijing, China (NCT03782350). Data from Fuwai Hospital were randomly assigned, with 80% for the training dataset and 20% for the testing dataset. The data from three other centers were used for the external validation dataset. Furthermore, the MIMIC-IV dataset was also utilized to validate the performance of the predictive model. The area under the receiver operating characteristic curve (ROC-AUC), the precision-recall curve (PR-AUC), and the calibration brier score were applied to evaluate the performance of the traditional logistic regression (LR) and eleven ML algorithms. Additionally, the Shapley Additive Explanations (SHAP) interpreter was used to explain the potential risk factors for CSA-AKI.ResultA total of 6495 eligible patients undergoing cardiopulmonary bypass (CPB) were eventually included in this study, 2416 of whom were from Fuwai Hospital (Beijing), for model development, 562 from three other cardiac centers in China, and 3517 from the MIMICIV dataset, were used, respectively, for external validation. The CatBoostClassifier algorithms outperformed other models, with excellent discrimination and calibration performance for the development, as well as the MIMIC-IV, datasets. In addition, the CatBoostClassifier achieved ROC-AUCs of 0.85, 0.67, and 0.77 and brier scores of 0.14, 0.19, and 0.16 in the testing, external, and MIMIC-IV datasets, respectively. Moreover, the utmost important risk factor, the N-terminal brain sodium peptide (NT-proBNP), was confirmed by the LASSO method in the feature section process. Notably, the SHAP explainer identified that the preoperative blood urea nitrogen level, prothrombin time, serum creatinine level, total bilirubin level, and age were positively correlated with CSA-AKI; preoperative platelets level, systolic and diastolic blood pressure, albumin level, and body weight were negatively associated with CSA-AKI.ConclusionsThe CatBoostClassifier algorithms outperformed other ML models in the discrimination and calibration of CSA-AKI prediction cardiac surgery with CPB, based on a multicenter RCT and MIMIC-IV dataset. Moreover, the preoperative NT-proBNP level was confirmed to be strongly related to CSA-AKI.

Project description:Cytoreductive surgery with hyperthermic intraperitoneal chemotherapy (CRS-HIPEC) has an emerging role in the treatment of peritoneal malignancies. The CRS-HIPEC approach has known treatment-related toxicities. This study sought to determine the predictors of major postoperative complications after CRS-HIPEC in a high-volume center.From a single-institution database, this study investigated complications experienced by patients undergoing CRS-HIPEC. Multiple preoperative and operative factors were analyzed for their ability to predict 60-day Clavien grade 3 and greater (major) complications by logistic regression. A predictive model was created from preoperative factors using multivariate logistic regression. The model was tested by Akaike's information criterion, the Hosmer and Lemeshow Goodness-of-Fit Test, the receiver operating characteristic, and the Youden Index.The study evaluated 247 patients undergoing CRS-HIPEC. The primary tumor site was the appendix in 166 cases (67.2 %), the colorectal area in 51 cases (20.6 %), the peritoneum (mesothelioma) in 22 cases (8.9 %), the ovary in 5 cases (2 %), and the small bowel in 3 cases (1.2 %). The median peritoneal cancer index was 14 (range 0-29), and 235 patients (95.1 %) had a complete (CC-0/1) cytoreduction. Major complications occurred for 41 patients (16.6 %), classified as grade 3 in 33 cases (13.4 %), grade 4 in 5 cases (2 %), and grade 5 (deaths) in 3 cases (1.2 %). The factors predictive of major complications in the multivariate analysis were a Charlson Comorbidity Index (CCI) score higher than 0 [odds ratio (OR), 2.505; p = 0.035], presence of preoperative symptoms (OR 1.951; p = 0.064), and prior resection status [no resection or prior CRS-HIPEC (OR 2.087) vs. prior resection without CRS-HIPEC (OR 3.209); p = 0.046]. These variables were used to create a tool predictive of postoperative complications.Presence of symptoms, CCI, and prior resection status predict major complications and define a low-risk population after CRS-HIPEC.

Project description:BackgroundSkeletal muscle loss during treatment is associated with poor survival outcomes in patients with ovarian cancer. Although changes in muscle mass can be assessed on computed tomography (CT) scans, this labour-intensive process can impair its utility in clinical practice. This study aimed to develop a machine learning (ML) model to predict muscle loss based on clinical data and to interpret the ML model by applying SHapley Additive exPlanations (SHAP) method.MethodsThis study included the data of 617 patients with ovarian cancer who underwent primary debulking surgery and platinum-based chemotherapy at a tertiary centre between 2010 and 2019. The cohort data were split into training and test sets based on the treatment time. External validation was performed using 140 patients from a different tertiary centre. The skeletal muscle index (SMI) was measured from pre- and post-treatment CT scans, and a decrease in SMI ≥ 5% was defined as muscle loss. We evaluated five ML models to predict muscle loss, and their performance was determined using the area under the receiver operating characteristic curve (AUC) and F1 score. The features for analysis included demographic and disease-specific characteristics and relative changes in body mass index (BMI), albumin, neutrophil-to-lymphocyte ratio (NLR), and platelet-to-lymphocyte ratio (PLR). The SHAP method was applied to determine the importance of the features and interpret the ML models.ResultsThe median (inter-quartile range) age of the cohort was 52 (46-59) years. After treatment, 204 patients (33.1%) experienced muscle loss in the training and test datasets, while 44 (31.4%) patients experienced muscle loss in the external validation dataset. Among the five evaluated ML models, the random forest model achieved the highest AUC (0.856, 95% confidence interval: 0.854-0.859) and F1 score (0.726, 95% confidence interval: 0.722-0.730). In the external validation, the random forest model outperformed all ML models with an AUC of 0.874 and an F1 score of 0.741. The results of the SHAP method showed that the albumin change, BMI change, malignant ascites, NLR change, and PLR change were the most important factors in muscle loss. At the patient level, SHAP force plots demonstrated insightful interpretation of our random forest model to predict muscle loss.ConclusionsExplainable ML model was developed using clinical data to identify patients experiencing muscle loss after treatment and provide information of feature contribution. Using the SHAP method, clinicians may better understand the contributors to muscle loss and target interventions to counteract muscle loss.

Dataset Information

Development and Validation of an Explainable Machine Learning Model for Major Complications After Cytoreductive Surgery.

Importance

Objective

Design, setting, and participants

Exposures

Main outcomes and measures

Results

Conclusions and relevance

Publications

Development and Validation of an Explainable Machine Learning Model for Major Complications After Cytoreductive Surgery.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets