Dataset Information

Applying Machine Learning Across Sites: External Validation of a Surgical Site Infection Detection Algorithm.

ABSTRACT:

Background

Surgical complications have tremendous consequences and costs. Complication detection is important for quality improvement, but traditional manual chart review is burdensome. Automated mechanisms are needed to make this more efficient. To understand the generalizability of a machine learning algorithm between sites, automated surgical site infection (SSI) detection algorithms developed at one center were tested at another distinct center.

Study design

NSQIP patients had electronic health record (EHR) data extracted at one center (University of Minnesota Medical Center, Site A) over a 4-year period for model development and internal validation, and at a second center (University of California San Francisco, Site B) over a subsequent 2-year period for external validation. Models for automated NSQIP SSI detection of superficial, organ space, and total SSI within 30 days postoperatively were validated using area under the curve (AUC) scores and corresponding 95% confidence intervals.

Results

For the 8,883 patients (Site A) and 1,473 patients (Site B), AUC scores were not statistically different for any outcome including superficial (external 0.804, internal [0.784, 0.874] AUC); organ/space (external 0.905, internal [0.867, 0.941] AUC); and total (external 0.855, internal [0.854, 0.908] AUC) SSI. False negative rates decreased with increasing case review volume and would be amenable to a strategy in which cases with low predicted probabilities of SSI could be excluded from chart review.

Conclusions

Our findings demonstrated that SSI detection machine learning algorithms developed at 1 site were generalizable to another institution. SSI detection models are practically applicable to accelerate and focus chart review.

SUBMITTER: Zhu Y

PROVIDER: S-EPMC8679130 | biostudies-literature | 2021 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Applying Machine Learning Across Sites: External Validation of a Surgical Site Infection Detection Algorithm.

Zhu Ying Y Simon Gyorgy J GJ Wick Elizabeth C EC Abe-Jones Yumiko Y Najafi Nader N Sheka Adam A Tourani Roshan R Skube Steven J SJ Hu Zhen Z Melton Genevieve B GB

Journal of the American College of Surgeons 20210405 6

<h4>Background</h4>Surgical complications have tremendous consequences and costs. Complication detection is important for quality improvement, but traditional manual chart review is burdensome. Automated mechanisms are needed to make this more efficient. To understand the generalizability of a machine learning algorithm between sites, automated surgical site infection (SSI) detection algorithms developed at one center were tested at another distinct center.<h4>Study design</h4>NSQIP patients had ...[more]

PMID: 33831539

Similar Datasets

Project description:PurposeExternal validation of machine learning predictive models is achieved through evaluation of model performance on different groups of patients than were used for algorithm development. This important step is uncommonly performed, inhibiting clinical translation of newly developed models. Machine learning analysis of the Norwegian Knee Ligament Register (NKLR) recently led to the development of a tool capable of estimating the risk of anterior cruciate ligament (ACL) revision ( https://swastvedt.shinyapps.io/calculator_rev/ ). The purpose of this study was to determine the external validity of the NKLR model by assessing algorithm performance when applied to patients from the Danish Knee Ligament Registry (DKLR).MethodsThe primary outcome measure of the NKLR model was probability of revision ACL reconstruction within 1, 2, and/or 5 years. For external validation, all DKLR patients with complete data for the five variables required for NKLR prediction were included. The five variables included graft choice, femur fixation device, KOOS QOL score at surgery, years from injury to surgery, and age at surgery. Predicted revision probabilities were calculated for all DKLR patients. The model performance was assessed using the same metrics as the NKLR study: concordance and calibration.ResultsIn total, 10,922 DKLR patients were included for analysis. Average follow-up time or time-to-revision was 8.4 (± 4.3) years and overall revision rate was 6.9%. Surgical technique trends (i.e., graft choice and fixation devices) and injury characteristics (i.e., concomitant meniscus and cartilage pathology) were dissimilar between registries. The model produced similar concordance when applied to the DKLR population compared to the original NKLR test data (DKLR: 0.68; NKLR: 0.68-0.69). Calibration was poorer for the DKLR population at one and five years post primary surgery but similar to the NKLR at two years.ConclusionThe NKLR machine learning algorithm demonstrated similar performance when applied to patients from the DKLR, suggesting that it is valid for application outside of the initial patient population. This represents the first machine learning model for predicting revision ACL reconstruction that has been externally validated. Clinicians can use this in-clinic calculator to estimate revision risk at a patient specific level when discussing outcome expectations pre-operatively. While encouraging, it should be noted that the performance of the model on patients undergoing ACL reconstruction outside of Scandinavia remains unknown.Level of evidenceIII.

Project description:BackgroundPhysiotherapy is a critical element in the successful conservative management of low back pain (LBP). A gold standard for quantitatively measuring physiotherapy participation is crucial to understanding physiotherapy adherence in managing recovery from LBP.ObjectiveThis study aimed to develop and evaluate a system with wearable inertial sensors to objectively detect the performance of unsupervised exercises for LBP comprising movement in multiple planes and sitting postures.MethodsA quantitative classification design was used within a machine learning framework to detect exercise performance and posture in a cohort of healthy participants. A set of 8 inertial sensors were placed on the participants, and data were acquired as they performed 7 McKenzie low back exercises and 3 sitting posture positions. Engineered time series features were extracted from the data and used to train 9 models by using a 6-fold cross-validation approach, from which the best 2 models were selected for further study. In addition, a convolutional neural network was trained directly on the time series data. A feature importance analysis was performed to identify sensor locations and channels that contributed the most to the models. Finally, a subset of sensor locations and channels was included in a hyperparameter grid search to identify the optimal sensor configuration and best performing algorithms for exercise and posture classification. The final models were evaluated using the F1 score in a 10-fold cross-validation approach.ResultsIn total, 19 healthy adults with no history of LBP each completed at least one full session of exercises and postures. Random forest and XGBoost (extreme gradient boosting) models performed the best out of the initial set of 9 engineered feature models. The optimal hardware configuration was identified as a 3-sensor setup-lower back, left thigh, and right ankle sensors with acceleration, gyroscope, and magnetometer channels. The XGBoost model achieved the highest exercise (F1 score: mean 0.94, SD 0.03) and posture (F1 score: mean 0.90, SD 0.11) classification scores. The convolutional neural network achieved similar results with the same sensor locations, using only the accelerometer and gyroscope channels for exercise classification (F1 score: mean 0.94, SD 0.02) and the accelerometer channel alone for posture classification (F1 score: mean 0.88, SD 0.07).ConclusionsThis study demonstrates the potential of a 3-sensor lower body wearable solution (eg, smart pants) that can identify exercises in multiple planes and proper sitting postures, which is suitable for the treatment of LBP. This technology has the potential to improve the effectiveness of LBP rehabilitation by facilitating quantitative feedback, early problem diagnosis, and possible remote monitoring.

Project description:AimsEchocardiographic strain imaging reflects myocardial deformation and is a sensitive measure of cardiac function and wall-motion abnormalities. Deep learning (DL) algorithms could automate the interpretation of echocardiographic strain imaging.Methods and resultsWe developed and trained an automated DL-based algorithm for left ventricular (LV) strain measurements in an internal dataset. Global longitudinal strain (GLS) was validated externally in (i) a real-world Taiwanese cohort of participants with and without heart failure (HF), (ii) a core-lab measured dataset from the multinational prevalence of microvascular dysfunction-HF and preserved ejection fraction (PROMIS-HFpEF) study, and regional strain in (iii) the HMC-QU-MI study of patients with suspected myocardial infarction. Outcomes included measures of agreement [bias, mean absolute difference (MAD), root-mean-squared-error (RMSE), and Pearson's correlation (R)] and area under the curve (AUC) to identify HF and regional wall-motion abnormalities. The DL workflow successfully analysed 3741 (89%) studies in the Taiwanese cohort, 176 (96%) in PROMIS-HFpEF, and 158 (98%) in HMC-QU-MI. Automated GLS showed good agreement with manual measurements (mean ± SD): -18.9 ± 4.5% vs. -18.2 ± 4.4%, respectively, bias 0.68 ± 2.52%, MAD 2.0 ± 1.67, RMSE = 2.61, R = 0.84 in the Taiwanese cohort; and -15.4 ± 4.1% vs. -15.9 ± 3.6%, respectively, bias -0.65 ± 2.71%, MAD 2.19 ± 1.71, RMSE = 2.78, R = 0.76 in PROMIS-HFpEF. In the Taiwanese cohort, automated GLS accurately identified patients with HF (AUC = 0.89 for total HF and AUC = 0.98 for HF with reduced ejection fraction). In HMC-QU-MI, automated regional strain identified regional wall-motion abnormalities with an average AUC = 0.80.ConclusionDL algorithms can interpret echocardiographic strain images with similar accuracy as conventional measurements. These results highlight the potential of DL algorithms to democratize the use of cardiac strain measurements and reduce time-spent and costs for echo labs globally.

Project description:BackgroundStep count is an intuitive measure of physical activity frequently quantified in a range of health-related studies; however, accurate quantification of step count can be difficult in the free-living environment, with step counting error routinely above 20% in both consumer and research-grade wrist-worn devices. This study aims to describe the development and validation of step count derived from a wrist-worn accelerometer and to assess its association with cardiovascular and all-cause mortality in a large prospective cohort study.MethodsWe developed and externally validated a hybrid step detection model that involves self-supervised machine learning, trained on a new ground truth annotated, free-living step count dataset (OxWalk, n=39, aged 19-81) and tested against other open-source step counting algorithms. This model was applied to ascertain daily step counts from raw wrist-worn accelerometer data of 75,493 UK Biobank participants without a prior history of cardiovascular disease (CVD) or cancer. Cox regression was used to obtain hazard ratios and 95% confidence intervals for the association of daily step count with fatal CVD and all-cause mortality after adjustment for potential confounders.FindingsThe novel step algorithm demonstrated a mean absolute percent error of 12.5% in free-living validation, detecting 98.7% of true steps and substantially outperforming other recent wrist-worn, open-source algorithms. Our data are indicative of an inverse dose-response association, where, for example, taking 6,596 to 8,474 steps per day was associated with a 39% [24-52%] and 27% [16-36%] lower risk of fatal CVD and all-cause mortality, respectively, compared to those taking fewer steps each day.InterpretationAn accurate measure of step count was ascertained using a machine learning pipeline that demonstrates state-of-the-art accuracy in internal and external validation. The expected associations with CVD and all-cause mortality indicate excellent face validity. This algorithm can be used widely for other studies that have utilised wrist-worn accelerometers and an open-source pipeline is provided to facilitate implementation.

Dataset Information

Applying Machine Learning Across Sites: External Validation of a Surgical Site Infection Detection Algorithm.

Background

Study design

Results

Conclusions

Publications

Applying Machine Learning Across Sites: External Validation of a Surgical Site Infection Detection Algorithm.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets