Dataset Information

External Validation of an Acute Respiratory Distress Syndrome Prediction Model Using Radiology Reports.

ABSTRACT:

Objectives

Acute respiratory distress syndrome is frequently under recognized and associated with increased mortality. Previously, we developed a model that used machine learning and natural language processing of text from radiology reports to identify acute respiratory distress syndrome. The model showed improved performance in diagnosing acute respiratory distress syndrome when compared to a rule-based method. In this study, our objective was to externally validate the natural language processing model in patients from an independent hospital setting.

Design

Secondary analysis of data across five prospective clinical studies.

Setting

An urban, tertiary care, academic hospital.

Patients

Adult patients admitted to the medical ICU and at-risk for acute respiratory distress syndrome.

Interventions

None.

Measurements and main results

The natural language processing model was previously derived and internally validated in burn, trauma, and medical patients at Loyola University Medical Center. Two machine learning models were examined with the following text features from qualifying radiology reports: 1) word representations (n-grams) and 2) standardized clinical named entity mentions mapped from the National Library of Medicine Unified Medical Language System. The models were externally validated in a cohort of 235 patients at the University of Chicago Medicine, among which 110 (47%) were diagnosed with acute respiratory distress syndrome by expert annotation. During external validation, the n-gram model demonstrated good discrimination between acute respiratory distress syndrome and nonacute respiratory distress syndrome patients (C-statistic, 0.78; 95% CI, 0.72-0.84). The n-gram model had a higher discrimination for acute respiratory distress syndrome when compared with the standardized named entity model, although not statistically significant (C-statistic 0.78 vs 0.72; p = 0.09). The most important features in the model had good face validity for acute respiratory distress syndrome characteristics but differences in frequencies did occur between hospital settings.

Conclusions

Our computable phenotype for acute respiratory distress syndrome had good discrimination in external validation and may be used by other health systems for case-identification. Discrepancies in feature representation are likely due to differences in characteristics of the patient cohorts.

SUBMITTER: Mayampurath A

PROVIDER: S-EPMC7872467 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundAcute respiratory distress syndrome (ARDS) is a common, but under-recognised, critical illness syndrome associated with high mortality. An important factor in its under-recognition is the variability in chest radiograph interpretation for ARDS. We sought to train a deep convolutional neural network (CNN) to detect ARDS findings on chest radiographs.MethodsCNNs were pretrained on 595 506 radiographs from two centres to identify common chest findings (eg, opacity and effusion), and then trained on 8072 radiographs annotated for ARDS by multiple physicians using various transfer learning approaches. The best performing CNN was tested on chest radiographs in an internal and external cohort, including a subset reviewed by six physicians, including a chest radiologist and physicians trained in intensive care medicine. Chest radiograph data were acquired from four US hospitals.FindingsIn an internal test set of 1560 chest radiographs from 455 patients with acute hypoxaemic respiratory failure, a CNN could detect ARDS with an area under the receiver operator characteristics curve (AUROC) of 0·92 (95% CI 0·89-0·94). In the subgroup of 413 images reviewed by at least six physicians, its AUROC was 0·93 (95% CI 0·88-0·96), sensitivity 83·0% (95% CI 74·0-91·1), and specificity 88·3% (95% CI 83·1-92·8). Among images with zero of six ARDS annotations (n=155), the median CNN probability was 11%, with six (4%) assigned a probability above 50%. Among images with six of six ARDS annotations (n=27), the median CNN probability was 91%, with two (7%) assigned a probability below 50%. In an external cohort of 958 chest radiographs from 431 patients with sepsis, the AUROC was 0·88 (95% CI 0·85-0·91). When radiographs annotated as equivocal were excluded, the AUROC was 0·93 (0·92-0·95).InterpretationA CNN can be trained to achieve expert physician-level performance in ARDS detection on chest radiographs. Further research is needed to evaluate the use of these algorithms to support real-time identification of ARDS patients to ensure fidelity with evidence-based care or to support ongoing ARDS research.FundingNational Institutes of Health, Department of Defense, and Department of Veterans Affairs.

Project description:BackgroundAcute kidney injury (AKI) can make cases of acute respiratory distress syndrome (ARDS) more complex, and the combination of the two can significantly worsen the prognosis. Our objective is to utilize machine learning (ML) techniques to construct models that can promptly identify the risk of AKI in ARDS patients.MethodWe obtained data regarding ARDS patients from the Medical Information Mart for Intensive Care III (MIMIC-III) and MIMIC-IV databases. Within the MIMIC-III dataset, we developed 11 ML prediction models. By evaluating various metrics, we visualized the importance of its features using Shapley additive explanations (SHAP). We then created a more concise model using fewer variables, and optimized it using hyperparameter optimization (HPO). The model was validated using the MIMIC-IV dataset.ResultA total of 928 ARDS patients without AKI were included in the analysis from the MIMIC-III dataset, and among them, 179 (19.3%) developed AKI after admission to the intensive care unit (ICU). In the MIMIC-IV dataset, there were 653 ARDS patients included in the analysis, and among them, 237 (36.3%) developed AKI. A total of 43 features were used to build the model. Among all models, eXtreme gradient boosting (XGBoost) performed the best. We used the top 10 features to build a compact model with an area under the curve (AUC) of 0.850, which improved to an AUC of 0.865 after the HPO. In extra validation set, XGBoost_HPO achieved an AUC of 0.854. The accuracy, sensitivity, specificity, positive prediction value (PPV), negative prediction value (NPV), and F1 score of the XGBoost_HPO model on the test set are 0.865, 0.813, 0.877, 0.578, 0.957 and 0.675, respectively. On extra validation set, they are 0.724, 0.789, 0.688, 0.590, 0.851, and 0.675, respectively.ConclusionML algorithms, especially XGBoost, are reliable for predicting AKI in ARDS patients. The compact model maintains excellent predictive ability, and the web-based calculator improves clinical convenience. This provides valuable guidance in identifying AKI in ARDS, leading to improved patient outcomes.