Dataset Information

Performance of intensive care unit severity scoring systems across different ethnicities in the USA: a retrospective observational study.

ABSTRACT:

Background

Despite wide use of severity scoring systems for case-mix determination and benchmarking in the intensive care unit (ICU), the possibility of scoring bias across ethnicities has not been examined. Guidelines on the use of illness severity scores to inform triage decisions for allocation of scarce resources, such as mechanical ventilation, during the current COVID-19 pandemic warrant examination for possible bias in these models. We investigated the performance of the severity scoring systems Acute Physiology and Chronic Health Evaluation IVa (APACHE IVa), Oxford Acute Severity of Illness Score (OASIS), and Sequential Organ Failure Assessment (SOFA) across four ethnicities in two large ICU databases to identify possible ethnicity-based bias.

Methods

Data from the electronic ICU Collaborative Research Database (eICU-CRD) and the Medical Information Mart for Intensive Care III (MIMIC-III) database, built from patient episodes in the USA from 2014-15 and 2001-12, respectively, were analysed for score performance in Asian, Black, Hispanic, and White people after appropriate exclusions. Hospital mortality was the outcome of interest. Discrimination and calibration were determined for all three scoring systems in all four groups, using area under receiver operating characteristic (AUROC) curve for different ethnicities to assess discrimination, and standardised mortality ratio (SMR) or proxy measures to assess calibration.

Findings

We analysed 166 751 participants (122 919 eICU-CRD and 43 832 MIMIC-III). Although measurements of discrimination were significantly different among the groups (AUROC ranging from 0·86 to 0·89 [p=0·016] with APACHE IVa and from 0·75 to 0·77 [p=0·85] with OASIS), they did not display any discernible systematic patterns of bias. However, measurements of calibration indicated persistent, and in some cases statistically significant, patterns of difference between Hispanic people (SMR 0·73 with APACHE IVa and 0·64 with OASIS) and Black people (0·67 and 0·68) versus Asian people (0·77 and 0·95) and White people (0·76 and 0·81). Although calibrations were imperfect for all groups, the scores consistently showed a pattern of overpredicting mortality for Black people and Hispanic people. Similar results were seen using SOFA scores across the two databases.

Interpretation

The systematic differences in calibration across ethnicities suggest that illness severity scores reflect statistical bias in their predictions of mortality.

Funding

There was no specific funding for this study.

SUBMITTER: Sarkar R

PROVIDER: S-EPMC8063502 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundMultiple scoring systems have been developed for both the intensive care unit (ICU) and the emergency department (ED) to risk stratify patients and predict mortality. However, it remains unclear whether the additional data needed to compute ICU scores improves mortality prediction for critically ill patients compared to the simpler ED scores.MethodsWe studied a prospective observational cohort of 227 critically ill patients admitted to the ICU directly from the ED at an academic, tertiary care medical center. We compared Acute Physiology and Chronic Health Evaluation (APACHE) II, APACHE III, Simplified Acute Physiology Score (SAPS) II, Modified Early Warning Score (MEWS), Rapid Emergency Medicine Score (REMS), Prince of Wales Emergency Department Score (PEDS), and a pre-hospital critical illness prediction score developed by Seymour et al. (JAMA 2010, 304(7):747-754). The primary endpoint was 60-day mortality. We compared the receiver operating characteristic (ROC) curves of the different scores and their calibration using the Hosmer-Lemeshow goodness-of-fit test and visual assessment.ResultsThe ICU scores outperformed the ED scores with higher area under the curve (AUC) values (p = 0.01). There were no differences in discrimination among the ED-based scoring systems (AUC 0.698 to 0.742; p = 0.45) or among the ICU-based scoring systems (AUC 0.779 to 0.799; p = 0.60). With the exception of the Seymour score, the ED-based scoring systems did not discriminate as well as the best-performing ICU-based scoring system, APACHE III (p = 0.005 to 0.01 for comparison of ED scores to APACHE III). The Seymour score had a superior AUC to other ED scores and, despite a lower AUC than all the ICU scores, was not significantly different than APACHE III (p = 0.09). When data from the first 24 h in the ICU was used to calculate the ED scores, the AUC for the ED scores improved numerically, but this improvement was not statistically significant. All scores had acceptable calibration.ConclusionsIn contrast to prior studies of patients based in the emergency department, ICU scores outperformed ED scores in critically ill patients admitted from the emergency department. This difference in performance seemed to be primarily due to the complexity of the scores rather than the time window from which the data was derived.

Project description:BackgroundAeromonas species, Gram-negative, non-sporulating, facultative, and anaerobic bacilli, widely distributed in aquatic environments, derive various infections, including bacteremia. Most of these infections were opportunistic and found in patients with predisposing conditions. Among the infections, bacteremia remains with notable mortality, reported from 15% to 45%. However, predicting systems for assessing the mortality risk of this disease have yet to be investigated. We aimed to validate the performance of specific predictive scoring systems to assess the clinical outcomes of Aeromonas bacteremia and applied the revised systems to predict mortality risk.MethodsA retrospective observational study reviewed patients with bacteremia caused by Aeromonas spp. based on at least one positive blood culture sample collected in the emergency department from January 2012 to December 2020. The outcome was in-hospital mortality. We used seven predictive scoring systems to predict the clinical outcome. According to the effectiveness in predicting mortality, we revised three of the seven predictive scoring systems by specific characteristics to refine their risk-predicting performances.ResultsWe enrolled 165 patients with bacteremia caused by Aeromonas spp., including 121 males (73.3%) and 44 females (26.7%), with a mean age of 66.1 ± 14.9 years and an average length of hospital stay of 12.4 ± 10.9 days. The overall mortality rate was 32.7% (54/165). The non-survivors had significantly higher scores in MEDS (6.7 ± 4.2 vs. 12.2 ± 3.3, p < 0.001), NEWS (4.0 ± 2.8 vs. 5.3 ± 3.0, p = 0.008), and qSOFA (0.3 ± 0.6 vs. 0.6 ± 0.7, p = 0.007). Regarding mortality risk prediction, the MEDS demonstrated the best predictive power with AUC of ROC measured up to 0.834, followed by NEWS (0.626) and qSOFA (0.608). We revised the MEDS, NEWS, and qSOFA by hemoglobin and lactate. We found that the revised scores had better powerful performance, including 0.859, 0.767, and 0.691 of the AUC of ROC, if the revised MEDS ≥10, revised NEWS ≥8, and revised qSOFA ≥2, respectively.ConclusionsMEDS, NEWS, and qSOFA were good tools for predicting outcomes in patients with Aeromonas spp. bacteremia. The revised MEDS, NEWS, and qSOFA demonstrated more powerful predicting performance than the original scoring systems. We suggested that patients with higher scores in revised MEDS (≥10), revised NEWS (≥8), and revised qSOFA (≥2) received early goal-directed therapy and appropriate broad-spectrum antibiotic treatment as early as possible to reduce mortality.