Dataset Information

Performance of intensive care unit severity scoring systems across different ethnicities in the USA: a retrospective observational study.

ABSTRACT:

Background

Despite wide use of severity scoring systems for case-mix determination and benchmarking in the intensive care unit (ICU), the possibility of scoring bias across ethnicities has not been examined. Guidelines on the use of illness severity scores to inform triage decisions for allocation of scarce resources, such as mechanical ventilation, during the current COVID-19 pandemic warrant examination for possible bias in these models. We investigated the performance of the severity scoring systems Acute Physiology and Chronic Health Evaluation IVa (APACHE IVa), Oxford Acute Severity of Illness Score (OASIS), and Sequential Organ Failure Assessment (SOFA) across four ethnicities in two large ICU databases to identify possible ethnicity-based bias.

Methods

Data from the electronic ICU Collaborative Research Database (eICU-CRD) and the Medical Information Mart for Intensive Care III (MIMIC-III) database, built from patient episodes in the USA from 2014-15 and 2001-12, respectively, were analysed for score performance in Asian, Black, Hispanic, and White people after appropriate exclusions. Hospital mortality was the outcome of interest. Discrimination and calibration were determined for all three scoring systems in all four groups, using area under receiver operating characteristic (AUROC) curve for different ethnicities to assess discrimination, and standardised mortality ratio (SMR) or proxy measures to assess calibration.

Findings

We analysed 166 751 participants (122 919 eICU-CRD and 43 832 MIMIC-III). Although measurements of discrimination were significantly different among the groups (AUROC ranging from 0·86 to 0·89 [p=0·016] with APACHE IVa and from 0·75 to 0·77 [p=0·85] with OASIS), they did not display any discernible systematic patterns of bias. However, measurements of calibration indicated persistent, and in some cases statistically significant, patterns of difference between Hispanic people (SMR 0·73 with APACHE IVa and 0·64 with OASIS) and Black people (0·67 and 0·68) versus Asian people (0·77 and 0·95) and White people (0·76 and 0·81). Although calibrations were imperfect for all groups, the scores consistently showed a pattern of overpredicting mortality for Black people and Hispanic people. Similar results were seen using SOFA scores across the two databases.

Interpretation

The systematic differences in calibration across ethnicities suggest that illness severity scores reflect statistical bias in their predictions of mortality.

Funding

There was no specific funding for this study.

SUBMITTER: Sarkar R

PROVIDER: S-EPMC8063502 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundMultiple scoring systems have been developed for both the intensive care unit (ICU) and the emergency department (ED) to risk stratify patients and predict mortality. However, it remains unclear whether the additional data needed to compute ICU scores improves mortality prediction for critically ill patients compared to the simpler ED scores.MethodsWe studied a prospective observational cohort of 227 critically ill patients admitted to the ICU directly from the ED at an academic, tertiary care medical center. We compared Acute Physiology and Chronic Health Evaluation (APACHE) II, APACHE III, Simplified Acute Physiology Score (SAPS) II, Modified Early Warning Score (MEWS), Rapid Emergency Medicine Score (REMS), Prince of Wales Emergency Department Score (PEDS), and a pre-hospital critical illness prediction score developed by Seymour et al. (JAMA 2010, 304(7):747-754). The primary endpoint was 60-day mortality. We compared the receiver operating characteristic (ROC) curves of the different scores and their calibration using the Hosmer-Lemeshow goodness-of-fit test and visual assessment.ResultsThe ICU scores outperformed the ED scores with higher area under the curve (AUC) values (p?=?0.01). There were no differences in discrimination among the ED-based scoring systems (AUC 0.698 to 0.742; p?=?0.45) or among the ICU-based scoring systems (AUC 0.779 to 0.799; p?=?0.60). With the exception of the Seymour score, the ED-based scoring systems did not discriminate as well as the best-performing ICU-based scoring system, APACHE III (p?=?0.005 to 0.01 for comparison of ED scores to APACHE III). The Seymour score had a superior AUC to other ED scores and, despite a lower AUC than all the ICU scores, was not significantly different than APACHE III (p?=?0.09). When data from the first 24 h in the ICU was used to calculate the ED scores, the AUC for the ED scores improved numerically, but this improvement was not statistically significant. All scores had acceptable calibration.ConclusionsIn contrast to prior studies of patients based in the emergency department, ICU scores outperformed ED scores in critically ill patients admitted from the emergency department. This difference in performance seemed to be primarily due to the complexity of the scores rather than the time window from which the data was derived.

Project description:PurposeTo compare the diagnostic performance and inter-observer agreement of five different CT chest severity scoring systems for COVID-19 to find the most precise one with the least interpretation time.Methods and materialsThis retrospective study included 85 patients (54 male and 31 female) with PCR-confirmed COVID-19. They underwent CT to assess the severity of pulmonary involvement. Three readers were asked to assess the pulmonary abnormalities and score the severity using five different systems, including chest CT severity score (CT-SS), chest CT score, total severity score (TSS), modified total severity score (m-TSS), and 3-level chest CT severity score. Time consumption on reporting of each system was calculated.ResultsTwo hundred fifty-five observations were reported for each system. There was a statistically significant inter-observer agreement in assessing qualitative lung involvement using the m-TSS and the other four quantitative systems. The ROC curves revealed excellent and very good diagnostic accuracy for all systems when cutoff values for detection severe cases were > 22, > 17, > 12, and > 26 for CT-SS, chest CT score, TSS, and 3-level CT severity score. The AUC was very good (0.86), excellent (0.90), very good (0.89), and very good (0.86), respectively. Chest CT score showed the highest specificity (95.2%) in discrimination of severe cases. Time consumption on reporting was significantly different (< 0.001): CT-SS > 3L-CT-SS > chest CT score > TSS.ConclusionAll chest CT severity scoring systems in this study demonstrated excellent inter-observer agreement and reasonable performance to assess COVID-19 in relation to the clinical severity. CT-SS and TSS had the highest specificity and least time for interpretation.Key points• All chest CT severity scoring systems discussed in this study revealed excellent inter-observer agreement and reasonable performance to assess COVID-19 in relation to the clinical severity. • Chest CT scoring system and TSS had the highest specificity. • Both TSS and m-TSS consumed the least time compared to the other three scoring systems.