Dataset Information

Statistical Modeling and Aggregate-Weighted Scoring Systems in Prediction of Mortality and ICU Transfer: A Systematic Review.

ABSTRACT:

Background

The clinical deterioration of patientsin general hospital wards is an important safety issue. Aggregate-weighted early warning systems (EWSs) may not detect risk until patients present with acute decline.

Purpose

We aimed to compare the prognostic test accuracy and clinical workloads generated by EWSs using statistical modeling (multivariable regression or machine learning) versus aggregate-weighted tools.

Data sources

We searched PubMed and CINAHL using terms that described clinical deterioration and use of an advanced EWS.

Study selection

The outcome was clinical deterioration (intensive care unit transfer or death) of adult patients on general hospital wards. We included studies published from January 1, 2012 to September 15, 2018.

Data extraction

Following 2015 PRIMSA systematic review protocol guidelines; 2015 TRIPOD criteria for predictive model evaluation; and the Cochrane Collaboration guidelines, we reported model performance, adjusted positive predictive value (PPV), and conducted simulations of workup-to-detection ratios.

Data synthesis

Of 285 articles, six studies reported the model performance of advanced EWSs, and five were of high quality. All EWSs using statistical modeling identified at-risk patients with greater precision than aggregate-weighted EWSs (mean AUC 0.80 vs 0.73). EWSs using statistical modeling generated 4.9 alerts to find one true positive case versus 7.1 alerts in aggregate-weighted EWSs; a nearly 50% relative workload increase for aggregate-weighted EWSs.

Conclusions

Compared with aggregate-weighted tools, EWSs using statistical modeling consistently demonstrated superior prognostic performance and generated less workload to identify and treat one true positive case. A standardized approach to reporting EWS model performance is needed, including outcome definitions, pretest probability, observed and adjusted PPV, and workup-to-detection ratio.

SUBMITTER: Linnen DT

PROVIDER: S-EPMC6628701 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundMultiple scoring systems have been developed for both the intensive care unit (ICU) and the emergency department (ED) to risk stratify patients and predict mortality. However, it remains unclear whether the additional data needed to compute ICU scores improves mortality prediction for critically ill patients compared to the simpler ED scores.MethodsWe studied a prospective observational cohort of 227 critically ill patients admitted to the ICU directly from the ED at an academic, tertiary care medical center. We compared Acute Physiology and Chronic Health Evaluation (APACHE) II, APACHE III, Simplified Acute Physiology Score (SAPS) II, Modified Early Warning Score (MEWS), Rapid Emergency Medicine Score (REMS), Prince of Wales Emergency Department Score (PEDS), and a pre-hospital critical illness prediction score developed by Seymour et al. (JAMA 2010, 304(7):747-754). The primary endpoint was 60-day mortality. We compared the receiver operating characteristic (ROC) curves of the different scores and their calibration using the Hosmer-Lemeshow goodness-of-fit test and visual assessment.ResultsThe ICU scores outperformed the ED scores with higher area under the curve (AUC) values (p?=?0.01). There were no differences in discrimination among the ED-based scoring systems (AUC 0.698 to 0.742; p?=?0.45) or among the ICU-based scoring systems (AUC 0.779 to 0.799; p?=?0.60). With the exception of the Seymour score, the ED-based scoring systems did not discriminate as well as the best-performing ICU-based scoring system, APACHE III (p?=?0.005 to 0.01 for comparison of ED scores to APACHE III). The Seymour score had a superior AUC to other ED scores and, despite a lower AUC than all the ICU scores, was not significantly different than APACHE III (p?=?0.09). When data from the first 24 h in the ICU was used to calculate the ED scores, the AUC for the ED scores improved numerically, but this improvement was not statistically significant. All scores had acceptable calibration.ConclusionsIn contrast to prior studies of patients based in the emergency department, ICU scores outperformed ED scores in critically ill patients admitted from the emergency department. This difference in performance seemed to be primarily due to the complexity of the scores rather than the time window from which the data was derived.

Project description:ObjectivesRapid and early severity-of-illness assessment appears to be important for critically ill patients with novel coronavirus disease (COVID-19). This study aimed to evaluate the performance of the rapid scoring system on admission of these patients.MethodsA total of 138 medical records of critically ill patients with COVID-19 were included in the study. Demographic and clinical characteristics on admission used for calculating Modified Early Warning Score (MEWS) and Rapid Emergency Medicine Score (REMS) and outcomes (survival or death) were collected for each case and extracted for analysis. All patients were divided into two age subgroups (<65 years and ≥65 years). The receiver operating characteristic (ROC) curve analyses were performed for overall patients and both subgroups.ResultsThe median [25th quartile, 75th quartile] of MEWS of survivors versus nonsurvivors were 1 [1, 2] and 2 [1, 3] and those of REMS were 5 [2, 6] and 7 [6, 10], respectively. In overall analysis, the area under the ROC curve for the REMS in predicting mortality was 0.833 (95% confidence interval [CI] = 0.737 to 0.928), higher than that of MEWS (0.677, 95% CI = 0.541 to 0.813). An optimal cutoff of REMS (≥6) had a sensitivity of 89.5%, a specificity of 69.8%, a positive predictive value of 39.5%, and a negative predictive value of 96.8%. In the analysis of subgroup of patients aged <65 years, the area under the ROC curve for the REMS in predicting mortality was 0.863 (95% CI = 0.743 to 0.941), higher than that of MEWS (0.603, 95% CI = 0.462 to 0.732).ConclusionTo our knowledge, this study was the first exploration on rapid scoring systems for critically ill patients with COVID-19. The REMS could provide emergency clinicians with an effective adjunct risk stratification tool for critically ill patients with COVID-19, especially for the patients aged <65 years. The effectiveness of REMS for screening these patients is attributed to its high negative predictive value.

Project description:BACKGROUND:An accurate injury severity measurement is essential for the evaluation of pediatric trauma care and outcome research. The traditional Injury Severity Score (ISS) does not consider the differential risks of the Abbreviated Injury Scale (AIS) from different body regions nor is it pediatric specific. The objective of this study was to develop a weighted ISS (wISS) system for pediatric blunt trauma patients with better predictive power than ISS. METHODS:Based on the association between mortality and AIS from each of the six ISS body regions, we generated different weights for the component AIS scores used in the calculation of ISS. The weights and wISS were generated using the National Trauma Data Bank. The Nationwide Emergency Department Sample (NEDS) was used to validate our main results. Pediatric blunt trauma patients younger than 16 years were included, and mortality was the outcome. Discrimination (areas under the receiver operating characteristic curve, sensitivity, specificity, positive predictive value, negative predictive value, concordance) and calibration (Hosmer-Lemeshow statistic) were compared between the wISS and ISS. RESULTS:The areas under the receiver operating characteristic curves from the wISS and ISS are 0.88 versus 0.86 in ISS of 1 to 74 and 0.77 versus 0.64 in ISS of 25 to 74 (p < 0.0001). The wISS showed higher specificity, positive predictive value, negative predictive value, and concordance when they were compared at similar levels of sensitivity. The wISS had better calibration (smaller Hosmer-Lemeshow statistic) than the ISS (11.6 vs. 19.7 for ISS = 1-74 and 10.9 vs. 12.6 for ISS = 25-74). The wISS showed even better discrimination with the Nationwide Emergency Department Sample. CONCLUSION:By weighting the AIS from different body regions, the wISS had significantly better predictive power for mortality than the ISS, especially in critically injured children. LEVEL OF EVIDENCE AND STUDY TYPE:Prognostic/epidemiological, level IV.

Project description:BackgroundCoronavirus disease 2019 (COVID-19) has been widely spread and caused tens of thousands of deaths, especially in patients with severe COVID-19. This analysis aimed to explore risk factors for mortality of severe COVID-19, and establish a scoring system to predict in-hospital deaths.MethodsPatients with COVID-19 were retrospectively analyzed and clinical characteristics were compared. LASSO regression as well as multivariable analysis were used to screen variables and establish prediction model.FindingsA total of 2529 patients with COVID-19 was retrospectively analyzed, and 452 eligible severe COVID-19 were used for finally analysis. In training cohort, the median age was 66•0 years while it was 73•0 years in non-survivors. Patients aged 60-75 years accounted for the largest proportion of infected populations and mortality toll. Anti-SARS-CoV-2 antibodies were monitored up to 54 days, and IgG levels reached the highest during 20-30 days. No differences were observed of antibody levels between severe and non-severe patients. About 60.2% of severe patients had complications. Among acute myocardial injury (AMI), acute kidney injury (AKI) and acute liver injury (ALI), the heart was the earliest injured organ, whereas the time from AKI to death was the shortest. Age, diabetes, coronary heart disease (CHD), percentage of lymphocytes (LYM%), procalcitonin (PCT), serum urea, C reactive protein and D-dimer (DD), were identified associated with mortality by LASSO binary logistic regression. Then multivariable analysis was performed to conclude that old age, CHD, LYM%, PCT and DD remained independent risk factors for mortality. Based on the above variables, a scoring system of COVID-19 (CSS) was established to divide patients into low-risk and high-risk groups. This model displayed good discrimination (AUC=0·919) and calibration (P=0·264). Complications in low-risk and high-risk groups were significantly different (P<0·05). Use of corticosteroids in low-risk groups increased hospital stays by 4·5 days (P=0·036) and durations of disease by 7·5 days (P=0·012) compared with no corticosteroids.InterpretationOld age, CHD, LYM%, PCT and DD were independently related to mortality. CSS was useful for predicting in-hospital mortality and complications, and it could help clinicians to identify high-risk patients with poor prognosis.FundingThis work was supported by the Key Project for Anti-2019 novel Coronavirus Pneumonia from the Ministry of Science and Technology, China (grant number 2020YFC0845500).

Project description:BACKGROUND:The Sequential Organ Failure Assessment (SOFA) score is commonly used in ICUs around the world, designed to assess the severity of the patient's clinical state based on function/dysfunction of six major organ systems. The goal of this work is to build a computational model to predict mortality based on a series of SOFA scores. In addition, we examined the possibility of improving the prediction by incorporating a new component designed to measure the performance of the gastrointestinal system, added to the other six components. METHODS:In this retrospective study, we used patients' three latest SOFA scores recorded during an individual ICU stay as input to different machine learning models and ensemble learning models. We added three validated parameters representing gastrointestinal failure. Among others, we used classification models such as Support Vector Machines (SVMs), Neural Networks, Logistic Regression and a penalty function used to increase model robustness in regard to certain extreme cases, which may be found in ICU population. We used the Area under Curve (AUC) performance metric to examine performance. RESULTS:We found an ensemble model of linear and logistic regression achieves a higher AUC compared related works in past years. After incorporating the gastrointestinal failure score along with the penalty function, our best performing ensemble model resulted in an additional improvement in terms of AUC metrics. We implemented and compared 36 different models that were built using both the information from the SOFA score as well as that of the gastrointestinal system. All compared models have approximately similar and relatively large AUC (between 0.8645 and 0.9146) with the best results are achieved by incorporating the gastrointestinal parameters into the prediction models. CONCLUSIONS:Our findings indicate that gastrointestinal parameters carry significant information as a mortality predictor in addition to the conventional SOFA score. This information improves the predictive power of machine learning models by extending the SOFA to include information related to gastrointestinal organ system. The described method improves mortality prediction by considering the dynamics of the extended SOFA score. Although tested on a limited data set, the results' stability across different models suggests robustness in real-time use.

Project description:BackgroundThe Criticality Index-Mortality uses physiology, therapy, and intensity of care to compute mortality risk for pediatric ICU patients. If the frequency of mortality risk computations were increased to every 3 h with model performance that could improve the assessment of severity of illness, it could be utilized to monitor patients for significant mortality risk change.ObjectivesTo assess the performance of a dynamic method of updating mortality risk every 3 h using the Criticality Index-Mortality methodology and identify variables that are significant contributors to mortality risk predictions.PopulationThere were 8,399 pediatric ICU admissions with 312 (3.7%) deaths from January 1, 2018 to February 29, 2020. We randomly selected 75% of patients for training, 13% for validation, and 12% for testing.ModelA neural network was trained to predict hospital survival or death during or following an ICU admission. Variables included age, gender, laboratory tests, vital signs, medications categories, and mechanical ventilation variables. The neural network was calibrated to mortality risk using nonparametric logistic regression.ResultsDiscrimination assessed across all time periods found an AUROC of 0.851 (0.841-0.862) and an AUPRC was 0.443 (0.417-0.467). When assessed for performance every 3 h, the AUROCs had a minimum value of 0.778 (0.689-0.867) and a maximum value of 0.885 (0.841,0.862); the AUPRCs had a minimum value 0.148 (0.058-0.328) and a maximum value of 0.499 (0.229-0.769). The calibration plot had an intercept of 0.011, a slope of 0.956, and the R2 was 0.814. Comparison of observed vs. expected proportion of deaths revealed that 95.8% of the 543 risk intervals were not statistically significantly different. Construct validity assessed by death and survivor risk trajectories analyzed by mortality risk quartiles and 7 high and low risk diseases confirmed a priori clinical expectations about the trajectories of death and survivors.ConclusionsThe Criticality Index-Mortality computing mortality risk every 3 h for pediatric ICU patients has model performance that could enhance the clinical assessment of severity of illness. The overall Criticality Index-Mortality framework was effectively applied to develop an institutionally specific, and clinically relevant model for dynamic risk assessment of pediatric ICU patients.