Dataset Information

Early warning score validation methodologies and performance metrics: a systematic review.

ABSTRACT:

Background

Early warning scores (EWS) have been developed as clinical prognostication tools to identify acutely deteriorating patients. In the past few years, there has been a proliferation of studies that describe the development and validation of novel machine learning-based EWS. Systematic reviews of published studies which focus on evaluating performance of both well-established and novel EWS have shown conflicting conclusions. A possible reason is the heterogeneity in validation methods applied. In this review, we aim to examine the methodologies and metrics used in studies which perform EWS validation.

Methods

A systematic review of all eligible studies from the MEDLINE database and other sources, was performed. Studies were eligible if they performed validation on at least one EWS and reported associations between EWS scores and inpatient mortality, intensive care unit (ICU) transfers, or cardiac arrest (CA) of adults. Two reviewers independently did a full-text review and performed data abstraction by using standardized data-worksheet based on the TRIPOD (Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) checklist. Meta-analysis was not performed due to heterogeneity.

Results

The key differences in validation methodologies identified were (1) validation dataset used, (2) outcomes of interest, (3) case definition, time of EWS use and aggregation methods, and (4) handling of missing values. In terms of case definition, among the 48 eligible studies, 34 used the patient episode case definition while 12 used the observation set case definition, and 2 did the validation using both case definitions. Of those that used the patient episode case definition, 18 studies validated the EWS at a single point of time, mostly using the first recorded observation. The review also found more than 10 different performance metrics reported among the studies.

Conclusions

Methodologies and performance metrics used in studies performing validation on EWS were heterogeneous hence making it difficult to interpret and compare EWS performance. Standardizing EWS validation methodology and reporting can potentially address this issue.

SUBMITTER: Fang AHS

PROVIDER: S-EPMC7301346 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:We sought to validate prognostic scores in coronavirus disease 2019 including National Early Warning Score, Modified Early Warning Score, and age-based modifications, and define their performance characteristics.DesignWe analyzed prospectively collected data from the Adaptive COVID-19 Treatment Trial. National Early Warning Score was collected daily during the trial, Modified Early Warning Score was calculated, and age applied to both scores. We assessed prognostic value for the end points of recovery, mechanical ventilation, and death for score at enrollment, average, and slope of score over the first 48 hours.SettingA multisite international inpatient trial.PatientsA total of 1,062 adult nonpregnant inpatients with severe coronavirus disease 2019 pneumonia.InterventionsAdaptive COVID-19 Treatment Trial 1 randomized participants to receive remdesivir or placebo. The prognostic value of predictive scores was evaluated in both groups separately to assess for differential performance in the setting of remdesivir treatment.Measurements and main resultsFor mortality, baseline National Early Warning Score and Modified Early Warning Score were weakly to moderately prognostic (c-index, 0.60-0.68), and improved with addition of age (c-index, 0.66-0.74). For recovery, baseline National Early Warning Score and Modified Early Warning Score demonstrated somewhat better prognostic ability (c-index, 0.65-0.69); however, National Early Warning Score+age and Modified Early Warning Score+age further improved performance (c-index, 0.68-0.71). For deterioration, baseline National Early Warning Score and Modified Early Warning Score were weakly to moderately prognostic (c-index, 0.59-0.69) and improved with addition of age (c-index, 0.63-0.70). All prognostic performance improvements due to addition of age were significant (p < 0.05).ConclusionsIn the Adaptive COVID-19 Treatment Trial 1 cohort, National Early Warning Score and Modified Early Warning Score demonstrated moderate prognostic performance in patients with severe coronavirus disease 2019, with improvement in predictive ability for National Early Warning Score+age and Modified Early Warning Score+age. Area under receiver operating curve for National Early Warning Score and Modified Early Warning Score improved in patients receiving remdesivir versus placebo early in the pandemic for recovery and mortality. Although these scores are simple and readily obtainable in myriad settings, in our data set, they were insufficiently predictive to completely replace clinical judgment in coronavirus disease 2019 and may serve best as an adjunct to triage, disposition, and resourcing decisions.

Project description:IntroductionSeveral versions of Early Warning Systems (EWS) are used in obstetrics to detect and treat early clinical deterioration to avert morbidity and mortality. EWS can potentially be useful to improve the quality of care and reduce the risk of maternal mortality in resource-limited settings. We conducted a systematic literature review of published obstetric early warning systems, define their predictive accuracy for morbidity and mortality, and their effectiveness in triggering corrective actions and improving health outcomes.MethodsWe systematically searched for primary research articles on obstetric EWS published in peer-reviewed journals between January 1997 and March 2018 in Medline, CINAHL, SCOPUS, Science Direct, and Science Citation Index. We also searched reference lists of relevant articles and websites of professional societies. We included studies that assessed the predictive accuracy of EWS to detect clinical deterioration, or/and their effectiveness in improving clinical outcomes in obstetric inpatients. We excluded studies with a paediatric or non-obstetric adult population. Cross-sectional and qualitative studies were also excluded. We performed a narrative synthesis since the outcomes reported were heterogeneous.ResultsA total of 381 papers were identified, 17 of which met the inclusion criteria. Eleven of the included studies evaluated the predictive accuracy of EWS for obstetric morbidity and mortality, 5 studies assessed the effectiveness of EWS in improving clinical outcomes, while one study addressed both. Sixteen published EWS versions were reviewed, 14 of which included five basic clinical observations (pulse rate, respiratory rate, temperature, blood pressure, and consciousness level). The obstetric EWS identified had very high median (inter-quartile range) sensitivity-89% (72% to 97%) and specificity-85% (67% to 98%) but low median (inter-quartile range) positive predictive values-41% (25% to 74%) for predicting morbidity or ICU admission. Obstetric EWS had a very high accuracy in predicting death (AUROC >0.80) among critically ill obstetric patients. Obstetric EWS improves the frequency of routine vital sign observation, reduces the interval between the recording of specifically defined abnormal clinical observations and corrective clinical actions, and can potentially reduce the severity of obstetric morbidity.ConclusionObstetric EWS are effective in predicting severe morbidity (in general obstetric population) and mortality (in critically ill obstetric patients). EWS can contribute to improved quality of care, prevent progressive obstetric morbidity and improve health outcomes. There is limited evidence of the effectiveness of EWS in reducing maternal death across all settings. Clinical parameters in most obstetric EWS versions are routinely collected in resource-limited settings, therefore implementing EWS may be feasible in such settings.

Project description:BackgroundMany studies have explored the accuracy of the National Early Warning Score 2 (NEWS2) in predicting mortality in prehospital and emergency settings, but their findings are inconsistent. Whether NEWS2 is reliable for the pre-examination and triage of patients in prehospital settings and emergency departments remains debatable. Hence, this study aimed to evaluate the accuracy of NEWS2 in predicting mortality in prehospital settings and emergency departments.MethodsWe searched PubMed, Embase, Cochrane Library, Web of Science, CNKI, Wan Fang Data, Vip Database and SinoMed from the inception of each database to January 2023. The inclusion criteria: (I) patients in the prehospital settings or emergency departments; (II) the NEWS2 for predicting 2-day mortality, 30-day mortality, and in-hospital mortality; (III) sufficient data, such as sensitivity, specificity, overall survival, and deaths, were provided for the study; (IV) the type of study was accuracy prediction study. Two authors independently extracted data, including authors, year of publication, country of origin, study design, sample size, threshold cutoff values of NEWS2, and mortality. The PROBAST was used to assess the risk of bias in the included studies.ResultsThirty studies with 185,835 participants were included. Among the 30 included studies, 13 have a high risk of bias, and 17 have a low risk of bias. The pooled sensitivity, specificity and AUC of 2-day mortality (early mortality), 30-day mortality and in-hospital mortality were 0.81 vs. 0.76 vs. 0.72 (95% CI: 0.61, 0.80), 0.81 vs. 0.69 vs. 0.78 (95% CI: 0.49, 0.93) and 0.88 vs. 0.80 vs. 0.78 (95% CI: 0.74, 0.82), respectively.ConclusionsNEWS2 has excellent sensitivity and specificity in predicting early mortality in patients in the prehospitals setting and emergency departments. Nonetheless, it has poor performance in predicting in-hospital mortality and 30-day mortality. Our findings underpin the use of NEWS2 as a pre-examination and triage tool to predict early death in the prehospital settings and emergency departments. To improve the predictive accuracy, it should be used to monitor patients continuously rather than at a single point-in-time.

Project description:BackgroundModified Early Warning Score (MEWS) and National Early Warning Score (NEWS) are widely used in predicting the mortality and intensive care unit (ICU) admission of critically ill patients. This study was conducted to evaluate and compare the prognostic value of NEWS and MEWS for predicting ICU readmission, mortality, and related outcomes in critically ill patients at the time of ICU discharge.MethodsThis multicenter, prospective, observational study was conducted over a year, from April 2019 to March 2020, in the general ICUs of two university-affiliated hospitals in Northwest Iran. MEWS and NEWS were compared based on the patients' outcomes (including mortality, ICU readmission, time to readmission, discharge type, mechanical ventilation (MV), MV duration, and multiple organ failure after readmission) using the univariable and multivariable binary logistic regression. The receiver operating characteristic (ROC) curve was used to determine the outcome predictability of MEWS and NEWS.ResultsA total of 410 ICU patients were enrolled in this study. According to multivariable logistic regression analysis, both MEWS and NEWS were predictors of ICU readmission, time to readmission, MV status after readmission, MV duration, and multiple organ failure after readmission. The area under the ROC curve (AUC) for predicting mortality was 0.91 (95% CI = 0.88-0.94, P < 0.0001) for the NEWS and 0.88 (95% CI = 0.84-0.91, P < 0.0001) for the MEWS. There was no significant difference between the AUC of the NEWS and the MEWS for predicting mortality (P = 0.082). However, for ICU readmission (0.84 vs. 0.71), time to readmission (0.82 vs. 0.67), MV after readmission (0.83 vs. 0.72), MV duration (0.81 vs. 0.67), and multiple organ failure (0.833 vs. 0.710), the AUCs of MEWS were significantly greater (P < 0.001).ConclusionNational Early Warning Score and MEWS values of >4 demonstrated high sensitivity and specificity in identifying the risk of mortality for the patients' discharge from ICU. However, we found that the MEWS showed superiority over the NEWS score in predicting other outcomes. Eventually, MEWS could be considered an efficient prediction score for morbidity and mortality of critically ill patients.