Dataset Information

Performance of universal early warning scores in different patient subgroups and clinical settings: a systematic review

ABSTRACT:

Objective

To assess predictive performance of universal early warning scores (EWS) in disease subgroups and clinical settings.

Design

Systematic review.

Data sources

Medline, CINAHL, Embase and Cochrane database of systematic reviews from 1997 to 2019.

Inclusion criteria

Randomised trials and observational studies of internal or external validation of EWS to predict deterioration (mortality, intensive care unit (ICU) transfer and cardiac arrest) in disease subgroups or clinical settings.

Results

We identified 770 studies, of which 103 were included. Study designs and methods were inconsistent, with significant risk of bias (high: n=16 and unclear: n=64 and low risk: n=28). There were only two randomised trials. There was a high degree of heterogeneity in all subgroups and in national early warning score (I2=72%–99%). Predictive accuracy (mean area under the curve; 95% CI) was highest in medical (0.74; 0.74 to 0.75) and surgical (0.77; 0.75 to 0.80) settings and respiratory diseases (0.77; 0.75 to 0.80). Few studies evaluated EWS in specific diseases, for example, cardiology (n=1) and respiratory (n=7). Mortality and ICU transfer were most frequently studied outcomes, and cardiac arrest was least examined (n=8). Integration with electronic health records was uncommon (n=9).

Conclusion

Methodology and quality of validation studies of EWS are insufficient to recommend their use in all diseases and all clinical settings despite good performance of EWS in some subgroups. There is urgent need for consistency in methods and study design, following consensus guidelines for predictive risk scores. Further research should consider specific diseases and settings, using electronic health record data, prior to large-scale implementation.

PROSPERO registration number

PROSPERO CRD42019143141.

SUBMITTER: Alhmoud B

PROVIDER: S-EPMC8039269 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundEarly warning scores (EWS) have been developed as clinical prognostication tools to identify acutely deteriorating patients. In the past few years, there has been a proliferation of studies that describe the development and validation of novel machine learning-based EWS. Systematic reviews of published studies which focus on evaluating performance of both well-established and novel EWS have shown conflicting conclusions. A possible reason is the heterogeneity in validation methods applied. In this review, we aim to examine the methodologies and metrics used in studies which perform EWS validation.MethodsA systematic review of all eligible studies from the MEDLINE database and other sources, was performed. Studies were eligible if they performed validation on at least one EWS and reported associations between EWS scores and inpatient mortality, intensive care unit (ICU) transfers, or cardiac arrest (CA) of adults. Two reviewers independently did a full-text review and performed data abstraction by using standardized data-worksheet based on the TRIPOD (Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) checklist. Meta-analysis was not performed due to heterogeneity.ResultsThe key differences in validation methodologies identified were (1) validation dataset used, (2) outcomes of interest, (3) case definition, time of EWS use and aggregation methods, and (4) handling of missing values. In terms of case definition, among the 48 eligible studies, 34 used the patient episode case definition while 12 used the observation set case definition, and 2 did the validation using both case definitions. Of those that used the patient episode case definition, 18 studies validated the EWS at a single point of time, mostly using the first recorded observation. The review also found more than 10 different performance metrics reported among the studies.ConclusionsMethodologies and performance metrics used in studies performing validation on EWS were heterogeneous hence making it difficult to interpret and compare EWS performance. Standardizing EWS validation methodology and reporting can potentially address this issue.

Project description:Early warning scores (EWS) are widely used in well-resourced healthcare settings to identify patients at risk of mortality. The Modified Early Warning Score (MEWS) is a well-known EWS used comprehensively in the United Kingdom. The HOTEL score (Hypotension, Oxygen saturation, Temperature, ECG abnormality, Loss of independence) was developed and tested in a European cohort; however, its validity is unknown in resource limited settings. This study compared the performance of both scores and suggested modifications to enhance accuracy.A prospective cohort study of adults (?18 yrs) admitted to medical wards at a Malawian hospital. Primary outcome was mortality within three days. Performance of MEWS and HOTEL were assessed using ROC analysis. Logistic regression analysis identified important predictors of mortality and from this a new score was defined.Three-hundred-and-two patients were included. Fifty-one (16.9%) died within three days of admission. With a cut-point ?2, the HOTEL score had sensitivity 70.6% (95% CI: 56.2 to 82.5) and specificity 59.4% (95% CI: 53.0 to 65.5), and was superior to MEWS (cut-point ?5); sensitivity: 58.8% (95% CI: 44.2 to 72.4), specificity: 56.2% (95% CI: 49.8 to 62.4). The new score, dubbed TOTAL (Tachypnoea, Oxygen saturation, Temperature, Alert, Loss of independence), showed slight improvement with a cut-point ?2; sensitivity 76.5% (95% CI: 62.5 to 87.2) and specificity 67.3% (95% CI: 61.1 to 73.1).Using an EWS generated in developed healthcare systems in resource limited settings results in loss of sensitivity and specificity. A score based on predictors of mortality specific to the Malawian population showed enhanced accuracy but not enough to warrant clinical use. Despite an assumption of common physiological responses, disease and population differences seem to strongly determine the performance of EWS. Local validation and impact assessment of these scores should precede their adoption in resource limited settings.

Project description:ObjectivesAdministrative and clinical efforts to improve hospital mortality and intensive care utilization commonly focus on patient rescue, where deteriorating patients are systematically identified and intervened upon. Patient rescue is known to depend on hospital context inclusive of technologic environment, structural features, and hospital organizational behavioral features. With widespread adoption of electronic medical records, early warning score (EWS) systems, which assign points to clinical data elements, are increasingly promoted as a tool for timely patient rescue by referencing their prediction of patient deterioration. We describe the extent to which EWS intervention studies describe the hospital environment of the intervention-details that would be critical for hospital leaders attempting to determine the real-world utility of EWSs in their own hospitals.Data sourcesWe searched CINAHL, PubMed, and Scopus databases for English language EWS implementation research published between 2009 and 2021 in adult medical-surgical inpatients.Study selectionStudies including pediatric, obstetric, psychiatric, prehospital, outpatient, step-down, or ICU patients were excluded.Data extractionTwo investigators independently reviewed titles/abstracts for eligibility based on prespecified exclusion criteria.Data synthesisWe identified 1,434 studies for title/abstract screening. In all, 352 studies underwent full-text review and 21 studies were summarized. The 21 studies (18 before-and-after, three randomized trials) detailed 1,107,883 patients across 54 hospitals. Twelve reported the staff composition of an EWS response team. Ten reported the proportion of surgical patients. One reported nursing ratios; none reported intensive care staffing with in-house critical-care physicians. None measured changes in bed utilization or availability. While 16 qualitatively described resources for education/technologic implementation, none estimated costs. None described workforce composition such as team stability or culture of safety in the hospitals.ConclusionsDespite hundreds of EWS-related publications, most do not report details of hospital context that would inform decisions about real-world EWS adoption. To make informed decisions about whether EWS implementation improves hospital quality, decision-makers may require alternatives such as peer networks and implementation pilots nested within local health systems.

Project description:ObjectivePaediatric Early Warning Scores (PEWS) are widely used in the UK, but the heterogeneity across tools and the limited data on their predictive performance represent obstacles to improving best practice. The standardisation of practice through the proposed National PEWS will rely on robust validation. Therefore, we compared the performance of the National PEWS with six other PEWS currently used in NHS hospitals, for their ability to predict critical care (CC) admission in febrile children attending the emergency department (ED).DesignRetrospective single-centre cohort study.SettingTertiary hospital paediatric ED.ParticipantsA total of 11 449 eligible febrile ED attendances were identified from the electronic patient record over a 2-year period. Seven PEWS scores were calculated (Alder Hey, Bedside, Bristol, National, Newcastle and Scotland PEWS, and the Paediatric Observation Priority Score, using the worst observations recorded during their ED stay.OutcomesThe primary outcome was CC admission within 48 hours, the secondary outcomes were hospital length of stay (LOS) >48 hours and sepsis-related mortality.ResultsOf 11 449 febrile children, 134 (1.2%) were admitted to CC within 48 hours of ED presentation, 606 (5.3%) had a hospital LOS >48 hours. 10 (0.09%) children died, 5 (0.04%) were sepsis-related. All seven PEWS demonstrated excellent discrimination for CC admission (range area under the receiver operating characteristic curves (AUC) 0.91-0.95) and sepsis-related mortality (range AUC 0.95-0.99), most demonstrated moderate discrimination for hospital LOS (range AUC 0.69-0.75). In CC admission threshold analyses, bedside PEWS (AUC 0.90; 95% CI 0.86 to 0.93) and National PEWS (AUC 0.90; 0.87-0.93) were the most discriminative, both at a threshold of ≥6.ConclusionsOur results support the use of the proposed National PEWS in the paediatric ED for the recognition of suspected sepsis to improve outcomes, but further validation is required in other settings and presentations.

Project description:ObjectiveTo provide an overview and critical appraisal of early warning scores for adult hospital patients.DesignSystematic review.Data sourcesMedline, CINAHL, PsycInfo, and Embase until June 2019.Eligibility criteria for study selectionStudies describing the development or external validation of an early warning score for adult hospital inpatients.Results13 171 references were screened and 95 articles were included in the review. 11 studies were development only, 23 were development and external validation, and 61 were external validation only. Most early warning scores were developed for use in the United States (n=13/34, 38%) and the United Kingdom (n=10/34, 29%). Death was the most frequent prediction outcome for development studies (n=10/23, 44%) and validation studies (n=66/84, 79%), with different time horizons (the most frequent was 24 hours). The most common predictors were respiratory rate (n=30/34, 88%), heart rate (n=28/34, 83%), oxygen saturation, temperature, and systolic blood pressure (all n=24/34, 71%). Age (n=13/34, 38%) and sex (n=3/34, 9%) were less frequently included. Key details of the analysis populations were often not reported in development studies (n=12/29, 41%) or validation studies (n=33/84, 39%). Small sample sizes and insufficient numbers of event patients were common in model development and external validation studies. Missing data were often discarded, with just one study using multiple imputation. Only nine of the early warning scores that were developed were presented in sufficient detail to allow individualised risk prediction. Internal validation was carried out in 19 studies, but recommended approaches such as bootstrapping or cross validation were rarely used (n=4/19, 22%). Model performance was frequently assessed using discrimination (development n=18/22, 82%; validation n=69/84, 82%), while calibration was seldom assessed (validation n=13/84, 15%). All included studies were rated at high risk of bias.ConclusionsEarly warning scores are widely used prediction models that are often mandated in daily clinical practice to identify early clinical deterioration in hospital patients. However, many early warning scores in clinical use were found to have methodological weaknesses. Early warning scores might not perform as well as expected and therefore they could have a detrimental effect on patient care. Future work should focus on following recommended approaches for developing and evaluating early warning scores, and investigating the impact and safety of using these scores in clinical practice.Systematic review registrationPROSPERO CRD42017053324.

Project description:ImportanceEarly warning scores (EWSs) are designed for in-hospital use but are widely used in the prehospital field, especially in select groups of patients potentially at high risk. To be useful for paramedics in daily prehospital clinical practice, evaluations are needed of the predictive value of EWSs based on first measured vital signs on scene in large cohorts covering unselected patients using ambulance services.ObjectiveTo validate EWSs' ability to predict mortality and intensive care unit (ICU) stay in an unselected cohort of adult patients who used ambulances.Design, setting, and participantsThis prognostic study conducted a validation based on a cohort of adult patients (aged ≥18 years) who used ambulances in the North Denmark Region from July 1, 2016, to December 31, 2020. EWSs (National Early Warning Score 2 [NEWS2], modified NEWS score without temperature [mNEWS], Quick Sepsis Related Organ Failure Assessment [qSOFA], Rapid Emergency Triage and Treatment System [RETTS], and Danish Emergency Process Triage [DEPT]) were calculated using first vital signs measured by ambulance personnel. Data were analyzed from September 2022 through May 2023.Main outcomes and measuresThe primary outcome was 30-day-mortality. Secondary outcomes were 1-day-mortality and ICU admission. Discrimination was assessed using area under the receiver operating characteristic curve (AUROC) and area under the precision recall curve (AUPRC).ResultsThere were 107 569 unique patients (52 650 females [48.9%]; median [IQR] age, 65 [45-77] years) from the entire cohort of 219 323 patients who used ambulance services, among whom 119 992 patients (54.7%) had called the Danish national emergency number. NEWS2, mNEWS, RETTS, and DEPT performed similarly concerning 30-day mortality (AUROC range, 0.67 [95% CI, 0.66-0.68] for DEPT to 0.68 [95% CI, 0.68-0.69] for mNEWS), while qSOFA had lower performance (AUROC, 0.59 [95% CI, 0.59-0.60]; P vs other scores < .001). All EWSs had low AUPRCs, ranging from 0.09 (95% CI, 0.09-0.09) for qSOFA to 0.14 (95% CI, 0.13-0.14) for mNEWS.. Concerning 1-day mortality and ICU admission NEWS2, mNEWS, RETTS, and DEPT performed similarly, with AUROCs ranging from 0.72 (95% CI, 0.71-0.73) for RETTS to 0.75 (95% CI, 0.74-0.76) for DEPT in 1-day mortality and 0.66 (95% CI, 0.65-0.67) for RETTS to 0.68 (95% CI, 0.67-0.69) for mNEWS in ICU admission, and all EWSs had low AUPRCs. These ranged from 0.02 (95% CI, 0.02-0.03) for qSOFA to 0.04 (95% CI, 0.04-0.04) for DEPT in 1-day mortality and 0.03 (95% CI, 0.03-0.03) for qSOFA to 0.05 (95% CI, 0.04-0.05) for DEPT in ICU admission.Conclusions and relevanceThis study found that EWSs in daily clinical use in emergency medical settings performed moderately in the prehospital field among unselected patients who used ambulances when assessed based on initial measurements of vital signs. These findings suggest the need of appropriate triage and early identification of patients at low and high risk with new and better EWSs also suitable for prehospital use.