Dataset Information

Comparing the predictive ability of a commercial artificial intelligence early warning system with physician judgement for clinical deterioration in hospitalised general internal medicine patients: a prospective observational study.

ABSTRACT: OBJECTIVE:Our study compares physician judgement with an automated early warning system (EWS) for predicting clinical deterioration of hospitalised general internal medicine patients. DESIGN:Prospective observational study of clinical predictions made at the end of the daytime work-shift for an academic general internal medicine floor team compared with the risk assessment from an automated EWS collected at the same time. SETTING:Internal medicine teaching wards at a single tertiary care academic medical centre in the USA. PARTICIPANTS:Intern physicians working on the internal medicine wards and an automated EWS (Rothman Index by PeraHealth). OUTCOME:Clinical deterioration within 24?hours including cardiac or pulmonary arrest, rapid response team activation or unscheduled intensive care unit transfer. RESULTS:We collected predictions for 1874 patient days and saw 35 clinical deteriorations (1.9%). The area under the receiver operating curve (AUROC) for the EWS was 0.73 vs 0.70 for physicians (p=0.571). A linear regression model combining physician and EWS predictions had an AUROC of 0.75, outperforming physicians (p=0.016) and the EWS (p=0.05). CONCLUSIONS:There is no significant difference in the performance of the EWS and physicians in predicting clinical deterioration at 24?hours on an inpatient general medicine ward. A combined model outperformed either alone. The EWS and physicians identify partially overlapping sets of at-risk patients suggesting they rely on different cues or decision rules for their predictions. TRIAL REGISTRATION NUMBER:NCT02648828.

SUBMITTER: Arnold J

PROVIDER: S-EPMC6797436 | biostudies-literature | 2019 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Comparing the predictive ability of a commercial artificial intelligence early warning system with physician judgement for clinical deterioration in hospitalised general internal medicine patients: a prospective observational study.

Arnold Jonathan J Davis Alex A Fischhoff Baruch B Yecies Emmanuelle E Grace Jon J Klobuka Andrew A Mohan Deepika D Hanmer Janel J

BMJ open 20191010 10

<h4>Objective</h4>Our study compares physician judgement with an automated early warning system (EWS) for predicting clinical deterioration of hospitalised general internal medicine patients.<h4>Design</h4>Prospective observational study of clinical predictions made at the end of the daytime work-shift for an academic general internal medicine floor team compared with the risk assessment from an automated EWS collected at the same time.<h4>Setting</h4>Internal medicine teaching wards at a single ...[more]

PMID: 31601602

Similar Datasets

Project description:ObjectiveTo assess (1) how well validated existing paediatric track and trigger tools (PTTT) are for predicting adverse outcomes in hospitalised children, and (2) how effective broader paediatric early warning systems are at reducing adverse outcomes in hospitalised children.DesignSystematic review.Data sourcesBritish Nursing Index, Cumulative Index of Nursing and Allied Health Literature, Cochrane Central Register of Controlled Trials, Database of Abstracts of Reviews of Effectiveness, EMBASE, Health Management Information Centre, Medline, Medline in Process, Scopus and Web of Knowledge searched through May 2018.Eligibility criteriaWe included (1) papers reporting on the development or validation of a PTTT or (2) the implementation of a broader early warning system in paediatric units (age 0-18 years), where adverse outcome metrics were reported. Several study designs were considered.Data extraction and synthesisData extraction was conducted by two independent reviewers using template forms. Studies were quality assessed using a modified Downs and Black rating scale.Results36 validation studies and 30 effectiveness studies were included, with 27 unique PTTT identified. Validation studies were largely retrospective case-control studies or chart reviews, while effectiveness studies were predominantly uncontrolled before-after studies. Metrics of adverse outcomes varied considerably. Some PTTT demonstrated good diagnostic accuracy in retrospective case-control studies (primarily for predicting paediatric intensive care unit transfers), but positive predictive value was consistently low, suggesting potential for alarm fatigue. A small number of effectiveness studies reported significant decreases in mortality, arrests or code calls, but were limited by methodological concerns. Overall, there was limited evidence of paediatric early warning system interventions leading to reductions in deterioration.ConclusionThere are several fundamental methodological limitations in the PTTT literature, and the predominance of single-site studies carried out in specialist centres greatly limits generalisability. With limited evidence of effectiveness, calls to make PTTT mandatory across all paediatric units are not supported by the evidence base.Prospero registration numberCRD42015015326.

Project description:BackgroundTimely identification of patients at a high risk of clinical deterioration is key to prioritizing care, allocating resources effectively, and preventing adverse outcomes. Vital signs-based, aggregate-weighted early warning systems are commonly used to predict the risk of outcomes related to cardiorespiratory instability and sepsis, which are strong predictors of poor outcomes and mortality. Machine learning models, which can incorporate trends and capture relationships among parameters that aggregate-weighted models cannot, have recently been showing promising results.ObjectiveThis study aimed to identify, summarize, and evaluate the available research, current state of utility, and challenges with machine learning-based early warning systems using vital signs to predict the risk of physiological deterioration in acutely ill patients, across acute and ambulatory care settings.MethodsPubMed, CINAHL, Cochrane Library, Web of Science, Embase, and Google Scholar were searched for peer-reviewed, original studies with keywords related to "vital signs," "clinical deterioration," and "machine learning." Included studies used patient vital signs along with demographics and described a machine learning model for predicting an outcome in acute and ambulatory care settings. Data were extracted following PRISMA, TRIPOD, and Cochrane Collaboration guidelines.ResultsWe identified 24 peer-reviewed studies from 417 articles for inclusion; 23 studies were retrospective, while 1 was prospective in nature. Care settings included general wards, intensive care units, emergency departments, step-down units, medical assessment units, postanesthetic wards, and home care. Machine learning models including logistic regression, tree-based methods, kernel-based methods, and neural networks were most commonly used to predict the risk of deterioration. The area under the curve for models ranged from 0.57 to 0.97.ConclusionsIn studies that compared performance, reported results suggest that machine learning-based early warning systems can achieve greater accuracy than aggregate-weighted early warning systems but several areas for further research were identified. While these models have the potential to provide clinical decision support, there is a need for standardized outcome measures to allow for rigorous evaluation of performance across models. Further research needs to address the interpretability of model outputs by clinicians, clinical efficacy of these systems through prospective study design, and their potential impact in different clinical settings.

Project description:Background: Early Warning Scores (EWS), including the National Early Warning Score 2 (NEWS2) and Modified NEWS (NEWS-C), have been recommended for triage decision in patients with COVID-19. However, the effectiveness of these EWS in COVID-19 has not been fully validated. The study aimed to investigate the predictive value of EWS to detect clinical deterioration in patients with COVID-19. Methods: Between February 7, 2020 and February 17, 2020, patients confirmed with COVID-19 were screened for this study. The outcomes were early deterioration of respiratory function (EDRF) and need for intensive respiratory support (IRS) during the treatment process. The EDRF was defined as changes in the respiratory component of the sequential organ failure assessment (SOFA) score at day 3 (ΔSOFAresp = SOFA resp at day 3-SOFAresp on admission), in which the positive value reflects clinical deterioration. The IRS was defined as the use of high flow nasal cannula oxygen therapy, noninvasive or invasive mechanical ventilation. The performances of EWS including NEWS, NEWS 2, NEWS-C, Modified Early Warning Scores (MEWS), Hamilton Early Warning Scores (HEWS), and quick sepsis-related organ failure assessment (qSOFA) for predicting EDRF and IRS were compared using the area under the receiver operating characteristic curve (AUROC). Results: A total of 116 patients were included in this study. Of them, 27 patients (23.3%) developed EDRF and 24 patients (20.7%) required IRS. Among these EWS, NEWS-C was the most accurate scoring system for predicting EDRF [AUROC 0.79 (95% CI, 0.69-0.89)] and IRS [AUROC 0.89 (95% CI, 0.82-0.96)], while NEWS 2 had the lowest accuracy in predicting EDRF [AUROC 0.59 (95% CI, 0.46-0.720)] and IRS [AUROC 0.69 (95% CI, 0.57-0.81)]. A NEWS-C ≥ 9 had a sensitivity of 59.3% and a specificity of 85.4% for predicting EDRF. For predicting IRS, a NEWS-C ≥ 9 had a sensitivity of 75% and a specificity of 88%. Conclusions: The NEWS-C was the most accurate scoring system among common EWS to identify patients with COVID-19 at risk for EDRF and need for IRS. The NEWS-C could be recommended as an early triage tool for patients with COVID-19.

Project description:ObjectivesPrevious studies have looked at National Early Warning Score performance in predicting in-hospital deterioration and death, but data are lacking with respect to patient outcomes following implementation of National Early Warning Score. We sought to determine the effectiveness of National Early Warning Score implementation on predicting and preventing patient deterioration in a clinical setting.DesignRetrospective cohort study.SettingTertiary care academic facility and a community hospital.PatientsPatients 18 years old or older hospitalized from March 1, 2014, to February 28, 2015, during preimplementation of National Early Warning Score to August 1, 2015, to July 31, 2016, after National Early Warning Score was implemented.InterventionsImplementation of National Early Warning Score within the electronic health record and associated best practice alert.Measurements and main resultsIn this study of 85,322 patients (42,402 patients pre-National Early Warning Score and 42,920 patients post-National Early Warning Score implementation), the primary outcome of rate of ICU transfer or death did not change after National Early Warning Score implementation, with adjusted hazard ratio of 0.94 (0.84-1.05) and 0.90 (0.77-1.05) at our academic and community hospital, respectively. In total, 175,357 best practice advisories fired during the study period, with the best practice advisory performing better at the community hospital than the academic at predicting an event within 12 hours 7.4% versus 2.2% of the time, respectively. Retraining National Early Warning Score with newly generated hospital-specific coefficients improved model performance.ConclusionsAt both our academic and community hospital, National Early Warning Score had poor performance characteristics and was generally ignored by frontline nursing staff. As a result, National Early Warning Score implementation had no appreciable impact on defined clinical outcomes. Refitting of the model using site-specific data improved performance and supports validating predictive models on local data.

Project description:ImportanceInpatient clinical deterioration is associated with substantial morbidity and mortality but may be easily missed by clinicians. Early warning scores have been developed to alert clinicians to patients at high risk of clinical deterioration, but there is limited evidence for their effectiveness.ObjectiveTo evaluate the effectiveness of an artificial intelligence deterioration model-enabled intervention to reduce the risk of escalations in care among hospitalized patients using a study design that facilitates stronger causal inference.Design, setting, and participantsThis cohort study used a regression discontinuity design that controlled for confounding and was based on Epic Deterioration Index (EDI; Epic Systems Corporation) prediction model scores. Compared with other observational research, the regression discontinuity design facilitates causal analysis. Hospitalized adults were included from 4 general internal medicine units in 1 academic hospital from January 17, 2021, through November 16, 2022.ExposureAn artificial intelligence deterioration model-enabled intervention, consisting of alerts based on an EDI score threshold with an associated collaborative workflow among nurses and physicians.Main outcomes and measuresThe primary outcome was escalations in care, including rapid response team activation, transfer to the intensive care unit, or cardiopulmonary arrest during hospitalization.ResultsDuring the study, 9938 patients were admitted to 1 of the 4 units, with 963 patients (median [IQR] age, 76.1 [64.2-86.2] years; 498 males [52.3%]) included within the primary regression discontinuity analysis. The median (IQR) Elixhauser Comorbidity Index score in the primary analysis cohort was 10 (0-24). The intervention was associated with a -10.4-percentage point (95% CI, -20.1 to -0.8 percentage points; P = .03) absolute risk reduction in the primary outcome for patients at the EDI score threshold. There was no evidence of a discontinuity in measured confounders at the EDI score threshold.Conclusions and relevanceUsing a regression discontinuity design, this cohort study found that the implementation of an artificial intelligence deterioration model-enabled intervention was associated with a significantly decreased risk of escalations in care among inpatients. These results provide evidence for the effectiveness of this intervention and support its further expansion and testing in other care settings.

Dataset Information

Comparing the predictive ability of a commercial artificial intelligence early warning system with physician judgement for clinical deterioration in hospitalised general internal medicine patients: a prospective observational study.

Publications

Comparing the predictive ability of a commercial artificial intelligence early warning system with physician judgement for clinical deterioration in hospitalised general internal medicine patients: a prospective observational study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets