Dataset Information

Sensitivity and specificity of an algorithm based on medico-administrative data to identify hospitalized patients with major bleeding presenting to an emergency department.

ABSTRACT: BACKGROUND:Validation studies on an ICD-10-based algorithm to identify major bleeding events are scarce, and mostly focused on positive predictive values. OBJECTIVE:To evaluate the sensitivity and specificity of an ICD-10-based algorithm in adult patients referred to hospital. METHODS:This was a cross-sectional, retrospective analysis. Among all hospital stays of adult patients referred to Rennes University Hospital, France, through the emergency ward in 2014, we identified major bleeding events according to an index test based on a list of ICD-10 diagnoses. As a reference, a two-step process was applied: firstly, a computerized request for electronic health records from the emergency ward, using several hemorrhage-related diagnostic codes and specific emergency therapies so as to discard stays with a very low probability of bleeding; secondly, a chart review of selected records was conducted by a medical expert blinded to the index test results and each hospital stay was classified into one of two exclusive categories: major bleeding or no major bleeding, according to pre-specified criteria. RESULTS:Out of 16,012 hospital stays, the reference identified 736 major bleeding events and left 15,276 stays considered as without the target condition. The index test identified 637 bleeding events: 293 intracranial hemorrhages, 197 gastrointestinal hemorrhages and 147 other bleeding events. Overall, sensitivity was 65% (95%CI, 62 to 69), and specificity was 99.0%. We observed differential sensitivity and specificity across bleeding types, with the highest values for intracranial hemorrhage. Positive predictive values ranged from 59% for "other" bleeding events, to 71% (95%CI, 65 to 78) for gastrointestinal hemorrhage, and 96% for intracranial hemorrhage. CONCLUSIONS:Low sensitivity and differential measures of accuracy across bleeding types support the need for specific data collection and medical validation rather than using an ICD-10-based algorithm for assessing the incidence of major bleeding.

SUBMITTER: Oger E

PROVIDER: S-EPMC6798331 | biostudies-literature | 2019 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Sensitivity and specificity of an algorithm based on medico-administrative data to identify hospitalized patients with major bleeding presenting to an emergency department.

Oger Emmanuel E Botrel Marie-Anne MA Juchault Catherine C Bouget Jacques J

BMC medical research methodology 20191018 1

<h4>Background</h4>Validation studies on an ICD-10-based algorithm to identify major bleeding events are scarce, and mostly focused on positive predictive values.<h4>Objective</h4>To evaluate the sensitivity and specificity of an ICD-10-based algorithm in adult patients referred to hospital.<h4>Methods</h4>This was a cross-sectional, retrospective analysis. Among all hospital stays of adult patients referred to Rennes University Hospital, France, through the emergency ward in 2014, we identified ...[more]

PMID: 31627721

Similar Datasets

Project description:Accurate stratification of sepsis can effectively guide the triage of patient care and shared decision making in the emergency department (ED). However, previous research on sepsis identification models focused mainly on ICU patients, and discrepancies in model performance between the development and external validation datasets are rarely evaluated. The aim of our study was to develop and externally validate a machine learning model to stratify sepsis patients in the ED. We retrospectively collected clinical data from two geographically separate institutes that provided a different level of care at different time periods. The Sepsis-3 criteria were used as the reference standard in both datasets for identifying true sepsis cases. An eXtreme Gradient Boosting (XGBoost) algorithm was developed to stratify sepsis patients and the performance of the model was compared with traditional clinical sepsis tools; quick Sequential Organ Failure Assessment (qSOFA) and Systemic Inflammatory Response Syndrome (SIRS). There were 8296 patients (1752 (21%) being septic) in the development and 1744 patients (506 (29%) being septic) in the external validation datasets. The mortality of septic patients in the development and validation datasets was 13.5% and 17%, respectively. In the internal validation, XGBoost achieved an area under the receiver operating characteristic curve (AUROC) of 0.86, exceeding SIRS (0.68) and qSOFA (0.56). The performance of XGBoost deteriorated in the external validation (the AUROC of XGBoost, SIRS and qSOFA was 0.75, 0.57 and 0.66, respectively). Heterogeneity in patient characteristics, such as sepsis prevalence, severity, age, comorbidity and infection focus, could reduce model performance. Our model showed good discriminative capabilities for the identification of sepsis patients and outperformed the existing sepsis identification tools. Implementation of the ML model in the ED can facilitate timely sepsis identification and treatment. However, dataset discrepancies should be carefully evaluated before implementing the ML approach in clinical practice. This finding reinforces the necessity for future studies to perform external validation to ensure the generalisability of any developed ML approaches.

Project description:BackgroundThe emergency department (ED) triage system to classify and prioritize patients from high risk to less urgent continues to be a challenge.ObjectiveThis study, comprising 80,433 patients, aims to develop a machine learning algorithm prediction model of critical care outcomes for adult patients using information collected during ED triage and compare the performance with that of the baseline model using the Korean Triage and Acuity Scale (KTAS).MethodsTo predict the need for critical care, we used 13 predictors from triage information: age, gender, mode of ED arrival, the time interval between onset and ED arrival, reason of ED visit, chief complaints, systolic blood pressure, diastolic blood pressure, pulse rate, respiratory rate, body temperature, oxygen saturation, and level of consciousness. The baseline model with KTAS was developed using logistic regression, and the machine learning model with 13 variables was generated using extreme gradient boosting (XGB) and deep neural network (DNN) algorithms. The discrimination was measured by the area under the receiver operating characteristic (AUROC) curve. The ability of calibration with Hosmer-Lemeshow test and reclassification with net reclassification index were evaluated. The calibration plot and partial dependence plot were used in the analysis.ResultsThe AUROC of the model with the full set of variables (0.833-0.861) was better than that of the baseline model (0.796). The XGB model of AUROC 0.861 (95% CI 0.848-0.874) showed a higher discriminative performance than the DNN model of 0.833 (95% CI 0.819-0.848). The XGB and DNN models proved better reclassification than the baseline model with a positive net reclassification index. The XGB models were well-calibrated (Hosmer-Lemeshow test; P>.05); however, the DNN showed poor calibration power (Hosmer-Lemeshow test; P<.001). We further interpreted the nonlinear association between variables and critical care prediction.ConclusionsOur study demonstrated that the performance of the XGB model using initial information at ED triage for predicting patients in need of critical care outperformed the conventional model with KTAS.

Project description:ObjectivesAdministrative claims data sets are often used for emergency care research and policy investigations of healthcare resource utilization, acute care practices, and evaluation of quality improvement interventions. Despite the high profile of emergency department (ED) visits in analyses using administrative claims, little work has evaluated the degree to which existing definitions based on claims data accurately captures conventionally defined hospital-based ED services. We sought to construct an operational definition for ED visitation using a comprehensive Medicare data set and to compare this definition to existing operational definitions used by researchers and policymakers.MethodsWe examined four operational definitions of an ED visit commonly used by researchers and policymakers using a 20% sample of the 2012 Medicare Chronic Condition Warehouse (CCW) data set. The CCW data set included all Part A (hospital) and Part B (hospital outpatient, physician) claims for a nationally representative sample of continuously enrolled Medicare fee-for-services beneficiaries. Three definitions were based on published research or existing quality metrics including: 1) provider claims-based definition, 2) facility claims-based definition, and 3) CMS Research Data Assistance Center (ResDAC) definition. In addition, we developed a fourth operational definition (Yale definition) that sought to incorporate additional coding rules for identifying ED visits. We report levels of agreement and disagreement among the four definitions.ResultsOf 10,717,786 beneficiaries included in the sample data set, 22% had evidence of ED use during the study year under any of the ED visit definitions. The definition using provider claims identified a total of 4,199,148 ED visits, the facility definition 4,795,057 visits, the ResDAC definition 5,278,980 ED visits, and the Yale definition 5,192,235 ED visits. The Yale definition identified a statistically different (p < 0.05) collection of ED visits than all other definitions including 17% more ED visits than the provider definition and 2% fewer visits than the ResDAC definition. Differences in ED visitation counts between each definition occurred for several reasons including the inclusion of critical care or observation services in the ED, discrepancies between facility and provider billing regulations, and operational decisions of each definition.ConclusionCurrent operational definitions of ED visitation using administrative claims produce different estimates of ED visitation based on the underlying assumptions applied to billing data and data set availability. Future analyses using administrative claims data should seek to validate specific definitions and inform the development of a consistent, consensus ED visitation definitions to standardize research reporting and the interpretation of policy interventions.

Project description:IntroductionThe American Hospital Association (AHA) has hospital-level data, while the Centers for Medicare & Medicaid Services (CMS) has patient-level data. Merging these with other distinct databases would permit analyses of hospital-based specialties, units, or departments, and patient outcomes. One distinct database is the National Emergency Department Inventory (NEDI), which contains information about all EDs in the United States. However, a challenge with merging these databases is that NEDI lists all US EDs individually, while the AHA and CMS group some EDs by hospital network. Consolidating data for this merge may be preferential to excluding grouped EDs. Our objectives were to consolidate ED data to enable linkage with administrative datasets and to determine the effect of excluding grouped EDs on ED-level summary results.MethodsUsing the 2014 NEDI-USA database, we surveyed all New England EDs. We individually matched NEDI EDs with corresponding EDs in the AHA and CMS. A "group match" was assigned when more than one NEDI ED was matched to a single AHA or CMS facility identification number. Within each group, we consolidated individual ED data to create a single observation based on sums or weighted averages of responses as appropriate.ResultsOf the 195 EDs in New England, 169 (87%) completed the NEDI survey. Among these, 130 (77%) EDs were individually listed in AHA and CMS, while 39 were part of groups consisting of 2-3 EDs but represented by one facility ID. Compared to the individually listed EDs, the 39 EDs included in a "group match" had a larger number of annual visits and beds, were more likely to be freestanding, and were less likely to be rural (all P<0.05). Two grouped EDs were excluded because the listed ED did not respond to the NEDI survey; the remaining 37 EDs were consolidated into 19 observations. Thus, the consolidated dataset contained 149 observations representing 171 EDs; this consolidated dataset yielded summary results that were similar to those of the 169 responding EDs.ConclusionExcluding grouped EDs would have resulted in a non-representative dataset. The original vs consolidated NEDI datasets yielded similar results and enabled linkage with large administrative datasets. This approach presents a novel opportunity to use characteristics of hospital-based specialties, units, and departments in studies of patient-level outcomes, to advance health services research.

Dataset Information

Sensitivity and specificity of an algorithm based on medico-administrative data to identify hospitalized patients with major bleeding presenting to an emergency department.

Publications

Sensitivity and specificity of an algorithm based on medico-administrative data to identify hospitalized patients with major bleeding presenting to an emergency department.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets