Dataset Information

Added value of clinical prediction rules for bacteremia in hemodialysis patients: An external validation study.

ABSTRACT:

Introduction

Having developed a clinical prediction rule (CPR) for bacteremia among hemodialysis (HD) outpatients (BAC-HD score), we performed external validation.

Materials & methods

Data were collected on maintenance HD patients at two Japanese tertiary-care hospitals from January 2013 to December 2015. We enrolled 429 consecutive patients (aged ? 18 y) on maintenance HD who had had two sets of blood cultures drawn on admission to assess for bacteremia. We validated the predictive ability of the CPR using two validation cohorts. Index tests were the BAC-HD score and a CPR developed by Shapiro et al. The outcome was bacteremia, based on the results of the admission blood cultures. For added value, we also measured changes in the area under the receiver operating characteristic curve (AUC) using logistic regression and Net Reclassification Improvement (NRI), in which each CPR was added to the basic model.

Results

In Validation cohort 1 (360 subjects), compared to a Model 1 (Basic Model) AUC of 0.69 (95% confidence interval [95% CI]: 0.59-0.80), the AUC of Model 2 (Basic model + BAC-HD score) and Model 3 (Basic model + Shapiro's score) increased to 0.8 (95% CI: 0.71-0.88) and 0.73 (95% CI: 0.63-0.83), respectively. In validation cohort 2 (96 subjects), compared to a Model 1 AUC of 0.81 (95% CI: 0.68-0.94), the AUCs of Model 2 and Model 3 increased to 0.83 (95% CI: 0.72-0.95) and 0.85 (95% CI: 0.76-0.94), respectively. NRIs on addition of the BAC-HD score and Shapiro's score were 0.3 and 0.06 in Validation cohort 1, and 0.27 and 0.13, respectively, in Validation cohort 2.

Conclusion

Either the BAC-HD score or Shapiro's score may improve the ability to diagnose bacteremia in HD patients. Reclassification was better with the BAC-HD score.

SUBMITTER: Sasaki S

PROVIDER: S-EPMC7899347 | biostudies-literature | 2021

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Added value of clinical prediction rules for bacteremia in hemodialysis patients: An external validation study.

Sasaki Sho S Raita Yoshihiko Y Murakami Minoru M Yamamoto Shungo S Tochitani Kentaro K Hasegawa Takeshi T Fujisaki Kiichiro K Fukuhara Shunichi S

PloS one 20210222 2

<h4>Introduction</h4>Having developed a clinical prediction rule (CPR) for bacteremia among hemodialysis (HD) outpatients (BAC-HD score), we performed external validation.<h4>Materials & methods</h4>Data were collected on maintenance HD patients at two Japanese tertiary-care hospitals from January 2013 to December 2015. We enrolled 429 consecutive patients (aged ≥ 18 y) on maintenance HD who had had two sets of blood cultures drawn on admission to assess for bacteremia. We validated the predicti ...[more]

PMID: 33617601

Similar Datasets

Project description:BackgroundClinical prediction rules (CPRs) should be externally validated by independent researchers. Although there are many cardiovascular CPRs, most have not been externally validated. It is not known why some CPRs are externally validated by independent researchers and others are not.MethodsWe analyzed cardiovascular risk CPRs included in a systematic review. Independent external validations were identified by forward citation searches of derivation studies. Time between the publication of a cardiovascular CPR and the first independent external validation was calculated. We assessed Kaplan-Meier estimates of the probability to have an independent external validation. Using univariable Cox regression, we explored whether characteristics of derivation (design, location, sample size, number of predictors, presentation format, validation in derivation), reporting (participants, predictors, outcomes, performance measure, information for risk calculation), and publication (journal impact factor) are associated with time to the first independent external validation.ResultsOf 125 cardiovascular risk CPRs, 29 had an independent external validation. The median follow-up was 118 months (95% CI, 99-130). The 25th percentile of event time was 122 months (95% CI, 91-299). Cardiovascular risk CPRs from the USA were 4.15 times (95% CI, 1.89-9.13) more likely to have an independent external validation. Increasing the sample size of derivation by ten times was associated with a 2.32-fold (95% CI, 1.37-3.91) increase in the probability of having an independent external validation. CPRs presented with an internal validation tend to get an independent external validation sooner (HR = 1.73, 95% CI, 0.77-3.93). CPRs reporting all the information necessary for calculating individual risk were 2.65 (95% CI, 1.01-6.96) times more likely to have an independent external validation. Publishing a cardiovascular risk CPR in a journal that has one unit higher impact factor was associated with a 6% (95% CI, 3-9) higher likelihood of an independent external validation.ConclusionsThe probability for cardiovascular risk CPRs to get an independent external validation was low even many years after their derivations. Authors of new cardiovascular risk CPRs should consider using adequate sample size, conducting an internal validation, and reporting all the information needed for risk calculation as these features were associated with an independent external validation.

Project description:IntroductionExternal validation studies are essential to study the generalizability of prediction models. Recently a permutation test, focusing on discrimination as quantified by the c-statistic, was proposed to judge whether a prediction model is transportable to a new setting. We aimed to evaluate this test and compare it to previously proposed procedures to judge any changes in c-statistic from development to external validation setting.MethodsWe compared the use of the permutation test to the use of benchmark values of the c-statistic following from a previously proposed framework to judge transportability of a prediction model. In a simulation study we developed a prediction model with logistic regression on a development set and validated them in the validation set. We concentrated on two scenarios: 1) the case-mix was more heterogeneous and predictor effects were weaker in the validation set compared to the development set, and 2) the case-mix was less heterogeneous in the validation set and predictor effects were identical in the validation and development set. Furthermore we illustrated the methods in a case study using 15 datasets of patients suffering from traumatic brain injury.ResultsThe permutation test indicated that the validation and development set were homogenous in scenario 1 (in almost all simulated samples) and heterogeneous in scenario 2 (in 17%-39% of simulated samples). Previously proposed benchmark values of the c-statistic and the standard deviation of the linear predictors correctly pointed at the more heterogeneous case-mix in scenario 1 and the less heterogeneous case-mix in scenario 2.ConclusionThe recently proposed permutation test may provide misleading results when externally validating prediction models in the presence of case-mix differences between the development and validation population. To correctly interpret the c-statistic found at external validation it is crucial to disentangle case-mix differences from incorrect regression coefficients.

Dataset Information

Added value of clinical prediction rules for bacteremia in hemodialysis patients: An external validation study.

Introduction

Materials & methods

Results

Conclusion

Publications

Added value of clinical prediction rules for bacteremia in hemodialysis patients: An external validation study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure