Dataset Information

Clinical prediction models for serious infections in children: external validation in ambulatory care.

ABSTRACT:

Background

Early distinction between mild and serious infections (SI) is challenging in children in ambulatory care. Clinical prediction models (CPMs), developed to aid physicians in clinical decision-making, require broad external validation before clinical use. We aimed to externally validate four CPMs, developed in emergency departments, in ambulatory care.

Methods

We applied the CPMs in a prospective cohort of acutely ill children presenting to general practices, outpatient paediatric practices or emergency departments in Flanders, Belgium. For two multinomial regression models, Feverkidstool and Craig model, discriminative ability and calibration were assessed, and a model update was performed by re-estimation of coefficients with correction for overfitting. For two risk scores, the SBI score and PAWS, the diagnostic test accuracy was assessed.

Results

A total of 8211 children were included, comprising 498 SI and 276 serious bacterial infections (SBI). Feverkidstool had a C-statistic of 0.80 (95% confidence interval 0.77-0.84) with good calibration for pneumonia and 0.74 (0.70-0.79) with poor calibration for other SBI. The Craig model had a C-statistic of 0.80 (0.77-0.83) for pneumonia, 0.75 (0.70-0.80) for complicated urinary tract infections and 0.63 (0.39-0.88) for bacteraemia, with poor calibration. The model update resulted in improved C-statistics for all outcomes and good overall calibration for Feverkidstool and the Craig model. SBI score and PAWS performed extremely weak with sensitivities of 0.12 (0.09-0.15) and 0.32 (0.28-0.37).

Conclusions

Feverkidstool and the Craig model show good discriminative ability for predicting SBI and a potential for early recognition of SBI, confirming good external validity in a low prevalence setting of SBI. The SBI score and PAWS showed poor diagnostic performance.

Trial registration

ClinicalTrials.gov, NCT02024282. Registered on 31 December 2013.

SUBMITTER: Bos DAG

PROVIDER: S-EPMC10114467 | biostudies-literature | 2023 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Clinical prediction models for serious infections in children: external validation in ambulatory care.

Bos David A G DAG De Burghgraeve Tine T De Sutter An A Buntinx Frank F Verbakel Jan Y JY

BMC medicine 20230418 1

<h4>Background</h4>Early distinction between mild and serious infections (SI) is challenging in children in ambulatory care. Clinical prediction models (CPMs), developed to aid physicians in clinical decision-making, require broad external validation before clinical use. We aimed to externally validate four CPMs, developed in emergency departments, in ambulatory care.<h4>Methods</h4>We applied the CPMs in a prospective cohort of acutely ill children presenting to general practices, outpatient pa ...[more]

PMID: 37072778

Similar Datasets

Project description:BackgroundDiagnosing serious infections in children is challenging, because of the low incidence of such infections and their non-specific presentation early in the course of illness. Prediction rules are promoted as a means to improve recognition of serious infections. A recent systematic review identified seven clinical prediction rules, of which only one had been prospectively validated, calling into question their appropriateness for clinical practice. We aimed to examine the diagnostic accuracy of these rules in multiple ambulatory care populations in Europe.MethodsFour clinical prediction rules and two national guidelines, based on signs and symptoms, were validated retrospectively in seven individual patient datasets from primary care and emergency departments, comprising 11,023 children from the UK, the Netherlands, and Belgium. The accuracy of each rule was tested, with pre-test and post-test probabilities displayed using dumbbell plots, with serious infection settings stratified as low prevalence (LP; <5%), intermediate prevalence (IP; 5 to 20%), and high prevalence (HP; >20%) . In LP and IP settings, sensitivity should be >90% for effective ruling out infection.ResultsIn LP settings, a five-stage decision tree and a pneumonia rule had sensitivities of >90% (at a negative likelihood ratio (NLR) of < 0.2) for ruling out serious infections, whereas the sensitivities of a meningitis rule and the Yale Observation Scale (YOS) varied widely, between 33 and 100%. In IP settings, the five-stage decision tree, the pneumonia rule, and YOS had sensitivities between 22 and 88%, with NLR ranging from 0.3 to 0.8. In an HP setting, the five-stage decision tree provided a sensitivity of 23%. In LP or IP settings, the sensitivities of the National Institute for Clinical Excellence guideline for feverish illness and the Dutch College of General Practitioners alarm symptoms ranged from 81 to 100%.ConclusionsNone of the clinical prediction rules examined in this study provided perfect diagnostic accuracy. In LP or IP settings, prediction rules and evidence-based guidelines had high sensitivity, providing promising rule-out value for serious infections in these datasets, although all had a percentage of residual uncertainty. Additional clinical assessment or testing such as point-of-care laboratory tests may be needed to increase clinical certainty. None of the prediction rules identified seemed to be valuable for HP settings such as emergency departments.

Project description:IntroductionExternal validation studies are essential to study the generalizability of prediction models. Recently a permutation test, focusing on discrimination as quantified by the c-statistic, was proposed to judge whether a prediction model is transportable to a new setting. We aimed to evaluate this test and compare it to previously proposed procedures to judge any changes in c-statistic from development to external validation setting.MethodsWe compared the use of the permutation test to the use of benchmark values of the c-statistic following from a previously proposed framework to judge transportability of a prediction model. In a simulation study we developed a prediction model with logistic regression on a development set and validated them in the validation set. We concentrated on two scenarios: 1) the case-mix was more heterogeneous and predictor effects were weaker in the validation set compared to the development set, and 2) the case-mix was less heterogeneous in the validation set and predictor effects were identical in the validation and development set. Furthermore we illustrated the methods in a case study using 15 datasets of patients suffering from traumatic brain injury.ResultsThe permutation test indicated that the validation and development set were homogenous in scenario 1 (in almost all simulated samples) and heterogeneous in scenario 2 (in 17%-39% of simulated samples). Previously proposed benchmark values of the c-statistic and the standard deviation of the linear predictors correctly pointed at the more heterogeneous case-mix in scenario 1 and the less heterogeneous case-mix in scenario 2.ConclusionThe recently proposed permutation test may provide misleading results when externally validating prediction models in the presence of case-mix differences between the development and validation population. To correctly interpret the c-statistic found at external validation it is crucial to disentangle case-mix differences from incorrect regression coefficients.

Project description:IntroductionNumerous prediction models have been developed to support treatment-related decisions for breast cancer patients. External validation, a prerequisite for implementation in clinical practice, has been performed for only a few models. This study aims to externally validate published clinical prediction models using population-based Dutch data.MethodsPatient-, tumor- and treatment-related data were derived from the Netherlands Cancer Registry (NCR). Model performance was assessed using the area under the receiver operating characteristic curve (AUC), scaled Brier score, and model calibration. Net benefit across applicable risk thresholds was evaluated with decision curve analysis.ResultsAfter assessing 922 models, 87 (9%) were included for validation. Models were excluded due to an incomplete model description (n = 262 (28%)), lack of required data (n = 521 (57%)), previously validated or developed with NCR data (n = 45 (5%)), or the associated NCR sample size was insufficient (n = 7 (1%)). The included models predicted survival (33 (38%) overall, 27 (31%) breast cancer-specific, and 3 (3%) other cause-specific), locoregional recurrence (n = 7 (8%)), disease free survival (n = 7 (8%)), metastases (n = 5 (6%)), lymph node involvement (n = 3 (3%)), pathologic complete response (n = 1 (1%)), and surgical margins (n = 1 (1%)). Seven models (8%) showed poor (AUC<0.6), 39 (45%) moderate (AUC:0.6-0.7), 38 (46%) good (AUC:0.7-0.9), and 3 (3%) excellent (AUC≥0.9) discrimination. Using the scaled Brier score, worse performance than an uninformative model was found in 34 (39%) models.ConclusionComprehensive registry data supports broad validation of published prediction models. Model performance varies considerably in new patient populations, affirming the importance of external validation studies before applying models in clinical practice. Well performing models could be clinically useful in a Dutch setting after careful impact evaluation.

Project description:ObjectiveTo validate all diagnostic prediction models for ruling out pulmonary embolism that are easily applicable in primary care.DesignSystematic review followed by independent external validation study to assess transportability of retrieved models to primary care medicine.Setting300 general practices in the Netherlands.ParticipantsIndividual patient dataset of 598 patients with suspected acute pulmonary embolism in primary care.Main outcome measuresDiscriminative ability of all models retrieved by systematic literature search, assessed by calculation and comparison of C statistics. After stratification into groups with high and low probability of pulmonary embolism according to pre-specified model cut-offs combined with qualitative D-dimer test, sensitivity, specificity, efficiency (overall proportion of patients with low probability of pulmonary embolism), and failure rate (proportion of pulmonary embolism cases in group of patients with low probability) were calculated for all models.ResultsTen published prediction models for the diagnosis of pulmonary embolism were found. Five of these models could be validated in the primary care dataset: the original Wells, modified Wells, simplified Wells, revised Geneva, and simplified revised Geneva models. Discriminative ability was comparable for all models (range of C statistic 0.75-0.80). Sensitivity ranged from 88% (simplified revised Geneva) to 96% (simplified Wells) and specificity from 48% (revised Geneva) to 53% (simplified revised Geneva). Efficiency of all models was between 43% and 48%. Differences were observed between failure rates, especially between the simplified Wells and the simplified revised Geneva models (failure rates 1.2% (95% confidence interval 0.2% to 3.3%) and 3.1% (1.4% to 5.9%), respectively; absolute difference -1.98% (-3.33% to -0.74%)). Irrespective of the diagnostic prediction model used, three patients were incorrectly classified as having low probability of pulmonary embolism; pulmonary embolism was diagnosed only after referral to secondary care.ConclusionsFive diagnostic pulmonary embolism prediction models that are easily applicable in primary care were validated in this setting. Whereas efficiency was comparable for all rules, the Wells rules gave the best performance in terms of lower failure rates.

Project description:ObjectiveTo derive, cross validate, and externally validate a clinical prediction model that assesses the risks of different serious bacterial infections in children with fever at the emergency department.DesignProspective observational diagnostic study.SettingThree paediatric emergency care units: two in the Netherlands and one in the United Kingdom.ParticipantsChildren with fever, aged 1 month to 15 years, at three paediatric emergency care units: Rotterdam (n=1750) and the Hague (n=967), the Netherlands, and Coventry (n=487), United Kingdom. A prediction model was constructed using multivariable polytomous logistic regression analysis and included the predefined predictor variables age, duration of fever, tachycardia, temperature, tachypnoea, ill appearance, chest wall retractions, prolonged capillary refill time (>3 seconds), oxygen saturation <94%, and C reactive protein.Main outcome measuresPneumonia, other serious bacterial infections (SBIs, including septicaemia/meningitis, urinary tract infections, and others), and no SBIs.ResultsOxygen saturation <94% and presence of tachypnoea were important predictors of pneumonia. A raised C reactive protein level predicted the presence of both pneumonia and other SBIs, whereas chest wall retractions and oxygen saturation <94% were useful to rule out the presence of other SBIs. Discriminative ability (C statistic) to predict pneumonia was 0.81 (95% confidence interval 0.73 to 0.88); for other SBIs this was even better: 0.86 (0.79 to 0.92). Risk thresholds of 10% or more were useful to identify children with serious bacterial infections; risk thresholds less than 2.5% were useful to rule out the presence of serious bacterial infections. External validation showed good discrimination for the prediction of pneumonia (0.81, 0.69 to 0.93); discriminative ability for the prediction of other SBIs was lower (0.69, 0.53 to 0.86).ConclusionA validated prediction model, including clinical signs, symptoms, and C reactive protein level, was useful for estimating the likelihood of pneumonia and other SBIs in children with fever, such as septicaemia/meningitis and urinary tract infections.

Project description:BackgroundNearly 150 million children under-5 years of age were stunted in 2020. We aimed to develop a clinical prediction rule (CPR) to identify children likely to experience additional stunting following acute diarrhea, to enable targeted approaches to prevent this irreversible outcome.MethodsWe used clinical and demographic data from the Global Enteric Multicenter Study (GEMS) to build predictive models of linear growth faltering (decrease of ≥0.5 or ≥1.0 in height-for-age z-score [HAZ] at 60-day follow-up) in children ≤59 months presenting with moderate-to-severe diarrhea, and community controls, in Africa and Asia. We screened variables using random forests, and assessed predictive performance with random forest regression and logistic regression using fivefold cross-validation. We used the Etiology, Risk Factors, and Interactions of Enteric Infections and Malnutrition and the Consequences for Child Health and Development (MAL-ED) study to (1) re-derive, and (2) externally validate our GEMS-derived CPR.ResultsOf 7639 children in GEMS, 1744 (22.8%) experienced severe growth faltering (≥0.5 decrease in HAZ). In MAL-ED, we analyzed 5683 diarrhea episodes from 1322 children, of which 961 (16.9%) episodes experienced severe growth faltering. Top predictors of growth faltering in GEMS were: age, HAZ at enrollment, respiratory rate, temperature, and number of people living in the household. The maximum area under the curve (AUC) was 0.75 (95% confidence interval [CI]: 0.75, 0.75) with 20 predictors, while 2 predictors yielded an AUC of 0.71 (95% CI: 0.71, 0.72). Results were similar in the MAL-ED re-derivation. A 2-variable CPR derived from children 0-23 months in GEMS had an AUC = 0.63 (95% CI: 0.62, 0.65), and AUC = 0.68 (95% CI: 0.63, 0.74) when externally validated in MAL-ED.ConclusionsOur findings indicate that use of prediction rules could help identify children at risk of poor outcomes after an episode of diarrheal illness. They may also be generalizable to all children, regardless of diarrhea status.FundingThis work was supported by the National Institutes of Health under Ruth L. Kirschstein National Research Service Award NIH T32AI055434 and by the National Institute of Allergy and Infectious Diseases (R01AI135114).

Dataset Information

Clinical prediction models for serious infections in children: external validation in ambulatory care.

Background

Methods

Results

Conclusions

Trial registration

Publications

Clinical prediction models for serious infections in children: external validation in ambulatory care.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets