Dataset Information

Implementation of the COVID-19 Vulnerability Index Across an International Network of Health Care Data Sets: Collaborative External Validation Study.

ABSTRACT:

Background

SARS-CoV-2 is straining health care systems globally. The burden on hospitals during the pandemic could be reduced by implementing prediction models that can discriminate patients who require hospitalization from those who do not. The COVID-19 vulnerability (C-19) index, a model that predicts which patients will be admitted to hospital for treatment of pneumonia or pneumonia proxies, has been developed and proposed as a valuable tool for decision-making during the pandemic. However, the model is at high risk of bias according to the "prediction model risk of bias assessment" criteria, and it has not been externally validated.

Objective

The aim of this study was to externally validate the C-19 index across a range of health care settings to determine how well it broadly predicts hospitalization due to pneumonia in COVID-19 cases.

Methods

We followed the Observational Health Data Sciences and Informatics (OHDSI) framework for external validation to assess the reliability of the C-19 index. We evaluated the model on two different target populations, 41,381 patients who presented with SARS-CoV-2 at an outpatient or emergency department visit and 9,429,285 patients who presented with influenza or related symptoms during an outpatient or emergency department visit, to predict their risk of hospitalization with pneumonia during the following 0-30 days. In total, we validated the model across a network of 14 databases spanning the United States, Europe, Australia, and Asia.

Results

The internal validation performance of the C-19 index had a C statistic of 0.73, and the calibration was not reported by the authors. When we externally validated it by transporting it to SARS-CoV-2 data, the model obtained C statistics of 0.36, 0.53 (0.473-0.584) and 0.56 (0.488-0.636) on Spanish, US, and South Korean data sets, respectively. The calibration was poor, with the model underestimating risk. When validated on 12 data sets containing influenza patients across the OHDSI network, the C statistics ranged between 0.40 and 0.68.

Conclusions

Our results show that the discriminative performance of the C-19 index model is low for influenza cohorts and even worse among patients with COVID-19 in the United States, Spain, and South Korea. These results suggest that C-19 should not be used to aid decision-making during the COVID-19 pandemic. Our findings highlight the importance of performing external validation across a range of settings, especially when a prediction model is being extrapolated to a different population. In the field of prediction, extensive validation is required to create appropriate trust in a model.

SUBMITTER: Reps JM

PROVIDER: S-EPMC8023380 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:ObjectiveTo evaluate the comparative effectiveness of external facilitation (EF) vs external + internal facilitation (EF/IF), on uptake of a collaborative chronic care model (CCM) in community practices that were slower to implement under low-level implementation support.Study settingPrimary data were collected from 43 community practices in Michigan and Colorado at baseline and for 12 months following randomization.Study designSites that failed to meet a pre-established implementation benchmark after six months of low-level implementation support were randomized to add either EF or EF/IF support for up to 12 months. Key outcomes were change in number of patients receiving the CCM and number of patients receiving a clinically significant dose of the CCM. Moderators' analyses further examined whether comparative effectiveness was dependent on prerandomization adoption, number of providers trained or practice size. Facilitation log data were used for exploratory follow-up analyses.Data collectionSites reported monthly on number of patients that had received the CCM. Facilitation logs were completed by study EF and site IFs and shared with the study team.Principal findingsN = 21 sites were randomized to EF and 22 to EF/IF. Overall, EF/IF practices saw more uptake than EF sites after 12 months (ΔEF/IF-EF = 4.4 patients, 95% CI = 1.87-6.87). Moderators' analyses, however, revealed that it was only sites with no prerandomization uptake of the CCM (nonadopter sites) that saw significantly more benefit from EF/IF (ΔEF/IF-EF = 9.2 patients, 95% CI: 5.72, 12.63). For sites with prerandomization uptake (adopter sites), EF/IF offered no additional benefit (ΔEF/IF-EF = -0.9; 95% CI: -4.40, 2.60). Number of providers trained and practice size were not significant moderators.ConclusionsAlthough stepping up to the more intensive EF/IF did outperform EF overall, its benefit was limited to sites that failed to deliver any CCM under the low-level strategy. Once one or more providers were delivering the CCM, additional on-site personnel did not appear to add value to the implementation effort.

Project description:Background and objectivesThe International IgA Nephropathy Network recently developed and externally validated two models to predict the risk of progression of IgA nephropathy: full models without and with race. This study sought to externally validate the International IgA Nephropathy Prediction Tool in a large, independent, and contemporary cohort in China.Design, setting, participants, & measurementsWe included 1373 patients with biopsy-confirmed primary IgA nephropathy from The First Affiliated Hospital of Zhengzhou University from January 2012 to May 2018 and calculated predicted risks for each patient. The outcomes of interest were a 50% decline in eGFR or kidney failure. We assessed the performance of both models using discrimination (concordance statistics and Kaplan-Meier curves between subgroups), calibration (calibration plots), reclassification (net reclassification improvement and integrated discrimination improvement), and clinical utility (decision curve analysis).ResultsThe median follow-up was 29 months (interquartile range, 21-43 months; range, 1-95 months), and 186 (14%) patients reached the kidney outcomes of interest. Both models showed excellent discrimination (concordance statistics >0.85 and well separated survival curves). Overall, the full model without race generally underestimated the risk of primary outcome, whereas the full model with race was well calibrated for predicting 5-year risk. Compared with the full model without race, the full model with race had significant improvement in reclassification, as assessed by the net reclassification improvement (0.49; 95% confidence interval, 0.41 to 0.59) and integrated discrimination improvement (0.06; 95% confidence interval, 0.04 to 0.08). Decision curve analysis showed that both full models had a higher net benefit than default strategies, and the model with race performed better.ConclusionsIn this study, both full models demonstrated remarkable discrimination, acceptable calibration, and satisfactory clinical utility. The relatively short follow-up time may have limited the validation of these models.

Project description:ObjectivesAn obstetric comorbidity index has been developed recently with superior performance characteristics relative to general comorbidity measures in an obstetric population. This study aimed to externally validate this index and to examine the impact of including hospitalisation/delivery records only when estimating comorbidity prevalence and discriminative performance of the obstetric comorbidity index.DesignValidation study.SettingAlberta, Canada.PopulationPregnant women who delivered a live or stillborn infant in hospital (n = 5995).MethodsAdministrative databases were linked to create a population-based cohort. Comorbid conditions were identified from diagnoses for the delivery hospitalisation, all hospitalisations and all healthcare contacts (i.e. hospitalisations, emergency room visits and physician visits) that occurred during pregnancy and 3 months pre-conception. Logistic regression was used to test the discriminative performance of the comorbidity index.Main outcome measuresMaternal end-organ damage and extended length of stay for delivery.ResultsAlthough prevalence estimates for comorbid conditions were consistently lower in delivery records and hospitalisation data than in data for all healthcare contacts, the discriminative performance of the comorbidity index was constant for maternal end-organ damage [all healthcare contacts area under the receiver operating characteristic curve (AUC) = 0.70; hospitalisation data AUC = 0.67; delivery data AUC = 0.65] and extended length of stay for delivery (all healthcare contacts AUC = 0.60; hospitalisation data AUC = 0.58; delivery data AUC = 0.58).ConclusionsThe obstetric comorbidity index shows similar performance characteristics in an external population and is a valid measure of comorbidity in an obstetric population. Furthermore, the discriminative performance of the comorbidity index was similar for comorbidities ascertained at the time of delivery, in hospitalisation data or through all healthcare contacts.