Dataset Information

The effects of misclassification in routine healthcare databases on the accuracy of prognostic prediction models: a case study of the CHA2DS2-VASc score in atrial fibrillation.

ABSTRACT:

Background

Research on prognostic prediction models frequently uses data from routine healthcare. However, potential misclassification of predictors when using such data may strongly affect the studied associations. There is no doubt that such misclassification could lead to the derivation of suboptimal prediction models. The extent to which misclassification affects the validation of existing prediction models is currently unclear.We aimed to quantify the amount of misclassification in routine care data and its effect on the validation of the existing risk prediction model. As an illustrative example, we validated the CHA2DS2-VASc prediction rule for predicting mortality in patients with atrial fibrillation (AF).

Methods

In a prospective cohort in general practice in the Netherlands, we used computerized retrieved data from the electronic medical records of patients known with AF as index predictors. Additionally, manually collected data after scrutinizing all complete medical files were used as reference predictors. Comparing the index with the reference predictors, we assessed misclassification in individual predictors by calculating Cohen's kappas and other diagnostic test accuracy measures. Predictive performance was quantified by the c-statistic and by determining calibration of multivariable models.

Results

In total, 2363 AF patients were included. After a median follow-up of 2.7 (IQR 2.3-3.0) years, 368 patients died (incidence rate 6.2 deaths per 100 person-years). Misclassification in individual predictors ranged from substantial (Cohen's kappa 0.56 for prior history of heart failure) to minor (kappa 0.90 for a history of type 2 diabetes). The overall model performance was not affected when using either index or reference predictors, with a c-statistic of 0.684 and 0.681, respectively, and similar calibration.

Conclusion

In a case study validating the CHA2DS2-VASc prediction model, we found substantial predictor misclassification in routine healthcare data with only limited effect on overall model performance. Our study should be repeated for other often applied prediction models to further evaluate the usefulness of routinely available healthcare data for validating prognostic models in the presence of predictor misclassification.

SUBMITTER: van Doorn S

PROVIDER: S-EPMC6460749 | biostudies-literature | 2017

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

The effects of misclassification in routine healthcare databases on the accuracy of prognostic prediction models: a case study of the CHA2DS2-VASc score in atrial fibrillation.

van Doorn S S Brakenhoff T B TB Moons K G M KGM Rutten F H FH Hoes A W AW Groenwold R H H RHH Geersing G J GJ

Diagnostic and prognostic research 20171116

<h4>Background</h4>Research on prognostic prediction models frequently uses data from routine healthcare. However, potential misclassification of predictors when using such data may strongly affect the studied associations. There is no doubt that such misclassification could lead to the derivation of suboptimal prediction models. The extent to which misclassification affects the validation of existing prediction models is currently unclear.We aimed to quantify the amount of misclassification in ...[more]

PMID: 31093547

Similar Datasets

Project description:BACKGROUND:In people with atrial fibrillation (AF), periods of sinus rhythm present an opportunity to detect prothrombotic atrial remodeling through measurement of P-wave indices (PWIs)-prolonged P-wave duration, abnormal P-wave axis, advanced interatrial block, and abnormal P-wave terminal force in lead V1. We hypothesized that the addition of PWIs to the CHA2DS2-VASc score would improve its ability to predict AF-related ischemic stroke. METHODS:We included 2229 participants from the ARIC study (Atherosclerosis Risk in Communities) and 700 participants from MESA (Multi-Ethnic Study of Atherosclerosis) with incident AF who were not on anticoagulants within 1 year of AF diagnosis. PWIs were obtained from study visit ECGs before development of AF. AF was ascertained using study visit ECGs and hospital records. Ischemic stroke cases were based on physician adjudication of hospital records. We used Cox proportional hazards models to estimate hazard ratios and 95% CIs of PWIs for ischemic stroke. Improvement in 1-year stroke prediction was assessed by C-statistic, categorical net reclassification improvement, and relative integrated discrimination improvement. RESULTS:Abnormal P-wave axis was the only PWI associated with increased ischemic stroke risk (hazard ratio, 1.84; 95% CI, 1.33-2.55) independent of CHA2DS2-VASc variables, and that resulted in meaningful improvement in stroke prediction. The ? estimate was approximately twice that of the CHA2DS2-VASc variables, and thus abnormal P-wave axis was assigned 2 points to create the P2-CHA2DS2-VASc score. This improved the C-statistic (95% CI) from 0.60 (0.51-0.69) to 0.67 (0.60-0.75) in ARIC and 0.68 (0.52-0.84) to 0.75 (0.60-0.91) in MESA (validation cohort). In ARIC and MESA, the categorical net reclassification improvements (95% CI) were 0.25 (0.13-0.39) and 0.51 (0.18-0.86), respectively, and the relative integrated discrimination improvement (95% CI) were 1.19 (0.96-1.44) and 0.82 (0.36-1.39), respectively. CONCLUSIONS:Abnormal P-wave axis-an ECG correlate of left atrial abnormality- improves ischemic stroke prediction in AF. Compared with CHA2DS2-VASc, the P2-CHA2DS2-VASc is a better prediction tool for AF-related ischemic stroke.

Project description:ImportanceCurrent guidelines support treating atrial fibrillation (AF) and atrial flutter (AFL) as equivalent risk factors for ischemic stroke stratified by CHA2DS2-VASc scores, recommending anticoagulation therapy for patients with a CHA2DS2-VASc score of 2 or higher, but some studies found differences in clinical outcomes.ObjectiveTo investigate differences in clinical outcomes among AF, AFL, and matched control cohorts.Design, setting, and participantsThis nationwide cohort study analyzed data from the Taiwan National Health Insurance Research Database from January 1, 2001, through December 31, 2012. Follow-up and data analysis ended December 31, 2012. A total of 219 416 age- and sex-matched individuals participated in the study. Clinical outcomes were compared after stratification by CHA2DS2-VASc score (possible score range, 0-9; higher scores indicate greater risk of ischemic stroke).Main outcomes and measuresIschemic stroke, heart failure hospitalization, and all-cause mortality among the AF, AFL, and matched control cohorts were analyzed using Cox proportional hazards regression.ResultsThis study comprised 188?811 patients in the AF cohort (mean [SD] age, 73.8 [13.4] years; 104 703 [55.5%] male), 6121 patients in the AFL cohort (mean [SD] age, 67.7 [15.8] years; 3735 [61.0%] male), and 24?484 patients in the matched control cohort (mean [SD] age, 67.3 [15.6] years; 14 940 [61.0%] male). The patients with AF were older, were more predominantly female, and had higher CHA2DS2-VASc scores than the patients with AFL and the control participants. After stratification by CHA2DS2-VASc score, the incidence densities (IDs; events per 100 person-years) of ischemic stroke (AF cohort: ID, 3.08; 95% CI, 3.03-3.13; AFL cohort: ID, 1.45; 95% CI, 1.28-1.62; controls: ID, 0.97; 95% CI, 0.92-1.03), heart failure hospitalization (AF cohort: ID, 3.39; 95% CI, 3.34-3.44; AFL cohort: ID, 1.57; 95% CI, 1.39-1.74; controls: ID, 0.32; 95% CI, 0.29-0.35), and all-cause mortality (AF cohort: ID, 17.8; 95% CI, 17.7-17.9; AFL cohort: ID, 13.9; 95% CI, 13.4-14.4; controls: ID, 4.2; 95% CI, 4.1-4.4) were significantly higher in the AF cohort than in the matched control cohort. For the AFL cohort vs the matched control cohort, the incidences of heart failure hospitalization and all-cause mortality were significantly higher across all levels, but the incidence of ischemic stroke was only significantly higher at CHA2DS2-VASc scores of 5 to 9. For the AF cohort vs the AFL cohort, the incidences of ischemic stroke and heart failure hospitalization were significantly higher at a CHA2DS2-VASc score of 1 or higher, but the incidence of all-cause mortality was significantly higher only at CHA2DS2-VASc scores of 1 to 3.Conclusions and relevanceThis study found different clinical outcomes between patients with AFL and AF and those without AF and AFL. The current recommended level of the CHA2DS2-VASc score in preventing ischemic stroke in patients with AFL should be reevaluated.

Project description:AimsThe GRACE and CHA2DS2-VASc risk score are developed for risk stratification in patients with acute coronary syndrome and AF, respectively. We aimed to assess the predictive performance of the GRACE score and CHA2DS2-VASc score among patients with atrial fibrillation (AF) undergoing percutaneous coronary intervention (PCI).MethodsConsecutive patients with a diagnosis of AF admitted to our hospital for PCI between January 2016 and December 2018 were included and followed up for at least 1 year. The primary endpoint was a composite of major adverse cardiac events (MACEs) including all-cause mortality, repeat revascularization, myocardial infarction, or ischaemic stroke.ResultsA total of 1452 patients were identified. Cox regression demonstrated that the GRACE (HR 1.014, 95% CI 1.008-1.020, p < 0.001) but not the CHA2DS2-VASc score was associated with the risk of MACEs. Both GRACE and CHA2DS2-VASc scores were predictive of all-cause mortality with HR of 1.028 (95% CI 1.020-1.037, p < 0.001) and 1.334 (95% CI 1.107-1.632, p = 0.003). Receiver operating characteristic analyses showed both scores had similar discrimination capacity for all-cause mortality (C-statistic: 0.708 for GRACE vs. 0.661 for CHA2DS2-VASc, p = 0.299). High GRACE score was also significantly associated with increased risk of ischaemic stroke (HR 1.018, 95% CI 1.005-1.031, p = 0.006) and major bleeding (HR 1.012, 95% CI 1.001-1.024, p = 0.039), whereas high CHA2DS2-VASc score was not.ConclusionsHigh GRACE score but not CHA2DS2-VASc score were both associated with an increased risk of MACEs after PCI in patients with AF. The GRACE and CHA2DS2-VASc scores have similar predictive performance for predicting all-cause mortality.Key messages:In patients with AF undergoing PCI, increasing GRACE but not CHA2DS2-VASc scores was independently associated high risk of MACEs.The GRACE score could also help identify patients at higher risk of stroke and major bleeding.Both GRACE and CHA2DS2-VASc scores showed good ability in the prediction of all-cause mortality.

Project description:Atrial fibrillation (AF) is associated with increased mortality, predictors of which are poorly characterized. We investigated the predictive power of the commonly used CHA2DS2-VASc score (congestive heart failure, hypertension, age ≥ 75 years [doubled], diabetes, stroke/transient ischemic attack/thromboembolism [doubled], vascular disease [prior myocardial infarction, peripheral artery disease, or aortic plaque], age 65-75 years, sex category [female]), the HAS-BLED score (hypertension, abnormal renal/liver function, stroke, bleeding history or predisposition, labile international normalized ratio [INR], elderly [age ≥ 65 years], drugs/alcohol concomitantly), and their combination for mortality in AF patients. The PREvention oF thromboembolic events-European Registry in Atrial Fibrillation (PREFER in AF) was a prospective registry including AF patients across seven European countries. We used logistic regression to analyze the relationship between the CHA2DS2-VASc and HAS-BLED scores and outcomes, including mortality, at one year. We evaluated the performance of logistic regression models by discrimination measures (C-index and DeLong test) and calibration measures (Hosmer and Lemeshow goodness-of-fit and integrated discrimination improvement (IDI), with bootstrap techniques for internal validation. In 5209 AF patients with complete information on both scores, average one-year mortality was 3.1%. We found strong gradients between stroke/systemic embolic events (SSE), major bleeding and-specifically-mortality for both CHA2DS2-VASc and HAS-BLED scores, with a similar C-statistic for event prediction. The predictive power of the models with both scores combined, removing overlapping components, was significantly enhanced (p < 0.01) compared to models including either CHA2DS2-VASc or HAS-BLED alone: for mortality, C-statistic: 0.740, compared to 0.707 for CHA2DS2-VASc or 0.646 for HAS-BLED alone. IDI analyses supported the significant improvement for the combined score model compared to separate score models for all outcomes. Both the CHA2DS2-VASc and the HAS-BLED scores predict mortality similarly in patients with AF, and a combination of their components increases prediction significantly. Such combination may be useful for investigational and-possibly-also clinical purposes.

Dataset Information

The effects of misclassification in routine healthcare databases on the accuracy of prognostic prediction models: a case study of the CHA2DS2-VASc score in atrial fibrillation.

Background

Methods

Results

Conclusion

Publications

The effects of misclassification in routine healthcare databases on the accuracy of prognostic prediction models: a case study of the CHA2DS2-VASc score in atrial fibrillation.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets