Dataset Information

The performance of seven QPrediction risk scores in an independent external sample of patients from general practice: a validation study.

ABSTRACT: To validate the performance of a set of risk prediction algorithms developed using the QResearch database, in an independent sample from general practices contributing to the Clinical Research Data Link (CPRD).Prospective open cohort study using practices contributing to the CPRD database and practices contributing to the QResearch database.The CPRD validation cohort consisted of 3.3 million patients, aged 25-99?years registered at 357 general practices between 1 Jan 1998 and 31 July 2012. The validation statistics for QResearch were obtained from the original published papers which used a one-third sample of practices separate to those used to derive the score. A cohort from QResearch was used to compare incidence rates and baseline characteristics and consisted of 6.8 million patients from 753 practices registered between 1 Jan 1998 and until 31 July 2013.Incident events relating to seven different risk prediction scores: QRISK2 (cardiovascular disease); QStroke (ischaemic stroke); QDiabetes (type 2 diabetes); QFracture (osteoporotic fracture and hip fracture); QKidney (moderate and severe kidney failure); QThrombosis (venous thromboembolism); QBleed (intracranial bleed and upper gastrointestinal haemorrhage). Measures of discrimination and calibration were calculated.Overall, the baseline characteristics of the CPRD and QResearch cohorts were similar though QResearch had higher recording levels for ethnicity and family history. The validation statistics for each of the risk prediction scores were very similar in the CPRD cohort compared with the published results from QResearch validation cohorts. For example, in women, the QDiabetes algorithm explained 50% of the variation within CPRD compared with 51% on QResearch and the receiver operator curve value was 0.85 on both databases. The scores were well calibrated in CPRD.Each of the algorithms performed practically as well in the external independent CPRD validation cohorts as they had in the original published QResearch validation cohorts.

SUBMITTER: Hippisley-Cox J

PROVIDER: S-EPMC4156807 | biostudies-other | 2014

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

The performance of seven QPrediction risk scores in an independent external sample of patients from general practice: a validation study.

Hippisley-Cox Julia J Coupland Carol C Brindle Peter P

BMJ open 20140828 8

<h4>Objectives</h4>To validate the performance of a set of risk prediction algorithms developed using the QResearch database, in an independent sample from general practices contributing to the Clinical Research Data Link (CPRD).<h4>Setting</h4>Prospective open cohort study using practices contributing to the CPRD database and practices contributing to the QResearch database.<h4>Participants</h4>The CPRD validation cohort consisted of 3.3 million patients, aged 25-99 years registered at 357 gene ...[more]

PMID: 25168040

Similar Datasets

Project description:Two clinical scoring systems, the status epilepticus severity score (STESS) and the epidemiology-based mortality score in status epilepticus (EMSE), are used to predict mortality in patients with status epilepticus (SE). The aim of this study was to compare the outcome-prediction function of the two scoring systems regarding in-hospital mortality using a multicenter large cohort of adult patients with SE. Moreover, we studied the potential role of these two scoring systems in predicting the functional outcome in patients with SE.The SE cohort consisted of patients from the epilepsy centers of eight academic tertiary medical centers in South Korea. The clinical and electroencephalography data for all adult patients with SE from January 2013 to December 2014 were derived from a prospective SE database. The primary outcome variable was defined as in-hospital death. The secondary outcome variable was defined as a poor functional outcome, i.e., a score of 1-3 on the Glasgow Outcome Scale, at discharge.Among the 120 non-hypoxic patients with SE recruited into the study, 16 (13.3%) died in the hospital and 64 (53.3%) were discharged with a poor functional outcome. The receiver-operating characteristic (ROC) curve for prediction of in-hospital death based on the STESS had an area under the curve of 0.673 with an optimal cutoff value for discrimination (best match for both sensitivity (0.56) and specificity (0.70)) that was ? 4 points. The two combinations of elements of the EMSE system (EMSE-ALDEg and EMSE-ECLEg) predicted not only in-hospital mortality with the best match for sensitivity (more than 0.6) and specificity (more than 0.6), but also a poor functional outcome with the best match for both sensitivity (>0.7) and specificity (>0.6). STESS did not predict a poor functional outcome (area under the ROC, 0.581; P = 0.23).Although the EMSE is a clinical scoring system that focuses on individual mortality, we did not find differences between the EMSE and STESS in the prediction of in-hospital death. The EMSE was useful in predicting poor functional outcome, as it was significantly better than STESS.

Project description:Risk-prediction models for health outcomes are used in practice as part of clinical decision-making, and it is essential that their performance be externally validated. An important aspect in the design of a validation study is choosing an adequate sample size. In this paper, we investigate the sample size requirements for validation studies with binary outcomes to estimate measures of predictive performance (C-statistic for discrimination and calibration slope and calibration in the large). We aim for sufficient precision in the estimated measures. In addition, we investigate the sample size to achieve sufficient power to detect a difference from a target value. Under normality assumptions on the distribution of the linear predictor, we obtain simple estimators for sample size calculations based on the measures above. Simulation studies show that the estimators perform well for common values of the C-statistic and outcome prevalence when the linear predictor is marginally Normal. Their performance deteriorates only slightly when the normality assumptions are violated. We also propose estimators which do not require normality assumptions but require specification of the marginal distribution of the linear predictor and require the use of numerical integration. These estimators were also seen to perform very well under marginal normality. Our sample size equations require a specified standard error (SE) and the anticipated C-statistic and outcome prevalence. The sample size requirement varies according to the prognostic strength of the model, outcome prevalence, choice of the performance measure and study objective. For example, to achieve an SE < 0.025 for the C-statistic, 60-170 events are required if the true C-statistic and outcome prevalence are between 0.64-0.85 and 0.05-0.3, respectively. For the calibration slope and calibration in the large, achieving SE < 0.15 would require 40-280 and 50-100 events, respectively. Our estimators may also be used for survival outcomes when the proportion of censored observations is high.

Project description:Cirrhosis poses an increased risk of postoperative mortality, yet it remains challenging to accurately risk stratify patients in clinical practice. The VOCAL-Penn cirrhosis surgical risk score was recently developed and internally validated in the national Veterans Affairs health system; however, to date this score has not been evaluated in independent cohorts. The goal of this study was to compare the predictive performance of the VOCAL-Penn to the Mayo risk, Model for End-Stage Liver Disease (MELD), and MELD-sodium (MELD-Na) scores in 2 large health systems. We performed a retrospective cohort study of patients with cirrhosis undergoing surgical procedures of interest at the Beth Israel Deaconess Medical Center or University of Pennsylvania Health System from January 1, 2008, to October 1, 2015. The outcomes of interest were 30-day and 90-day postoperative mortality. Concordance statistics (C-statistics), calibration curves, Brier scores, and the index of prediction accuracy (IPA) were compared for each predictive model. A total of 855 surgical procedures were identified. The VOCAL-Penn score had the numerically highest C-statistic for 90-day postoperative mortality (eg, 0.82 versus 0.79 Mayo versus 0.78 MELD-Na versus 0.79 MELD), although differences were not statistically significant. Calibrations were excellent for the VOCAL-Penn, MELD, and MELD-Na; however, the Mayo score consistently overestimated risk. The VOCAL-Penn had the lowest Brier score and highest IPA at both time points, suggesting superior overall predictive model performance. In subgroup analyses of patients with higher MELD scores, the VOCAL-Penn had significantly higher C-statistics compared with the MELD and MELD-Na. The VOCAL-Penn score (www.vocalpennscore.com) has excellent discrimination and calibration for postoperative mortality among diverse patients with cirrhosis. Overall performance is superior to the Mayo, MELD, and MELD-Na scores. In contrast to the MELD/MELD-Na, the VOCAL-Penn retains excellent discrimination among patients with higher MELD scores.

Project description:BackgroundClinical prediction rules (CPRs) should be externally validated by independent researchers. Although there are many cardiovascular CPRs, most have not been externally validated. It is not known why some CPRs are externally validated by independent researchers and others are not.MethodsWe analyzed cardiovascular risk CPRs included in a systematic review. Independent external validations were identified by forward citation searches of derivation studies. Time between the publication of a cardiovascular CPR and the first independent external validation was calculated. We assessed Kaplan-Meier estimates of the probability to have an independent external validation. Using univariable Cox regression, we explored whether characteristics of derivation (design, location, sample size, number of predictors, presentation format, validation in derivation), reporting (participants, predictors, outcomes, performance measure, information for risk calculation), and publication (journal impact factor) are associated with time to the first independent external validation.ResultsOf 125 cardiovascular risk CPRs, 29 had an independent external validation. The median follow-up was 118 months (95% CI, 99-130). The 25th percentile of event time was 122 months (95% CI, 91-299). Cardiovascular risk CPRs from the USA were 4.15 times (95% CI, 1.89-9.13) more likely to have an independent external validation. Increasing the sample size of derivation by ten times was associated with a 2.32-fold (95% CI, 1.37-3.91) increase in the probability of having an independent external validation. CPRs presented with an internal validation tend to get an independent external validation sooner (HR = 1.73, 95% CI, 0.77-3.93). CPRs reporting all the information necessary for calculating individual risk were 2.65 (95% CI, 1.01-6.96) times more likely to have an independent external validation. Publishing a cardiovascular risk CPR in a journal that has one unit higher impact factor was associated with a 6% (95% CI, 3-9) higher likelihood of an independent external validation.ConclusionsThe probability for cardiovascular risk CPRs to get an independent external validation was low even many years after their derivations. Authors of new cardiovascular risk CPRs should consider using adequate sample size, conducting an internal validation, and reporting all the information needed for risk calculation as these features were associated with an independent external validation.

Project description:ObjectiveTo identify existing prediction models for the risk of development of type 2 diabetes and to externally validate them in a large independent cohort.Data sourcesSystematic search of English, German, and Dutch literature in PubMed until February 2011 to identify prediction models for diabetes.DesignPerformance of the models was assessed in terms of discrimination (C statistic) and calibration (calibration plots and Hosmer-Lemeshow test).The validation study was a prospective cohort study, with a case cohort study in a random subcohort.SettingModels were applied to the Dutch cohort of the European Prospective Investigation into Cancer and Nutrition cohort study (EPIC-NL).Participants38,379 people aged 20-70 with no diabetes at baseline, 2506 of whom made up the random subcohort.Outcome measureIncident type 2 diabetes.ResultsThe review identified 16 studies containing 25 prediction models. We considered 12 models as basic because they were based on variables that can be assessed non-invasively and 13 models as extended because they additionally included conventional biomarkers such as glucose concentration. During a median follow-up of 10.2 years there were 924 cases in the full EPIC-NL cohort and 79 in the random subcohort. The C statistic for the basic models ranged from 0.74 (95% confidence interval 0.73 to 0.75) to 0.84 (0.82 to 0.85) for risk at 7.5 years. For prediction models including biomarkers the C statistic ranged from 0.81 (0.80 to 0.83) to 0.93 (0.92 to 0.94). Most prediction models overestimated the observed risk of diabetes, particularly at higher observed risks. After adjustment for differences in incidence of diabetes, calibration improved considerably.ConclusionsMost basic prediction models can identify people at high risk of developing diabetes in a time frame of five to 10 years. Models including biomarkers classified cases slightly better than basic ones. Most models overestimated the actual risk of diabetes. Existing prediction models therefore perform well to identify those at high risk, but cannot sufficiently quantify actual risk of future diabetes.

Dataset Information

The performance of seven QPrediction risk scores in an independent external sample of patients from general practice: a validation study.

Publications

The performance of seven QPrediction risk scores in an independent external sample of patients from general practice: a validation study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets