Dataset Information

Multivariate meta-analysis of individual participant data helped externally validate the performance and implementation of a prediction model.

ABSTRACT: Our aim was to improve meta-analysis methods for summarizing a prediction model's performance when individual participant data are available from multiple studies for external validation.We suggest multivariate meta-analysis for jointly synthesizing calibration and discrimination performance, while accounting for their correlation. The approach estimates a prediction model's average performance, the heterogeneity in performance across populations, and the probability of "good" performance in new populations. This allows different implementation strategies (e.g., recalibration) to be compared. Application is made to a diagnostic model for deep vein thrombosis (DVT) and a prognostic model for breast cancer mortality.In both examples, multivariate meta-analysis reveals that calibration performance is excellent on average but highly heterogeneous across populations unless the model's intercept (baseline hazard) is recalibrated. For the cancer model, the probability of "good" performance (defined by C statistic ?0.7 and calibration slope between 0.9 and 1.1) in a new population was 0.67 with recalibration but 0.22 without recalibration. For the DVT model, even with recalibration, there was only a 0.03 probability of "good" performance.Multivariate meta-analysis can be used to externally validate a prediction model's calibration and discrimination performance across multiple populations and to evaluate different implementation strategies.

SUBMITTER: Snell KI

PROVIDER: S-EPMC4688112 | biostudies-literature | 2016 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Multivariate meta-analysis of individual participant data helped externally validate the performance and implementation of a prediction model.

Snell Kym I E KI Hua Harry H Debray Thomas P A TP Ensor Joie J Look Maxime P MP Moons Karel G M KG Riley Richard D RD

Journal of clinical epidemiology 20150516

<h4>Objectives</h4>Our aim was to improve meta-analysis methods for summarizing a prediction model's performance when individual participant data are available from multiple studies for external validation.<h4>Study design and setting</h4>We suggest multivariate meta-analysis for jointly synthesizing calibration and discrimination performance, while accounting for their correlation. The approach estimates a prediction model's average performance, the heterogeneity in performance across populatio ...[more]

PMID: 26142114

Similar Datasets

Project description:ObjectiveTo externally validate various prognostic models and scoring rules for predicting short term mortality in patients admitted to hospital for covid-19.DesignTwo stage individual participant data meta-analysis.SettingSecondary and tertiary care.Participants46 914 patients across 18 countries, admitted to a hospital with polymerase chain reaction confirmed covid-19 from November 2019 to April 2021.Data sourcesMultiple (clustered) cohorts in Brazil, Belgium, China, Czech Republic, Egypt, France, Iran, Israel, Italy, Mexico, Netherlands, Portugal, Russia, Saudi Arabia, Spain, Sweden, United Kingdom, and United States previously identified by a living systematic review of covid-19 prediction models published in The BMJ, and through PROSPERO, reference checking, and expert knowledge.Model selection and eligibility criteriaPrognostic models identified by the living systematic review and through contacting experts. A priori models were excluded that had a high risk of bias in the participant domain of PROBAST (prediction model study risk of bias assessment tool) or for which the applicability was deemed poor.MethodsEight prognostic models with diverse predictors were identified and validated. A two stage individual participant data meta-analysis was performed of the estimated model concordance (C) statistic, calibration slope, calibration-in-the-large, and observed to expected ratio (O:E) across the included clusters.Main outcome measures30 day mortality or in-hospital mortality.ResultsDatasets included 27 clusters from 18 different countries and contained data on 46 914patients. The pooled estimates ranged from 0.67 to 0.80 (C statistic), 0.22 to 1.22 (calibration slope), and 0.18 to 2.59 (O:E ratio) and were prone to substantial between study heterogeneity. The 4C Mortality Score by Knight et al (pooled C statistic 0.80, 95% confidence interval 0.75 to 0.84, 95% prediction interval 0.72 to 0.86) and clinical model by Wang et al (0.77, 0.73 to 0.80, 0.63 to 0.87) had the highest discriminative ability. On average, 29% fewer deaths were observed than predicted by the 4C Mortality Score (pooled O:E 0.71, 95% confidence interval 0.45 to 1.11, 95% prediction interval 0.21 to 2.39), 35% fewer than predicted by the Wang clinical model (0.65, 0.52 to 0.82, 0.23 to 1.89), and 4% fewer than predicted by Xie et al's model (0.96, 0.59 to 1.55, 0.21 to 4.28).ConclusionThe prognostic value of the included models varied greatly between the data sources. Although the Knight 4C Mortality Score and Wang clinical model appeared most promising, recalibration (intercept and slope updates) is needed before implementation in routine care.

Project description:BACKGROUND:The usefulness of estimated glomerular filtration rate (eGFR) and albuminuria for prediction of cardiovascular outcomes is controversial. We aimed to assess the addition of creatinine-based eGFR and albuminuria to traditional risk factors for prediction of cardiovascular risk with a meta-analytic approach. METHODS:We meta-analysed individual-level data for 637 315 individuals without a history of cardiovascular disease from 24 cohorts (median follow-up 4·2-19·0 years) included in the Chronic Kidney Disease Prognosis Consortium. We assessed C statistic difference and reclassification improvement for cardiovascular mortality and fatal and non-fatal cases of coronary heart disease, stroke, and heart failure in a 5 year timeframe, contrasting prediction models for traditional risk factors with and without creatinine-based eGFR, albuminuria (either albumin-to-creatinine ratio [ACR] or semi-quantitative dipstick proteinuria), or both. FINDINGS:The addition of eGFR and ACR significantly improved the discrimination of cardiovascular outcomes beyond traditional risk factors in general populations, but the improvement was greater with ACR than with eGFR, and more evident for cardiovascular mortality (C statistic difference 0·0139 [95% CI 0·0105-0·0174] for ACR and 0·0065 [0·0042-0·0088] for eGFR) and heart failure (0·0196 [0·0108-0·0284] and 0·0109 [0·0059-0·0159]) than for coronary disease (0·0048 [0·0029-0·0067] and 0·0036 [0·0019-0·0054]) and stroke (0·0105 [0·0058-0·0151] and 0·0036 [0·0004-0·0069]). Dipstick proteinuria showed smaller improvement than ACR. The discrimination improvement with eGFR or ACR was especially evident in individuals with diabetes or hypertension, but remained significant with ACR for cardiovascular mortality and heart failure in those without either of these disorders. In individuals with chronic kidney disease, the combination of eGFR and ACR for risk discrimination outperformed most single traditional predictors; the C statistic for cardiovascular mortality fell by 0·0227 (0·0158-0·0296) after omission of eGFR and ACR compared with less than 0·007 for any single modifiable traditional predictor. INTERPRETATION:Creatinine-based eGFR and albuminuria should be taken into account for cardiovascular prediction, especially when these measures are already assessed for clinical purpose or if cardiovascular mortality and heart failure are outcomes of interest. ACR could have particularly broad implications for cardiovascular prediction. In populations with chronic kidney disease, the simultaneous assessment of eGFR and ACR could facilitate improved classification of cardiovascular risk, supporting current guidelines for chronic kidney disease. Our results lend some support to also incorporating eGFR and ACR into assessments of cardiovascular risk in the general population. FUNDING:US National Kidney Foundation, National Institute of Diabetes and Digestive and Kidney Diseases.

Dataset Information

Multivariate meta-analysis of individual participant data helped externally validate the performance and implementation of a prediction model.

Publications

Multivariate meta-analysis of individual participant data helped externally validate the performance and implementation of a prediction model.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets