Dataset Information

Developing and validating risk prediction models in an individual participant data meta-analysis.

ABSTRACT: Risk prediction models estimate the risk of developing future outcomes for individuals based on one or more underlying characteristics (predictors). We review how researchers develop and validate risk prediction models within an individual participant data (IPD) meta-analysis, in order to assess the feasibility and conduct of the approach.A qualitative review of the aims, methodology, and reporting in 15 articles that developed a risk prediction model using IPD from multiple studies.The IPD approach offers many opportunities but methodological challenges exist, including: unavailability of requested IPD, missing patient data and predictors, and between-study heterogeneity in methods of measurement, outcome definitions and predictor effects. Most articles develop their model using IPD from all available studies and perform only an internal validation (on the same set of data). Ten of the 15 articles did not allow for any study differences in baseline risk (intercepts), potentially limiting their model's applicability and performance in some populations. Only two articles used external validation (on different data), including a novel method which develops the model on all but one of the IPD studies, tests performance in the excluded study, and repeats by rotating the omitted study.An IPD meta-analysis offers unique opportunities for risk prediction research. Researchers can make more of this by allowing separate model intercept terms for each study (population) to improve generalisability, and by using 'internal-external cross-validation' to simultaneously develop and validate their model. Methodological challenges can be reduced by prospectively planned collaborations that share IPD for risk prediction.

SUBMITTER: Ahmed I

PROVIDER: S-EPMC3890557 | biostudies-literature | 2014 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Developing and validating risk prediction models in an individual participant data meta-analysis.

Ahmed Ikhlaaq I Debray Thomas P A TP Moons Karel G M KG Riley Richard D RD

BMC medical research methodology 20140108

<h4>Background</h4>Risk prediction models estimate the risk of developing future outcomes for individuals based on one or more underlying characteristics (predictors). We review how researchers develop and validate risk prediction models within an individual participant data (IPD) meta-analysis, in order to assess the feasibility and conduct of the approach.<h4>Methods</h4>A qualitative review of the aims, methodology, and reporting in 15 articles that developed a risk prediction model using IPD ...[more]

PMID: 24397587

Similar Datasets

Project description:ObjectiveTo externally validate various prognostic models and scoring rules for predicting short term mortality in patients admitted to hospital for covid-19.DesignTwo stage individual participant data meta-analysis.SettingSecondary and tertiary care.Participants46 914 patients across 18 countries, admitted to a hospital with polymerase chain reaction confirmed covid-19 from November 2019 to April 2021.Data sourcesMultiple (clustered) cohorts in Brazil, Belgium, China, Czech Republic, Egypt, France, Iran, Israel, Italy, Mexico, Netherlands, Portugal, Russia, Saudi Arabia, Spain, Sweden, United Kingdom, and United States previously identified by a living systematic review of covid-19 prediction models published in The BMJ, and through PROSPERO, reference checking, and expert knowledge.Model selection and eligibility criteriaPrognostic models identified by the living systematic review and through contacting experts. A priori models were excluded that had a high risk of bias in the participant domain of PROBAST (prediction model study risk of bias assessment tool) or for which the applicability was deemed poor.MethodsEight prognostic models with diverse predictors were identified and validated. A two stage individual participant data meta-analysis was performed of the estimated model concordance (C) statistic, calibration slope, calibration-in-the-large, and observed to expected ratio (O:E) across the included clusters.Main outcome measures30 day mortality or in-hospital mortality.ResultsDatasets included 27 clusters from 18 different countries and contained data on 46 914patients. The pooled estimates ranged from 0.67 to 0.80 (C statistic), 0.22 to 1.22 (calibration slope), and 0.18 to 2.59 (O:E ratio) and were prone to substantial between study heterogeneity. The 4C Mortality Score by Knight et al (pooled C statistic 0.80, 95% confidence interval 0.75 to 0.84, 95% prediction interval 0.72 to 0.86) and clinical model by Wang et al (0.77, 0.73 to 0.80, 0.63 to 0.87) had the highest discriminative ability. On average, 29% fewer deaths were observed than predicted by the 4C Mortality Score (pooled O:E 0.71, 95% confidence interval 0.45 to 1.11, 95% prediction interval 0.21 to 2.39), 35% fewer than predicted by the Wang clinical model (0.65, 0.52 to 0.82, 0.23 to 1.89), and 4% fewer than predicted by Xie et al's model (0.96, 0.59 to 1.55, 0.21 to 4.28).ConclusionThe prognostic value of the included models varied greatly between the data sources. Although the Knight 4C Mortality Score and Wang clinical model appeared most promising, recalibration (intercept and slope updates) is needed before implementation in routine care.

Project description:BackgroundPre-eclampsia is a leading cause of maternal and perinatal mortality and morbidity. Early identification of women at risk during pregnancy is required to plan management. Although there are many published prediction models for pre-eclampsia, few have been validated in external data. Our objective was to externally validate published prediction models for pre-eclampsia using individual participant data (IPD) from UK studies, to evaluate whether any of the models can accurately predict the condition when used within the UK healthcare setting.MethodsIPD from 11 UK cohort studies (217,415 pregnant women) within the International Prediction of Pregnancy Complications (IPPIC) pre-eclampsia network contributed to external validation of published prediction models, identified by systematic review. Cohorts that measured all predictor variables in at least one of the identified models and reported pre-eclampsia as an outcome were included for validation. We reported the model predictive performance as discrimination (C-statistic), calibration (calibration plots, calibration slope, calibration-in-the-large), and net benefit. Performance measures were estimated separately in each available study and then, where possible, combined across studies in a random-effects meta-analysis.ResultsOf 131 published models, 67 provided the full model equation and 24 could be validated in 11 UK cohorts. Most of the models showed modest discrimination with summary C-statistics between 0.6 and 0.7. The calibration of the predicted compared to observed risk was generally poor for most models with observed calibration slopes less than 1, indicating that predictions were generally too extreme, although confidence intervals were wide. There was large between-study heterogeneity in each model's calibration-in-the-large, suggesting poor calibration of the predicted overall risk across populations. In a subset of models, the net benefit of using the models to inform clinical decisions appeared small and limited to probability thresholds between 5 and 7%.ConclusionsThe evaluated models had modest predictive performance, with key limitations such as poor calibration (likely due to overfitting in the original development datasets), substantial heterogeneity, and small net benefit across settings. The evidence to support the use of these prediction models for pre-eclampsia in clinical decision-making is limited. Any models that we could not validate should be examined in terms of their predictive performance, net benefit, and heterogeneity across multiple UK settings before consideration for use in practice.Trial registrationPROSPERO ID: CRD42015029349 .

Project description:BackgroundSome studies have suggested a link between antihypertensive medication and cancer, but the evidence is so far inconclusive. Thus, we aimed to investigate this association in a large individual patient data meta-analysis of randomised clinical trials.MethodsWe searched PubMed, MEDLINE, The Cochrane Central Register of Controlled Trials, and ClinicalTrials.gov from Jan 1, 1966, to Sept 1, 2019, to identify potentially eligible randomised controlled trials. Eligible studies were randomised controlled trials comparing one blood pressure lowering drug class with a placebo, inactive control, or other blood pressure lowering drug. We also required that trials had at least 1000 participant years of follow-up in each treatment group. Trials without cancer event information were excluded. We requested individual participant data from the authors of eligible trials. We pooled individual participant-level data from eligible trials and assessed the effects of angiotensin-converting enzyme inhibitors (ACEIs), angiotensin II receptor blockers (ARBs), β blockers, calcium channel blockers, and thiazide diuretics on cancer risk in one-stage individual participant data and network meta-analyses. Cause-specific fixed-effects Cox regression models, stratified by trial, were used to calculate hazard ratios (HRs). The primary outcome was any cancer event, defined as the first occurrence of any cancer diagnosed after randomisation. This study is registered with PROSPERO (CRD42018099283).Findings33 trials met the inclusion criteria, and included 260 447 participants with 15 012 cancer events. Median follow-up of included participants was 4·2 years (IQR 3·0-5·0). In the individual participant data meta-analysis comparing each drug class with all other comparators, no associations were identified between any antihypertensive drug class and risk of any cancer (HR 0·99 [95% CI 0·95-1·04] for ACEIs; 0·96 [0·92-1·01] for ARBs; 0·98 [0·89-1·07] for β blockers; 1·01 [0·95-1·07] for thiazides), with the exception of calcium channel blockers (1·06 [1·01-1·11]). In the network meta-analysis comparing drug classes against placebo, we found no excess cancer risk with any drug class (HR 1·00 [95% CI 0·93-1·09] for ACEIs; 0·99 [0·92-1·06] for ARBs; 0·99 [0·89-1·11] for β blockers; 1·04 [0·96-1·13] for calcium channel blockers; 1·00 [0·90-1·10] for thiazides).InterpretationWe found no consistent evidence that antihypertensive medication use had any effect on cancer risk. Although such findings are reassuring, evidence for some comparisons was insufficient to entirely rule out excess risk, in particular for calcium channel blockers.FundingBritish Heart Foundation, National Institute for Health Research, Oxford Martin School.

Project description:BackgroundObservational studies of a putative association between hormonal contraception (HC) and HIV acquisition have produced conflicting results. We conducted an individual participant data (IPD) meta-analysis of studies from sub-Saharan Africa to compare the incidence of HIV infection in women using combined oral contraceptives (COCs) or the injectable progestins depot-medroxyprogesterone acetate (DMPA) or norethisterone enanthate (NET-EN) with women not using HC.Methods and findingsEligible studies measured HC exposure and incident HIV infection prospectively using standardized measures, enrolled women aged 15-49 y, recorded ≥15 incident HIV infections, and measured prespecified covariates. Our primary analysis estimated the adjusted hazard ratio (aHR) using two-stage random effects meta-analysis, controlling for region, marital status, age, number of sex partners, and condom use. We included 18 studies, including 37,124 women (43,613 woman-years) and 1,830 incident HIV infections. Relative to no HC use, the aHR for HIV acquisition was 1.50 (95% CI 1.24-1.83) for DMPA use, 1.24 (95% CI 0.84-1.82) for NET-EN use, and 1.03 (95% CI 0.88-1.20) for COC use. Between-study heterogeneity was mild (I(2) < 50%). DMPA use was associated with increased HIV acquisition compared with COC use (aHR 1.43, 95% CI 1.23-1.67) and NET-EN use (aHR 1.32, 95% CI 1.08-1.61). Effect estimates were attenuated for studies at lower risk of methodological bias (compared with no HC use, aHR for DMPA use 1.22, 95% CI 0.99-1.50; for NET-EN use 0.67, 95% CI 0.47-0.96; and for COC use 0.91, 95% CI 0.73-1.41) compared to those at higher risk of bias (p(interaction) = 0.003). Neither age nor herpes simplex virus type 2 infection status modified the HC-HIV relationship.ConclusionsThis IPD meta-analysis found no evidence that COC or NET-EN use increases women's risk of HIV but adds to the evidence that DMPA may increase HIV risk, underscoring the need for additional safe and effective contraceptive options for women at high HIV risk. A randomized controlled trial would provide more definitive evidence about the effects of hormonal contraception, particularly DMPA, on HIV risk.

Project description:OBJECTIVE:To investigate whether the genetic burden of type 2 diabetes modifies the association between the quality of dietary fat and the incidence of type 2 diabetes. DESIGN:Individual participant data meta-analysis. DATA SOURCES:Eligible prospective cohort studies were systematically sourced from studies published between January 1970 and February 2017 through electronic searches in major medical databases (Medline, Embase, and Scopus) and discussion with investigators. REVIEW METHODS:Data from cohort studies or multicohort consortia with available genome-wide genetic data and information about the quality of dietary fat and the incidence of type 2 diabetes in participants of European descent was sought. Prospective cohorts that had accrued five or more years of follow-up were included. The type 2 diabetes genetic risk profile was characterized by a 68-variant polygenic risk score weighted by published effect sizes. Diet was recorded by using validated cohort-specific dietary assessment tools. Outcome measures were summary adjusted hazard ratios of incident type 2 diabetes for polygenic risk score, isocaloric replacement of carbohydrate (refined starch and sugars) with types of fat, and the interaction of types of fat with polygenic risk score. RESULTS:Of 102 305 participants from 15 prospective cohort studies, 20 015 type 2 diabetes cases were documented after a median follow-up of 12 years (interquartile range 9.4-14.2). The hazard ratio of type 2 diabetes per increment of 10 risk alleles in the polygenic risk score was 1.64 (95% confidence interval 1.54 to 1.75, I2=7.1%, τ2=0.003). The increase of polyunsaturated fat and total omega 6 polyunsaturated fat intake in place of carbohydrate was associated with a lower risk of type 2 diabetes, with hazard ratios of 0.90 (0.82 to 0.98, I2=18.0%, τ2=0.006; per 5% of energy) and 0.99 (0.97 to 1.00, I2=58.8%, τ2=0.001; per increment of 1 g/d), respectively. Increasing monounsaturated fat in place of carbohydrate was associated with a higher risk of type 2 diabetes (hazard ratio 1.10, 95% confidence interval 1.01 to 1.19, I2=25.9%, τ2=0.006; per 5% of energy). Evidence of small study effects was detected for the overall association of polyunsaturated fat with the risk of type 2 diabetes, but not for the omega 6 polyunsaturated fat and monounsaturated fat associations. Significant interactions between dietary fat and polygenic risk score on the risk of type 2 diabetes (P>0.05 for interaction) were not observed. CONCLUSIONS:These data indicate that genetic burden and the quality of dietary fat are each associated with the incidence of type 2 diabetes. The findings do not support tailoring recommendations on the quality of dietary fat to individual type 2 diabetes genetic risk profiles for the primary prevention of type 2 diabetes, and suggest that dietary fat is associated with the risk of type 2 diabetes across the spectrum of type 2 diabetes genetic risk.

Dataset Information

Developing and validating risk prediction models in an individual participant data meta-analysis.

Publications

Developing and validating risk prediction models in an individual participant data meta-analysis.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets