Dataset Information

Meta-analysis of prediction model performance across multiple studies: Which scale helps ensure between-study normality for the C-statistic and calibration measures?

ABSTRACT: If individual participant data are available from multiple studies or clusters, then a prediction model can be externally validated multiple times. This allows the model's discrimination and calibration performance to be examined across different settings. Random-effects meta-analysis can then be used to quantify overall (average) performance and heterogeneity in performance. This typically assumes a normal distribution of 'true' performance across studies. We conducted a simulation study to examine this normality assumption for various performance measures relating to a logistic regression prediction model. We simulated data across multiple studies with varying degrees of variability in baseline risk or predictor effects and then evaluated the shape of the between-study distribution in the C-statistic, calibration slope, calibration-in-the-large, and E/O statistic, and possible transformations thereof. We found that a normal between-study distribution was usually reasonable for the calibration slope and calibration-in-the-large; however, the distributions of the C-statistic and E/O were often skewed across studies, particularly in settings with large variability in the predictor effects. Normality was vastly improved when using the logit transformation for the C-statistic and the log transformation for E/O, and therefore we recommend these scales to be used for meta-analysis. An illustrated example is given using a random-effects meta-analysis of the performance of QRISK2 across 25 general practices.

SUBMITTER: Snell KI

PROVIDER: S-EPMC6193210 | biostudies-literature | 2018 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Meta-analysis of prediction model performance across multiple studies: Which scale helps ensure between-study normality for the C-statistic and calibration measures?

Snell Kym Ie KI Ensor Joie J Debray Thomas Pa TP Moons Karel Gm KG Riley Richard D RD

Statistical methods in medical research 20170508 11

If individual participant data are available from multiple studies or clusters, then a prediction model can be externally validated multiple times. This allows the model's discrimination and calibration performance to be examined across different settings. Random-effects meta-analysis can then be used to quantify overall (average) performance and heterogeneity in performance. This typically assumes a normal distribution of 'true' performance across studies. We conducted a simulation study to exa ...[more]

PMID: 28480827

Similar Datasets

Project description:BackgroundMicroarray experiments rely on several critical steps that may introduce biases and uncertainty in downstream analyses. These steps include mRNA sample extraction, amplification and labelling, hybridization, and scanning causing chip-specific systematic variations on the raw intensity level. Also the chosen array-type and the up-to-dateness of the genomic information probed on the chip affect the quality of the expression measures. In the accompanying publication we presented theory and algorithm of the so-called hook method which aims at correcting expression data for systematic biases using a series of new chip characteristics.ResultsIn this publication we summarize the essential chip characteristics provided by this method, analyze special benchmark experiments to estimate transcript related expression measures and illustrate the potency of the method to detect and to quantify the quality of a particular hybridization. It is shown that our single-chip approach provides expression measures responding linearly on changes of the transcript concentration over three orders of magnitude. In addition, the method calculates a detection call judging the relation between the signal and the detection limit of the particular measurement. The performance of the method in the context of different chip generations and probe set assignments is illustrated. The hook method characterizes the RNA-quality in terms of the 3'/5'-amplification bias and the sample-specific calling rate. We show that the proper judgement of these effects requires the disentanglement of non-specific and specific hybridization which, otherwise, can lead to misinterpretations of expression changes. The consequences of modifying probe/target interactions by either changing the labelling protocol or by substituting RNA by DNA targets are demonstrated.ConclusionThe single-chip based hook-method provides accurate expression estimates and chip-summary characteristics using the natural metrics given by the hybridization reaction with the potency to develop new standards for microarray quality control and calibration.

Project description:ObjectivesAlthough cardiovascular disease (CVD) is a common comorbidity associated with chronic obstructive pulmonary disease (COPD), it is unknown how to improve prediction of cardiovascular (CV) risk in individuals with COPD. Traditional CV risk scores have been tested in different populations but not uniquely in COPD. The potential of alternative markers to improve CV risk prediction in individuals with COPD is unknown. We aimed to determine the predictive value of conventional CVD risk factors in COPD and to determine if additional markers improve prediction beyond conventional factors.DesignData from the Evaluation of the Role of Inflammation in Chronic Airways disease cohort, which enrolled 729 individuals with Global Initiative for Chronic Obstructive Lung Disease (GOLD) stage II-IV COPD were used. Linked hospital episode statistics and survival data were prospectively collected for a median 4.6 years of follow-up.SettingFive UK centres interested in COPD.ParticipantsPopulation-based sample including 714 individuals with spirometry-defined COPD, smoked at least 10 pack years and who were clinically stable for >4 weeks.InterventionsBaseline measurements included aortic pulse wave velocity (aPWV), carotid intima-media thickness (CIMT), C reactive protein (CRP), fibrinogen, spirometry and Body mass index, airflow Obstruction, Dyspnoea and Exercise capacity (BODE) Index, 6 min walk test (6MWT) and 4 m gait speed (4MGS) test.Primary and secondary outcome measuresNew occurrence (first event) of fatal or non-fatal hospitalised CVD, and all-cause and cause-specific mortality.ResultsOut of 714 participants, 192 (27%) had CV hospitalisation and 6 died due to CVD. The overall CV risk model C-statistic was 0.689 (95% CI 0.688 to 0.691). aPWV and CIMT neither had an association with study outcome nor improved model prediction. CRP, fibrinogen, GOLD stage, BODE Index, 4MGS and 6MWT were associated with the outcome, independently of conventional risk factors (p<0.05 for all). However, only 6MWT improved model discrimination (C=0.727, 95% CI 0.726 to 0.728).ConclusionPoor physical performance defined by the 6MWT improves prediction of CV hospitalisation in individuals with COPD.Trial registration numberID 11101.

Dataset Information

Meta-analysis of prediction model performance across multiple studies: Which scale helps ensure between-study normality for the C-statistic and calibration measures?

Publications

Meta-analysis of prediction model performance across multiple studies: Which scale helps ensure between-study normality for the C-statistic and calibration measures?

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets