Sample size considerations for the external validation of a multivariable prognostic model: a resampling study.
Ontology highlight
ABSTRACT:
SUBMITTER: Collins GS
PROVIDER: S-EPMC4738418 | biostudies-literature | 2016 Jan
REPOSITORIES: biostudies-literature
Publications
Sample size considerations for the external validation of a multivariable prognostic model: a resampling study.
Collins Gary S GS Ogundimu Emmanuel O EO Altman Douglas G DG
Statistics in medicine 20151109 2
After developing a prognostic model, it is essential to evaluate the performance of the model in samples independent from those used to develop the model, which is often referred to as external validation. However, despite its importance, very little is known about the sample size requirements for conducting an external validation. Using a large real data set and resampling methods, we investigate the impact of sample size on the performance of six published prognostic models. Focussing on unbia ...[more]
Similar Datasets
Project description:Published models inconsistently associate glioblastoma size with overall survival (OS). This study aimed to investigate the prognostic effect of tumour size in a large cohort of patients diagnosed with GBM and interrogate how sample size and non-linear transformations may impact on the likelihood of finding a prognostic effect. In total, 279 patients with a IDH-wildtype unifocal WHO grade 4 GBM between 2014 and 2020 from a retrospective cohort were included. Uni-/multivariable association between core volume, whole volume (CV and WV), and diameter with OS was assessed with (1) Cox proportional hazard models +/- log transformation and (2) resampling with 1,000,000 repetitions and varying sample size to identify the percentage of models, which showed a significant effect of tumour size. Models adjusted for operation type and a diameter model adjusted for all clinical variables remained significant (p = 0.03). Multivariable resampling increased the significant effects (p < 0.05) of all size variables as sample size increased. Log transformation also had a large effect on the chances of a prognostic effect of WV. For models adjusted for operation type, 19.5% of WV vs. 26.3% log-WV (n = 50) and 69.9% WV and 89.9% log-WV (n = 279) were significant. In this large well-curated cohort, multivariable modelling and resampling suggest tumour volume is prognostic at larger sample sizes and with log transformation for WV.
| S-EPMC11011077 | biostudies-literature
Project description:Clinical cohorts with time-to-event endpoints are increasingly characterized by measurements of a number of single nucleotide polymorphisms that is by a magnitude larger than the number of measurements typically considered at the gene level. At the same time, the size of clinical cohorts often is still limited, calling for novel analysis strategies for identifying potentially prognostic SNPs that can help to better characterize disease processes. We propose such a strategy, drawing on univariate testing ideas from epidemiological case-controls studies on the one hand, and multivariable regression techniques as developed for gene expression data on the other hand. In particular, we focus on stable selection of a small set of SNPs and corresponding genes for subsequent validation. For univariate analysis, a permutation-based approach is proposed to test at the gene level. We use regularized multivariable regression models for considering all SNPs simultaneously and selecting a small set of potentially important prognostic SNPs. Stability is judged according to resampling inclusion frequencies for both the univariate and the multivariable approach. The overall strategy is illustrated with data from a cohort of acute myeloid leukemia patients and explored in a simulation study. The multivariable approach is seen to automatically focus on a smaller set of SNPs compared to the univariate approach, roughly in line with blocks of correlated SNPs. This more targeted extraction of SNPs results in more stable selection at the SNP as well as at the gene level. Thus, the multivariable regression approach with resampling provides a perspective in the proposed analysis strategy for SNP data in clinical cohorts highlighting what can be added by regularized regression techniques compared to univariate analyses.
| S-EPMC4861340 | biostudies-literature
Project description:Risk-prediction models for health outcomes are used in practice as part of clinical decision-making, and it is essential that their performance be externally validated. An important aspect in the design of a validation study is choosing an adequate sample size. In this paper, we investigate the sample size requirements for validation studies with binary outcomes to estimate measures of predictive performance (C-statistic for discrimination and calibration slope and calibration in the large). We aim for sufficient precision in the estimated measures. In addition, we investigate the sample size to achieve sufficient power to detect a difference from a target value. Under normality assumptions on the distribution of the linear predictor, we obtain simple estimators for sample size calculations based on the measures above. Simulation studies show that the estimators perform well for common values of the C-statistic and outcome prevalence when the linear predictor is marginally Normal. Their performance deteriorates only slightly when the normality assumptions are violated. We also propose estimators which do not require normality assumptions but require specification of the marginal distribution of the linear predictor and require the use of numerical integration. These estimators were also seen to perform very well under marginal normality. Our sample size equations require a specified standard error (SE) and the anticipated C-statistic and outcome prevalence. The sample size requirement varies according to the prognostic strength of the model, outcome prevalence, choice of the performance measure and study objective. For example, to achieve an SE < 0.025 for the C-statistic, 60-170 events are required if the true C-statistic and outcome prevalence are between 0.64-0.85 and 0.05-0.3, respectively. For the calibration slope and calibration in the large, achieving SE < 0.15 would require 40-280 and 50-100 events, respectively. Our estimators may also be used for survival outcomes when the proportion of censored observations is high.
| S-EPMC8529102 | biostudies-literature
Project description:BACKGROUND:Using two large datasets from Dorset, we previously reported an internally validated multivariable risk model for predicting the risk of GI malignancy in IDA-the IDIOM score. The aim of this retrospective observational study was to validate the IDIOM model using two independent external datasets. METHODS:The external validation datasets were collected, in a secondary care setting, by different investigators from cohorts in Oxford and Sheffield derived under different circumstances, comprising 1117 and 474 patients with confirmed IDA respectively. The data were anonymised prior to analysis. The predictive performance of the original model was evaluated by estimating measures of calibration, discrimination and clinical utility using the validation datasets. RESULTS:The discrimination of the original model using the external validation data was 70% (95% CI 65, 75) for the Oxford dataset and 70% (95% CI 61, 79) for the Sheffield dataset. The analysis of mean, weak, flexible and across the risk groups' calibration showed no tendency for under or over-estimated risks in the combined validation data. Decision curve analysis demonstrated the clinical value of the IDIOM model with a net benefit that is higher than 'investigate all' and 'investigate no-one' strategies up to a threshold of 18% in the combined validation data, using a risk cut-off of around 1.2% to categorise patients into the very low risk group showed that none of the patients stratified in this risk group proved to have GI cancer on investigation in the validation datasets. CONCLUSION:This external validation exercise has shown promising results for the IDIOM model in predicting the risk of underlying GI malignancy in independent IDA datasets collected in different clinical settings.
| S-EPMC8672477 | biostudies-literature
Project description:Background. National valuation studies are costly, with ∼1000 face-to-face interviews recommended, and some countries may deem such studies infeasible. Building on previous studies exploring sample size, we determined the effect of sample size and alternative model specifications on prediction accuracy of modeled coefficients in EQ-5D-5L value set generating regression analyses. Methods. Data sets (n = 50 to ∼1000) were simulated from 3 valuation studies, resampled at the respondent level and randomly drawn 1000 times with replacement. We estimated utilities for each subsample with leave-one-out at the block level using regression models (8 or 20 parameter; with or without a random intercept; time tradeoff [TTO] data only or TTO + discrete choice experiment [DCE] data). Prediction accuracy, root mean square error (RMSE), was calculated by comparing to censored mean predicted values to the left-out block in the full data set. Linear regression was used to estimate the relative effect of changes in sample size and each model specification. Results. Results showed that doubling the sample size decreased RMSE by on average 0.012. Effects of other model specifications were smaller but can when combined compensate for loss in prediction accuracy from a small sample size. For models using TTO data only, 8-parameter models clearly outperformed 20-parameter models. Adding a random intercept, or including DCE responses, also improved mean RMSE, most prominently for variants of the 20-parameter models. Conclusions. The prediction accuracy impact of further increases in sample size after 300 to 500 were smaller than the impact of combining alternative modeling choices. Hybrid modeling, use of constrained models, and inclusion of random intercepts all substantially improve the expected prediction accuracy. Beyond a minimum of 300 to 500 respondents, the sample size may be better informed by other considerations, such as legitimacy and representativeness, than by the technical prediction accuracy achievable.HighlightsIncreases in sample size beyond a minimum in the range of 300 to 500 respondents provide smaller gains in expected prediction accuracy than alternative modeling approaches.Constrained, nonlinear models; time tradeoff + discrete choice experiment hybrid modeling; and including a random intercept all improved the prediction accuracy of models estimating values for the EQ-5D-5L based on data from 3 different valuation studies.The tested modeling choices can compensate for smaller sample sizes.
| S-EPMC8905070 | biostudies-literature
Project description:BackgroundLinear regression analysis is a widely used statistical technique in practical applications. For planning and appraising validation studies of simple linear regression, an approximate sample size formula has been proposed for the joint test of intercept and slope coefficients.MethodsThe purpose of this article is to reveal the potential drawback of the existing approximation and to provide an alternative and exact solution of power and sample size calculations for model validation in linear regression analysis.ResultsA fetal weight example is included to illustrate the underlying discrepancy between the exact and approximate methods. Moreover, extensive numerical assessments were conducted to examine the relative performance of the two distinct procedures.ConclusionsThe results show that the exact approach has a distinct advantage over the current method with greater accuracy and high robustness.
| S-EPMC6416874 | biostudies-literature
Project description:Statistical models are often fitted to obtain a concise description of the association of an outcome variable with some covariates. Even if background knowledge is available to guide preselection of covariates, stepwise variable selection is commonly applied to remove irrelevant ones. This practice may introduce additional variability and selection is rarely certain. However, these issues are often ignored and model stability is not questioned. Several resampling-based measures were proposed to describe model stability, including variable inclusion frequencies (VIFs), model selection frequencies, relative conditional bias (RCB), and root mean squared difference ratio (RMSDR). The latter two were recently proposed to assess bias and variance inflation induced by variable selection. Here, we study the consistency and accuracy of resampling estimates of these measures and the optimal choice of the resampling technique. In particular, we compare subsampling and bootstrapping for assessing stability of linear, logistic, and Cox models obtained by backward elimination in a simulation study. Moreover, we exemplify the estimation and interpretation of all suggested measures in a study on cardiovascular risk. The VIF and the model selection frequency are only consistently estimated in the subsampling approach. By contrast, the bootstrap is advantageous in terms of bias and precision for estimating the RCB as well as the RMSDR. Though, unbiased estimation of the latter quantity requires independence of covariates, which is rarely encountered in practice. Our study stresses the importance of addressing model stability after variable selection and shows how to cope with it.
| S-EPMC7820988 | biostudies-literature
Project description:ObjectiveTo study the performance of the 'van Loendersloot' prognostic model for our clinic's in vitro fertilisation (IVF) in its original version, the refitted version and in an adapted version replacing previous by current cycle IVF laboratory variables.MethodsThis retrospective cohort study in our academic tertiary fertility clinic analysed 1281 IVF cycles of 591 couples, who completed at least one 2nd-6th IVF cycle with own fresh gametes after a previous IVF cycle with the same partner in our clinic between 2010 and 2018. The outcome of interest was the chance on a live birth after one complete IVF cycle (including all fresh and frozen embryo transfers from the same episode of ovarian stimulation). Model performance was expressed in terms of discrimination (c-statistics) and calibration (calibration model, comparison of prognosis to observed ratios of five disjoint groups formed by the quintiles of the IVF prognoses and a calibration plot).ResultsA total of 344 live births were obtained (26.9%). External validation of the original van Loendersloot model showed a poor c-statistic of 0.64 (95% CI: 0.61 to 0.68) and an underestimation of IVF success. The refitted and the adapted models showed c-statistics of respectively 0.68 (95% CI: 0.65 to 0.71) and 0.74 (95% CI: 0.70 to 0.77). Similar c-statistics were found with cross-validation. Both models showed a good calibration model; refitted model: intercept=0.00 (95% CI: -0.23 to 0.23) and slope=1.00 (95% CI: 0.79 to 1.21); adapted model: intercept=0.00 (95% CI: -0.18 to 0.18) and slope=1.00 (95% CI: 0.83 to 1.17). Prognoses and observed success rates of the disjoint groups matched well for the refitted model and even better for the adapted model.ConclusionExternal validation of the original van Loendersloot model indicated that model updating was recommended. The good performance of the refitted and adapted models allows informing couples about their IVF prognosis prior to an IVF cycle and at the time of embryo transfer. Whether this has an impact on couple's expected success rates, distress and IVF discontinuation can now be studied.
| S-EPMC7545639 | biostudies-literature
Project description:BackgroundPrognostic stratification is the cornerstone of management in nonmetastatic prostate cancer (PCa). However, existing prognostic models are inadequate-often using treatment outcomes rather than survival, stratifying by broad heterogeneous groups and using heavily treated cohorts. To address this unmet need, we developed an individualised prognostic model that contextualises PCa-specific mortality (PCSM) against other cause mortality, and estimates the impact of treatment on survival.Methods and findingsUsing records from the United Kingdom National Cancer Registration and Analysis Service (NCRAS), data were collated for 10,089 men diagnosed with nonmetastatic PCa between 2000 and 2010 in Eastern England. Median follow-up was 9.8 years with 3,829 deaths (1,202 PCa specific). Totals of 19.8%, 14.1%, 34.6%, and 31.5% of men underwent conservative management, prostatectomy, radiotherapy (RT), and androgen deprivation monotherapy, respectively. A total of 2,546 men diagnosed in Singapore over a similar time period represented an external validation cohort. Data were randomly split 70:30 into model development and validation cohorts. Fifteen-year PCSM and non-PCa mortality (NPCM) were explored using separate multivariable Cox models within a competing risks framework. Fractional polynomials (FPs) were utilised to fit continuous variables and baseline hazards. Model accuracy was assessed by discrimination and calibration using the Harrell C-index and chi-squared goodness of fit, respectively, within both validation cohorts. A multivariable model estimating individualised 10- and 15-year survival outcomes was constructed combining age, prostate-specific antigen (PSA), histological grade, biopsy core involvement, stage, and primary treatment, which were each independent prognostic factors for PCSM, and age and comorbidity, which were prognostic for NPCM. The model demonstrated good discrimination, with a C-index of 0.84 (95% CI: 0.82-0.86) and 0.84 (95% CI: 0.80-0.87) for 15-year PCSM in the UK and Singapore validation cohorts, respectively, comparing favourably to international risk-stratification criteria. Discrimination was maintained for overall mortality, with C-index 0.77 (95% CI: 0.75-0.78) and 0.76 (95% CI: 0.73-0.78). The model was well calibrated with no significant difference between predicted and observed PCa-specific (p = 0.19) or overall deaths (p = 0.43) in the UK cohort. Key study limitations were a relatively small external validation cohort, an inability to account for delayed changes to treatment beyond 12 months, and an absence of tumour-stage subclassifications.Conclusions'PREDICT Prostate' is an individualised multivariable PCa prognostic model built from baseline diagnostic information and the first to our knowledge that models potential treatment benefits on overall survival. Prognostic power is high despite using only routinely collected clinicopathological information.
| S-EPMC6413892 | biostudies-literature
Project description:Background and Objectives: Acute kidney injury (AKI) that results from ischemia is a common clinical syndrome and correlates with high morbidity and mortality among hospitalized patients. However, a clinical tool to predict mortality risk of ischemic AKI is not available. In this study, we aimed to develop and validate models to predict the 30-day and 1-year mortality risk of hospitalized patients with ischemic AKI. Methods A total of 1,836 admissions with ischemic AKI were recruited from 277,898 inpatients admitted to three affiliated tertiary general hospitals of Central South University in China between January 2015 and December 2015. Patients in the final analysis were followed up for 1 year. Study patients were randomly divided in a 7:3 ratio to form the training cohort and validation cohort. Multivariable regression analyses were used for developing mortality prediction models. Results Hepatorenal syndrome, shock, central nervous system failure, Charlson comorbidity index (≥2 points), mechanical ventilation, renal function at discharge were independent risk factors for 30-day mortality after ischemic AKI, while malignancy, sepsis, heart failure, liver failure, Charlson comorbidity index (≥2 points), mechanical ventilation, and renal function at discharge were predictors for 1-year mortality. The area under the receiver operating characteristic curves (AUROCs) of 30-day prediction model were 0.878 (95% confidence interval (CI): 0.849-0.908) in the training cohort and 0.867 (95% CI: 0.820–0.913) in the validation cohort. The AUROCs of the 1-year mortality prediction in the training and validation cohort were 0.803 (95% CI: 0.772–0.834) and 0.788 (95% CI: 0.741–0.835), respectively. Conclusion Our easily applied prediction models can effectively identify individuals at high mortality risk within 30 days or 1 year in hospitalized patients with ischemic AKI. It can guide the optimal clinical management to minimize mortality after an episode of ischemic AKI.
| S-EPMC9420861 | biostudies-literature