Dataset Information

Testing calibration of risk models at extremes of disease risk.

ABSTRACT: Risk-prediction models need careful calibration to ensure they produce unbiased estimates of risk for subjects in the underlying population given their risk-factor profiles. As subjects with extreme high or low risk may be the most affected by knowledge of their risk estimates, checking the adequacy of risk models at the extremes of risk is very important for clinical applications. We propose a new approach to test model calibration targeted toward extremes of disease risk distribution where standard goodness-of-fit tests may lack power due to sparseness of data. We construct a test statistic based on model residuals summed over only those individuals who pass high and/or low risk thresholds and then maximize the test statistic over different risk thresholds. We derive an asymptotic distribution for the max-test statistic based on analytic derivation of the variance-covariance function of the underlying Gaussian process. The method is applied to a large case-control study of breast cancer to examine joint effects of common single nucleotide polymorphisms (SNPs) discovered through recent genome-wide association studies. The analysis clearly indicates a non-additive effect of the SNPs on the scale of absolute risk, but an excellent fit for the linear-logistic model even at the extremes of risks.

SUBMITTER: Song M

PROVIDER: S-EPMC4263225 | biostudies-literature | 2015 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Testing calibration of risk models at extremes of disease risk.

Song Minsun M Kraft Peter P Joshi Amit D AD Barrdahl Myrto M Chatterjee Nilanjan N

Biostatistics (Oxford, England) 20140714 1

Risk-prediction models need careful calibration to ensure they produce unbiased estimates of risk for subjects in the underlying population given their risk-factor profiles. As subjects with extreme high or low risk may be the most affected by knowledge of their risk estimates, checking the adequacy of risk models at the extremes of risk is very important for clinical applications. We propose a new approach to test model calibration targeted toward extremes of disease risk distribution where sta ...[more]

PMID: 25027274

Similar Datasets

Project description:BackgroundAssessing calibration-the agreement between estimated risk and observed proportions-is an important component of deriving and validating clinical prediction models. Methods for assessing the calibration of prognostic models for use with competing risk data have received little attention.MethodsWe propose a method for graphically assessing the calibration of competing risk regression models. Our proposed method can be used to assess the calibration of any model for estimating incidence in the presence of competing risk (e.g., a Fine-Gray subdistribution hazard model; a combination of cause-specific hazard functions; or a random survival forest). Our method is based on using the Fine-Gray subdistribution hazard model to regress the cumulative incidence function of the cause-specific outcome of interest on the predicted outcome risk of the model whose calibration we want to assess. We provide modifications of the integrated calibration index (ICI), of E50 and of E90, which are numerical calibration metrics, for use with competing risk data. We conducted a series of Monte Carlo simulations to evaluate the performance of these calibration measures when the underlying model has been correctly specified and when the model was mis-specified and when the incidence of the cause-specific outcome differed between the derivation and validation samples. We illustrated the usefulness of calibration curves and the numerical calibration metrics by comparing the calibration of a Fine-Gray subdistribution hazards regression model with that of random survival forests for predicting cardiovascular mortality in patients hospitalized with heart failure.ResultsThe simulations indicated that the method for constructing graphical calibration curves and the associated calibration metrics performed as desired. We also demonstrated that the numerical calibration metrics can be used as optimization criteria when tuning machine learning methods for competing risk outcomes.ConclusionsThe calibration curves and numeric calibration metrics permit a comprehensive comparison of the calibration of different competing risk models.

Project description:Polycystic kidney diseases (PKDs) are genetic disorders characterized by the formation and expansion of numerous fluid-filled renal cysts, damaging normal parenchyma and often leading to kidney failure. Although PKDs comprise a broad range of different diseases, with substantial genetic and phenotypic heterogeneity, an association with primary cilia represents a common theme. Great strides have been made in the identification of causative genes, furthering our understanding of the genetic complexity and disease mechanisms, but only one therapy so far has shown success in clinical trials and advanced to US Food and Drug Administration approval. A key step in understanding disease pathogenesis and testing potential therapeutics is developing orthologous experimental models that accurately recapitulate the human phenotype. This has been particularly important for PKDs because cellular models have been of limited value; however, the advent of organoid usage has expanded capabilities in this area but does not negate the need for whole-organism models where renal function can be assessed. Animal model generation is further complicated in the most common disease type, autosomal dominant PKD, by homozygous lethality and a very limited cystic phenotype in heterozygotes while for autosomal recessive PKD, mouse models have a delayed and modest kidney disease, in contrast to humans. However, for autosomal dominant PKD, the use of conditional/inducible and dosage models have resulted in some of the best disease models in nephrology. These have been used to help understand pathogenesis, to facilitate genetic interaction studies, and to perform preclinical testing. Whereas for autosomal recessive PKD, using alternative species and digenic models has partially overcome these deficiencies. Here, we review the experimental models that are currently available and most valuable for therapeutic testing in PKD, their applications, success in preclinical trials, advantages and limitations, and where further improvements are needed.

Dataset Information

Testing calibration of risk models at extremes of disease risk.

Publications

Testing calibration of risk models at extremes of disease risk.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets