Validation of discrete time-to-event prediction models in the presence of competing risks.
Ontology highlight
ABSTRACT:
SUBMITTER: Heyard R
PROVIDER: S-EPMC7217187 | biostudies-literature | 2020 May
REPOSITORIES: biostudies-literature
Publications
Validation of discrete time-to-event prediction models in the presence of competing risks.
Heyard Rachel R Timsit Jean-François JF Held Leonhard L
Biometrical journal. Biometrische Zeitschrift 20190731 3
Clinical prediction models play a key role in risk stratification, therapy assignment and many other fields of medical decision making. Before they can enter clinical practice, their usefulness has to be demonstrated using systematic validation. Methods to assess their predictive performance have been proposed for continuous, binary, and time-to-event outcomes, but the literature on validation methods for discrete time-to-event models with competing risks is sparse. The present paper tries to fi ...[more]
Similar Datasets
Project description:BACKGROUND:In time-to-event analyses, there is limited guidance on when persons who are lost to follow-up (LTFU) should be censored. METHODS:We simulated bias in risk estimates for: (1) a composite event of measured (outcome only observable in a patient encounter) and captured events (outcome observable outside a patient encounter); and a (2) measured or (3) captured event in the presence of a competing event of the other type, under three censoring strategies: (i) censor at the last study encounter; (ii) censor when LTFU definition is met; and (iii) a new, hybrid censoring strategy. We demonstrate the real-world impact of this decision by estimating: (1) time to acquired immune deficiency syndrome (AIDS) diagnosis or death, (2) time to initiation of antiretroviral therapy (ART), and (3) time to death before ART initiation among adults engaged in HIV care. RESULTS:For (1) our hybrid censoring strategy was least biased. In our example, 5-year risk of AIDS or death was overestimated using last-encounter censoring (25%) and under-estimated using LTFU-definition censoring (21%), compared with results from our hybrid approach (24%). Last-encounter censoring was least biased for (2). When estimating 5-year risk of ART initiation, LTFU-definition censoring underestimated risk (80% vs. 85% using last-encounter censoring). LTFU-definition censoring was least biased for (3). When estimating 5-year risk of death before ART initiation, last-encounter censoring overestimated risk (5.2% vs. 4.7% using LTFU-definition censoring). CONCLUSIONS:The least biased censoring strategy for time-to-event analyses in the presence of LTFU depends on the event and estimand of interest.
| S-EPMC6768756 | biostudies-literature
Project description:BackgroundIn health research, several chronic diseases are susceptible to competing risks (CRs). Initially, statistical models (SM) were developed to estimate the cumulative incidence of an event in the presence of CRs. As recently there is a growing interest in applying machine learning (ML) for clinical prediction, these techniques have also been extended to model CRs but literature is limited. Here, our aim is to investigate the potential role of ML versus SM for CRs within non-complex data (small/medium sample size, low dimensional setting).MethodsA dataset with 3826 retrospectively collected patients with extremity soft-tissue sarcoma (eSTS) and nine predictors is used to evaluate model-predictive performance in terms of discrimination and calibration. Two SM (cause-specific Cox, Fine-Gray) and three ML techniques are compared for CRs in a simple clinical setting. ML models include an original partial logistic artificial neural network for CRs (PLANNCR original), a PLANNCR with novel specifications in terms of architecture (PLANNCR extended), and a random survival forest for CRs (RSFCR). The clinical endpoint is the time in years between surgery and disease progression (event of interest) or death (competing event). Time points of interest are 2, 5, and 10 years.ResultsBased on the original eSTS data, 100 bootstrapped training datasets are drawn. Performance of the final models is assessed on validation data (left out samples) by employing as measures the Brier score and the Area Under the Curve (AUC) with CRs. Miscalibration (absolute accuracy error) is also estimated. Results show that the ML models are able to reach a comparable performance versus the SM at 2, 5, and 10 years regarding both Brier score and AUC (95% confidence intervals overlapped). However, the SM are frequently better calibrated.ConclusionsOverall, ML techniques are less practical as they require substantial implementation time (data preprocessing, hyperparameter tuning, computational intensity), whereas regression methods can perform well without the additional workload of model training. As such, for non-complex real life survival data, these techniques should only be applied complementary to SM as exploratory tools of model's performance. More attention to model calibration is urgently needed.
| S-EPMC9951458 | biostudies-literature
Project description:BackgroundEvaluating a candidate marker or developing a model for predicting risk of future conditions is one of the major goals in medicine. However, model development and assessment for a time-to-event outcome may be complicated in the presence of competing risks. In this manuscript, we propose a local and a global estimators of cause-specific AUC for right-censored survival times in the presence of competing risks.MethodsThe local estimator - cause-specific weighted mean rank (cWMR) - is a local average of time-specific observed cause-specific AUCs within a neighborhood of given time t. The global estimator - cause-specific fractional polynomials (cFPL) - is based on modelling the cause-specific AUC as a function of t through fractional polynomials.ResultsWe investigated the performance of the proposed cWMR and cFPL estimators through simulation studies and real-life data analysis. The estimators perform well in small samples, have minimal bias and appropriate coverage.ConclusionsThe local estimator cWMR and the global estimator cFPL will provide computationally efficient options for assessing the prognostic accuracy of markers for time-to-event outcome in the presence of competing risks in many practical settings.
| S-EPMC7456384 | biostudies-literature
Project description:BACKGROUND:Different strategies toward implementing competing risks in discrete-event simulation (DES) models are available. This study aims to provide recommendations regarding modeling approaches that can be defined based on these strategies by performing a quantitative comparison of alternative modeling approaches. METHODS:Four modeling approaches were defined: 1) event-specific distribution (ESD), 2) event-specific probability and distribution (ESPD), 3) unimodal joint distribution and regression model (UDR), and 4) multimodal joint distribution and regression model (MDR). Each modeling approach was applied to uncensored individual patient data in a simulation study and a case study in colorectal cancer. Their performance was assessed in terms of relative event incidence difference, relative absolute event incidence difference, and relative entropy of time-to-event distributions. Differences in health economic outcomes were also illustrated for the case study. RESULTS:In the simulation study, the ESPD and MDR approaches outperformed the ESD and UDR approaches, in terms of both event incidence differences and relative entropy. Disease pathway and data characteristics, such as the number of competing risks and overlap between competing time-to-event distributions, substantially affected the approaches' performance. Although no considerable differences in health economic outcomes were observed, the case study showed that the ESPD approach was most sensitive to low event rates, which negatively affected performance. CONCLUSIONS:Based on overall performance, the recommended modeling approach for implementing competing risks in DES models is the MDR approach, which is defined according to the general strategy of selecting the time-to-event first and the corresponding event second. The ESPD approach is a less complex and equally performing alternative if sufficient observations are available for each competing event (i.e., the internal validity shows appropriate data representation).
| S-EPMC6311678 | biostudies-literature
Project description:BackgroundPrediction models for time-to-event outcomes are commonly used in biomedical research to obtain subject-specific probabilities that aid in making important clinical care decisions. There are several regression and machine learning methods for building these models that have been designed or modified to account for the censoring that occurs in time-to-event data. Discrete-time survival models, which have often been overlooked in the literature, provide an alternative approach for predictive modeling in the presence of censoring with limited loss in predictive accuracy. These models can take advantage of the range of nonparametric machine learning classification algorithms and their available software to predict survival outcomes.MethodsDiscrete-time survival models are applied to a person-period data set to predict the hazard of experiencing the failure event in pre-specified time intervals. This framework allows for any binary classification method to be applied to predict these conditional survival probabilities. Using time-dependent performance metrics that account for censoring, we compare the predictions from parametric and machine learning classification approaches applied within the discrete time-to-event framework to those from continuous-time survival prediction models. We outline the process for training and validating discrete-time prediction models, and demonstrate its application using the open-source R statistical programming environment.ResultsUsing publicly available data sets, we show that some discrete-time prediction models achieve better prediction performance than the continuous-time Cox proportional hazards model. Random survival forests, a machine learning algorithm adapted to survival data, also had improved performance compared to the Cox model, but was sometimes outperformed by the discrete-time approaches. In comparing the binary classification methods in the discrete time-to-event framework, the relative performance of the different methods varied depending on the data set.ConclusionsWe present a guide for developing survival prediction models using discrete-time methods and assessing their predictive performance with the aim of encouraging their use in medical research settings. These methods can be applied to data sets that have continuous time-to-event outcomes and multiple clinical predictors. They can also be extended to accommodate new binary classification algorithms as they become available. We provide R code for fitting discrete-time survival prediction models in a github repository.
| S-EPMC9316420 | biostudies-literature
Project description:The concordance probability is a widely used measure to assess discrimination of prognostic models with binary and survival endpoints. We formally define the concordance probability for a prognostic model of the absolute risk of an event of interest in the presence of competing risks and relate it to recently proposed time-dependent area under the receiver operating characteristic curve measures. For right-censored data, we investigate inverse probability of censoring weighted (IPCW) estimates of a truncated concordance index based on a working model for the censoring distribution. We demonstrate consistency and asymptotic normality of the IPCW estimate if the working model is correctly specified and derive an explicit formula for the asymptotic variance under independent censoring. The small sample properties of the estimator are assessed in a simulation study also against misspecification of the working model. We further illustrate the methods by computing the concordance probability for a prognostic model of coronary heart disease (CHD) events in the presence of the competing risk of non-CHD death.
| S-EPMC4059461 | biostudies-literature
Project description:We develop methods for competing risks analysis when individual event times are correlated within clusters. Clustering arises naturally in clinical genetic studies and other settings. We develop a nonparametric estimator of cumulative incidence, and obtain robust pointwise standard errors that account for within-cluster correlation. We modify the two-sample Gray and Pepe-Mori tests for correlated competing risks data, and propose a simple two-sample test of the difference in cumulative incidence at a landmark time. In simulation studies, our estimators are asymptotically unbiased, and the modified test statistics control the type I error. The power of the respective two-sample tests is differentially sensitive to the degree of correlation; the optimal test depends on the alternative hypothesis of interest and the within-cluster correlation. For purposes of illustration, we apply our methods to a family-based prospective cohort study of hereditary breast/ovarian cancer families. For women with BRCA1 mutations, we estimate the cumulative incidence of breast cancer in the presence of competing mortality from ovarian cancer, accounting for significant within-family correlation.
| S-EPMC3125987 | biostudies-literature
Project description:BackgroundVarious hypertension predictive models have been developed worldwide; however, there is no existing predictive model for hypertension among Chinese rural populations.MethodsThis is a 6-year population-based prospective cohort in rural areas of China. Data was collected in 2007-2008 (baseline survey) and 2013-2014 (follow-up survey) from 8319 participants ranging in age from 35 to 74 years old. Specified gender hypertension predictive models were established based on multivariate Cox regression, Artificial Neural Network (ANN), Naive Bayes Classifier (NBC), and Classification and Regression Tree (CART) in the training set. External validation was conducted in the testing set. The estimated models were assessed by discrimination and calibration, respectively.ResultsDuring the follow-up period, 432 men and 604 women developed hypertension in the training set. Assessment for established models in men suggested men office-based model (M1) was better than others. C-index of M1 model in the testing set was 0.771 (95% confidence Interval (CI) = 0.750, 0.791), and calibration χ2 = 6.3057 (P = 0.7090). In women, women office-based model (W1) and ANN were better than the other models assessed. The C-indexes for the W1 model and the ANN model in the testing set were 0.765 (95% CI = 0.746, 0.783) and 0.756 (95% CI = 0.737, 0.775) and the calibrations χ2 were 6.7832 (P = 0.1478) and 4.7447 (P = 0.3145), respectively.ConclusionsNot all machine-learning models performed better than the traditional Cox regression models. The W1 and ANN models for women and M1 model for men have better predictive performance which could potentially be recommended for predicting hypertension risk among rural populations.
| S-EPMC6875679 | biostudies-literature
Project description:This article considers nonparametric methods for studying recurrent disease and death with competing risks. We first point out that comparisons based on the well-known cumulative incidence function can be confounded by different prevalence rates of the competing events, and that comparisons of the conditional distribution of the survival time given the failure event type are more relevant for investigating the prognosis of different patterns of recurrence disease. We then propose nonparametric estimators for the conditional cumulative incidence function as well as the conditional bivariate cumulative incidence function for the bivariate gap times, that is, the time to disease recurrence and the residual lifetime after recurrence. To quantify the association between the two gap times in the competing risks setting, a modified Kendall's tau statistic is proposed. The proposed estimators for the conditional bivariate cumulative incidence distribution and the association measure account for the induced dependent censoring for the second gap time. Uniform consistency and weak convergence of the proposed estimators are established. Hypothesis testing procedures for two-sample comparisons are discussed. Numerical simulation studies with practical sample sizes are conducted to evaluate the performance of the proposed nonparametric estimators and tests. An application to data from a pancreatic cancer study is presented to illustrate the methods developed in this article.
| S-EPMC5014616 | biostudies-literature
Project description:Predicting the occurrence of an adverse event over time is an important issue in clinical medicine. Clinical prediction models and associated points-based risk-scoring systems are popular statistical methods for summarizing the relationship between a multivariable set of patient risk factors and the risk of the occurrence of an adverse event. Points-based risk-scoring systems are popular amongst physicians as they permit a rapid assessment of patient risk without the use of computers or other electronic devices. The use of such points-based risk-scoring systems facilitates evidence-based clinical decision making. There is a growing interest in cause-specific mortality and in non-fatal outcomes. However, when considering these types of outcomes, one must account for competing risks whose occurrence precludes the occurrence of the event of interest. We describe how points-based risk-scoring systems can be developed in the presence of competing events. We illustrate the application of these methods by developing risk-scoring systems for predicting cardiovascular mortality in patients hospitalized with acute myocardial infarction. Code in the R statistical programming language is provided for the implementation of the described methods. © 2016 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.
| S-EPMC5084773 | biostudies-literature