Dataset Information

Extrapolating Parametric Survival Models in Health Technology Assessment: A Simulation Study.

ABSTRACT: Extrapolations of parametric survival models fitted to censored data are routinely used in the assessment of health technologies to estimate mean survival, particularly in diseases that potentially reduce the life expectancy of patients. Akaike's information criterion (AIC) and Bayesian information criterion (BIC) are commonly used in health technology assessment alongside an assessment of plausibility to determine which statistical model best fits the data and should be used for prediction of long-term treatment effects. We compare fit and estimates of restricted mean survival time (RMST) from 8 parametric models and contrast models preferred in terms of AIC, BIC, and log-likelihood, without considering model plausibility. We assess the methods' suitability for selecting a parametric model through simulation of data replicating the follow-up of intervention arms for various time-to-event outcomes from 4 clinical trials. Follow-up was replicated through the consideration of recruitment duration and minimum and maximum follow-up times. Ten thousand simulations of each scenario were performed. We demonstrate that the different methods can result in disagreement over the best model and that it is inappropriate to base model selection solely on goodness-of-fit statistics without consideration of hazard behavior and plausibility of extrapolations. We show that typical trial follow-up can be unsuitable for extrapolation, resulting in unreliable estimation of multiple parameter models, and infer that selecting survival models based only on goodness-of-fit statistics is unsuitable due to the high level of uncertainty in a cost-effectiveness analysis. This article demonstrates the potential problems of overreliance on goodness-of-fit statistics when selecting a model for extrapolation. When follow-up is more mature, BIC appears superior to the other selection methods, selecting models with the most accurate and least biased estimates of RMST.

SUBMITTER: Gallacher D

PROVIDER: S-EPMC7780268 | biostudies-literature | 2021 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Extrapolating Parametric Survival Models in Health Technology Assessment: A Simulation Study.

Gallacher Daniel D Kimani Peter P Stallard Nigel N

Medical decision making : an international journal of the Society for Medical Decision Making 20201207 1

Extrapolations of parametric survival models fitted to censored data are routinely used in the assessment of health technologies to estimate mean survival, particularly in diseases that potentially reduce the life expectancy of patients. Akaike's information criterion (AIC) and Bayesian information criterion (BIC) are commonly used in health technology assessment alongside an assessment of plausibility to determine which statistical model best fits the data and should be used for prediction of l ...[more]

PMID: 33283635

Similar Datasets

Project description:IntroductionExtrapolation of time-to-event data from clinical trials is commonly used in decision models for health technology assessment (HTA). The objective of this study was to assess performance of standard parametric survival analysis techniques for extrapolation of time-to-event data for a single event from clinical trials with limited data due to small samples or short follow-up.MethodsSimulated populations with 50,000 individuals were generated with an exponential hazard rate for the event of interest. A scenario consisted of 5000 repetitions with six sample size groups (30-500 patients) artificially censored after every 10% of events observed. Goodness-of-fit statistics (AIC, BIC) were used to determine the best-fitting among standard parametric distributions (exponential, Weibull, log-normal, log-logistic, generalized gamma, Gompertz). Median survival, one-year survival probability, time horizon (1% survival time, or 99th percentile of survival distribution) and restricted mean survival time (RMST) were compared to population values to assess coverage and error (e.g., mean absolute percentage error).ResultsThe true exponential distribution was correctly identified using goodness-of-fit according to BIC more frequently compared to AIC (average 92% vs 68%). Under-coverage and large errors were observed for all outcomes when distributions were specified by AIC and for time horizon and RMST with BIC. Error in point estimates were found to be strongly associated with sample size and completeness of follow-up. Small samples produced larger average error, even with complete follow-up, than large samples with short follow-up. Correctly specifying the event distribution reduced magnitude of error in larger samples but not in smaller samples.ConclusionsLimited clinical data from small samples, or short follow-up of large samples, produce large error in estimates relevant to HTA regardless of whether the correct distribution is specified. The associated uncertainty in estimated parameters may not capture the true population values. Decision models that base lifetime time horizon on the model's extrapolated output are not likely to reliably estimate mean survival or its uncertainty. For data with an exponential event distribution, BIC more reliably identified the true distribution than AIC. These findings have important implications for health decision modelling and HTA of novel therapies seeking approval with limited evidence.

Project description:BackgroundEvidence-based treatment decisions in medicine are made founded on population-level evidence obtained during randomized clinical trials. In an era of personalized medicine, these decisions should be based on the predicted benefit of a treatment on a patient-level. Survival prediction models play a central role as they incorporate the time-to-event and censoring. In medical applications uncertainty is critical especially when treatments differ in their side effect profiles or costs. Additionally, models must be adapted to local populations without diminishing performance and often without the original training data available due to privacy concern. Both points are supported by Bayesian models-yet they are rarely used. The aim of this work is to evaluate Bayesian parametric survival models on public datasets including cardiology, infectious diseases, and oncology.Materials and methodsBayesian parametric survival models based on the Exponential and Weibull distribution were implemented as a Python package. A linear combination and a neural network were used for predicting the parameters of the distributions. A superiority design was used to assess whether Bayesian models are better than commonly used models such as Cox Proportional Hazards, Random Survival Forest, and Neural Network-based Cox Proportional Hazards. In a secondary analysis, overfitting was compared between these models. An equivalence design was used to assess whether the prediction performance of Bayesian models after model updating using Bayes rule is equivalent to retraining on the full dataset.ResultsIn this study, we found that Bayesian parametric survival models perform as good as state-of-the art models while requiring less hyperparameters to be tuned and providing a measure of the uncertainty of the predictions. In addition, these models were less prone to overfitting. Furthermore, we show that updating these models using Bayes rule yields equivalent performance compared to models trained on combined original and new datasets.ConclusionsBayesian parametric survival models are non-inferior to conventional survival models while requiring less hyperparameter tuning, being less prone to overfitting, and allowing model updating using Bayes rule. Further, the Bayesian models provide a measure of the uncertainty on the statistical inference, and, in particular, on the prediction.

Project description:BackgroundIn health technology assessment, restricted mean survival time and life expectancy are commonly evaluated. Parametric models are typically used for extrapolation. Spline models using a relative survival framework have been shown to estimate life expectancy of cancer patients more reliably; however, more research is needed to assess spline models using an all-cause survival framework and standard parametric models using a relative survival framework.AimTo assess survival extrapolation using standard parametric models and spline models within relative survival and all-cause survival frameworks.MethodsFrom the Swedish Cancer Registry, we identified patients diagnosed with 5 types of cancer (colon, breast, melanoma, prostate, and chronic myeloid leukemia) between 1981 and 1990 with follow-up until 2020. Patients were categorized into 15 cancer cohorts by cancer and age group (18-59, 60-69, and 70-99 y). We right-censored the follow-up at 2, 3, 5, and 10 y and fitted the parametric models within an all-cause and a relative survival framework to extrapolate to 10 y and lifetime in comparison with the observed Kaplan-Meier survival estimates. All cohorts were modeled with 6 standard parametric models (exponential, Weibull, Gompertz, log-logistic, log-normal, and generalized gamma) and 3 spline models (on hazard, odds, and normal scales).ResultsFor predicting 10-y survival, spline models generally performed better than standard parametric models. However, using an all-cause or a relative survival framework did not show any distinct difference. For lifetime survival, extrapolating from a relative survival framework agreed better with the observed survival, particularly using spline models.ConclusionsFor extrapolation to 10 y, we recommend spline models. For extrapolation to lifetime, we suggest extrapolating in a relative survival framework, especially using spline models.HighlightsFor survival extrapolation to 10 y, spline models generally performed better than standard parametric models did. However, using an all-cause or a relative survival framework showed no distinct difference under the same parametric model.Survival extrapolation to lifetime within a relative survival framework agreed well with the observed data, especially using spline models.Extrapolating parametric models within an all-cause survival framework may overestimate survival proportions at lifetime; models for the relative survival approach may underestimate instead.

Project description:BackgroundIn decision modeling with time-to-event data, there are a variety of parametric models that can be used to extrapolate the survival function. Each model implies a different hazard function, and in situations in which there is moderate censoring, this can result in quite different survival projections. External information such as expert opinion on long-term survival can more accurately characterize the uncertainty in these extrapolations.ObjectiveWe present a general and easily implementable approach to incorporate various types of expert opinions into parametric survival models, focusing on opinions about survival at various landmark time points.MethodsExpert opinion is incorporated into parametric survival models using Bayesian and frequentist approaches. In the Bayesian method, expert opinion is included through a loss function and in the frequentist approach by penalizing the likelihood function, although in both cases the core approach is the same. The issue of aggregating multiple expert opinions is also considered.ResultsWe apply this method to data from a leukemia trial and use previously elicited expert opinion on survival probabilities for that particular trial population at years 4 and 5 to inform our analysis. We take a robust approach to modeling expert opinion by using pooled distributions and fit a broad class of parametric models to the data. We also assess statistical goodness of fit of the models to both the observed data and expert opinion.ConclusionsExpert opinions can be implemented in a straightforward manner using this novel approach; however, more work is required on the correct elicitation of these quantities.HighlightsPresentation of a novel and open-source method to incorporate expert opinion into decision modeling.Extends upon earlier work in that expert opinion can be incorporated into a wide range of parametric models.Provides methodological guidance for directly including expert opinion in decision modeling, which is a research focus area in NICE TSD 21.1.

Project description:BackgroundFollowing approximately 10 years from the beginning of Iran's national Health Technology Assessment (HTA) programme, the present study aims to evaluate its success by examining the impact of HTA and identifying the determinant factors leading to the implementation of HTA report results.MethodsThe triangulation method was employed herein. HTA reports were initially identified and their impact and determinant factors were then examined from the perspectives of both researchers (by preparing a questionnaire according to the Payback model and sending it to HTA principle investigators) and stakeholders (semi-structured interviews held with each HTA stakeholder). Simultaneously, the quality of the HTA reports was examined with relevant critical appraisal checklists.ResultsThe impact of 19 equipment technologies and four pharmaceutical technologies were assessed in this study. Twenty researchers replied (response rate, 86.96%) to the questionnaire on the impact of HTA reports from the researcher's perspective. To assess the impact of HTA reports from the stakeholder's perspective, seven policy-makers were chosen and interviewed as the main target audience. The most common step taken to disseminate the results of the HTA projects was publication. Conducting the HTA had taught researchers and their colleagues' new skills and had facilitated the securing of research grants from other organisations. Most reports had used the systematic review method but the relevant details had been scarcely presented regarding outcomes, costs and analysis. The greatest impact of HTA reports on decision-making had been on policy-makers providing and allocating finances. Barriers in stewardship, identification and prioritisation of topics, performance and dissemination of HTA results were the main barriers of implementing HTAs.ConclusionsIn most aspects, the status of HTA impact reports need improvement. Thus far, the barriers and facilitators of the HTA programme in Iran have been investigated in other studies. These findings should be pooled to reach a solution that can be actively applied to the health system to improve the status quo of HTA in Iran.

Dataset Information

Extrapolating Parametric Survival Models in Health Technology Assessment: A Simulation Study.

Publications

Extrapolating Parametric Survival Models in Health Technology Assessment: A Simulation Study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets