Dataset Information

Development and Validation of the TRansparent Uncertainty ASsessmenT (TRUST) Tool for Assessing Uncertainties in Health Economic Decision Models.

ABSTRACT: BACKGROUND:An increasing number of technologies are obtaining marketing authorisation based on sparse evidence, which causes growing uncertainty and risk within health technology reimbursement decision making. To ensure that uncertainty is considered and addressed within health technology assessment (HTA) recommendations, uncertainties need to be identified, included in health economic models, and reported. OBJECTIVE:Our objective was to develop the TRansparent Uncertainty ASsessmenT (TRUST) tool for systematically identifying, assessing, and reporting uncertainties in decision models, with the aim of making uncertainties and their impact on cost effectiveness more explicit and transparent. METHODS:TRUST was developed by drawing on the uncertainty and risk assessment literature. To develop and validate this tool, we conducted HTA stakeholder discussion meetings and interviews and applied it in six real-world HTA case studies in the Netherlands and the UK. RESULTS:The TRUST tool enables the identification and categorisation of uncertainty according to its source (transparency issues, methodology issues, and issues with evidence: imprecision, bias and indirectness, and unavailability) in each model aspect. The source of uncertainty determines the appropriate analysis. The impact of uncertainties on cost effectiveness is also assessed. Stakeholders found using the tool to be feasible and of value for transparent uncertainty assessment. TRUST can be used during model development and/or model review. CONCLUSION:The TRUST tool enables systematic identification, assessment, and reporting of uncertainties in health economic models and may contribute to more informed and transparent decision making in the face of uncertainty.

SUBMITTER: Grimm SE

PROVIDER: S-EPMC7081657 | biostudies-literature | 2020 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Development and Validation of the TRansparent Uncertainty ASsessmenT (TRUST) Tool for Assessing Uncertainties in Health Economic Decision Models.

Grimm Sabine E SE Pouwels Xavier X Ramaekers Bram L T BLT Wijnen Ben B Knies Saskia S Grutters Janneke J Joore Manuela A MA

PharmacoEconomics 20200201 2

<h4>Background</h4>An increasing number of technologies are obtaining marketing authorisation based on sparse evidence, which causes growing uncertainty and risk within health technology reimbursement decision making. To ensure that uncertainty is considered and addressed within health technology assessment (HTA) recommendations, uncertainties need to be identified, included in health economic models, and reported.<h4>Objective</h4>Our objective was to develop the TRansparent Uncertainty ASsessm ...[more]

PMID: 31709496

Similar Datasets

Project description:Typical applications of process- or physically-based models aim to gain a better process understanding or provide the basis for a decision-making process. To adequately represent the physical system, models should include all essential processes. However, model errors can still occur. Other than large systematic observation errors, simplified, misrepresented, inadequately parametrised or missing processes are potential sources of errors. This study presents a set of methods and a proposed workflow for analysing errors of process-based models as a basis for relating them to process representations. The evaluated approach consists of three steps: (1) training a machine-learning (ML) error model using the input data of the process-based model and other available variables, (2) estimation of local explanations (i.e., contributions of each variable to an individual prediction) for each predicted model error using SHapley Additive exPlanations (SHAP) in combination with principal component analysis, (3) clustering of SHAP values of all predicted errors to derive groups with similar error generation characteristics. By analysing these groups of different error-variable association, hypotheses on error generation and corresponding processes can be formulated. That can ultimately lead to improvements in process understanding and prediction. The approach is applied to a process-based stream water temperature model HFLUX in a case study for modelling an alpine stream in the Canadian Rocky Mountains. By using available meteorological and hydrological variables as inputs, the applied ML model is able to predict model residuals. Clustering of SHAP values results in three distinct error groups that are mainly related to shading and vegetation-emitted long wave radiation. Model errors are rarely random and often contain valuable information. Assessing model error associations is ultimately a way of enhancing trust in implemented processes and of providing information on potential areas of improvement to the model.

Project description:IntroductionExternal validation studies are essential to study the generalizability of prediction models. Recently a permutation test, focusing on discrimination as quantified by the c-statistic, was proposed to judge whether a prediction model is transportable to a new setting. We aimed to evaluate this test and compare it to previously proposed procedures to judge any changes in c-statistic from development to external validation setting.MethodsWe compared the use of the permutation test to the use of benchmark values of the c-statistic following from a previously proposed framework to judge transportability of a prediction model. In a simulation study we developed a prediction model with logistic regression on a development set and validated them in the validation set. We concentrated on two scenarios: 1) the case-mix was more heterogeneous and predictor effects were weaker in the validation set compared to the development set, and 2) the case-mix was less heterogeneous in the validation set and predictor effects were identical in the validation and development set. Furthermore we illustrated the methods in a case study using 15 datasets of patients suffering from traumatic brain injury.ResultsThe permutation test indicated that the validation and development set were homogenous in scenario 1 (in almost all simulated samples) and heterogeneous in scenario 2 (in 17%-39% of simulated samples). Previously proposed benchmark values of the c-statistic and the standard deviation of the linear predictors correctly pointed at the more heterogeneous case-mix in scenario 1 and the less heterogeneous case-mix in scenario 2.ConclusionThe recently proposed permutation test may provide misleading results when externally validating prediction models in the presence of case-mix differences between the development and validation population. To correctly interpret the c-statistic found at external validation it is crucial to disentangle case-mix differences from incorrect regression coefficients.

Dataset Information

Development and Validation of the TRansparent Uncertainty ASsessmenT (TRUST) Tool for Assessing Uncertainties in Health Economic Decision Models.

Publications

Development and Validation of the TRansparent Uncertainty ASsessmenT (TRUST) Tool for Assessing Uncertainties in Health Economic Decision Models.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets