Dataset Information

Does technique matter; a pilot study exploring weighting techniques for a multi-criteria decision support framework.

ABSTRACT:

Background

There is an increased interest in the use of multi-criteria decision analysis (MCDA) to support regulatory and reimbursement decision making. The EVIDEM framework was developed to provide pragmatic multi-criteria decision support in health care, to estimate the value of healthcare interventions, and to aid in priority-setting. The objectives of this study were to test 1) the influence of different weighting techniques on the overall outcome of an MCDA exercise, 2) the discriminative power in weighting different criteria of such techniques, and 3) whether different techniques result in similar weights in weighting the criteria set proposed by the EVIDEM framework.

Methods

A sample of 60 Dutch and Canadian students participated in the study. Each student used an online survey to provide weights for 14 criteria with two different techniques: a five-point rating scale and one of the following techniques selected randomly: ranking, point allocation, pairwise comparison and best worst scaling.

Results

The results of this study indicate that there is no effect of differences in weights on value estimates at the group level. On an individual level, considerable differences in criteria weights and rank order occur as a result of the weight elicitation method used, and the ability of different techniques to discriminate in criteria importance. Of the five techniques tested, the pair-wise comparison of criteria has the highest ability to discriminate in weights when fourteen criteria are compared.

Conclusions

When weights are intended to support group decisions, the choice of elicitation technique has negligible impact on criteria weights and the overall value of an innovation. However, when weights are used to support individual decisions, the choice of elicitation technique influences outcome and studies that use dissimilar techniques cannot be easily compared. Weight elicitation through pairwise comparison of criteria is preferred when taking into account its superior ability to discriminate between criteria and respondents' preferences.

SUBMITTER: van Til J

PROVIDER: S-EPMC4406027 | biostudies-literature | 2014

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Does technique matter; a pilot study exploring weighting techniques for a multi-criteria decision support framework.

van Til Janine J Groothuis-Oudshoorn Catharina C Lieferink Marijke M Dolan James J Goetghebeur Mireille M

Cost effectiveness and resource allocation : C/E 20141118

<h4>Background</h4>There is an increased interest in the use of multi-criteria decision analysis (MCDA) to support regulatory and reimbursement decision making. The EVIDEM framework was developed to provide pragmatic multi-criteria decision support in health care, to estimate the value of healthcare interventions, and to aid in priority-setting. The objectives of this study were to test 1) the influence of different weighting techniques on the overall outcome of an MCDA exercise, 2) the discrimi ...[more]

PMID: 25904823

Similar Datasets

Project description:The Centers for Disease Control and Prevention (CDC) Select Agent Program establishes a list of biological agents and toxins that potentially threaten public health and safety, the procedures governing the possession, utilization, and transfer of those agents, and training requirements for entities working with them. Every 2 years the Program reviews the select agent list, utilizing subject matter expert (SME) assessments to rank the agents. In this study, we explore the applicability of multi-criteria decision analysis (MCDA) techniques and logic tree analysis to support the CDC Select Agent Program biennial review process, applying the approach broadly to include non-select agents to evaluate its generality. We conducted a literature search for over 70 pathogens against 15 criteria for assessing public health and bioterrorism risk and documented the findings for archiving. The most prominent data gaps were found for aerosol stability and human infectious dose by inhalation and ingestion routes. Technical review of published data and associated scoring recommendations by pathogen-specific SMEs was found to be critical for accuracy, particularly for pathogens with very few known cases, or where proxy data (e.g., from animal models or similar organisms) were used to address data gaps. Analysis of results obtained from a two-dimensional plot of weighted scores for difficulty of attack (i.e., exposure and production criteria) vs. consequences of an attack (i.e., consequence and mitigation criteria) provided greater fidelity for understanding agent placement compared to a 1-to-n ranking and was used to define a region in the upper right-hand quadrant for identifying pathogens for consideration as select agents. A sensitivity analysis varied the numerical weights attributed to various properties of the pathogens to identify potential quantitative (x and y) thresholds for classifying select agents. The results indicate while there is some clustering of agent scores to suggest thresholds, there are still pathogens that score close to any threshold, suggesting that thresholding "by eye" may not be sufficient. The sensitivity analysis indicates quantitative thresholds are plausible, and there is good agreement of the analytical results with select agent designations. A second analytical approach that applied the data using a logic tree format to rule out pathogens for consideration as select agents arrived at similar conclusions.

Project description:In densely populated areas, surface waters are affected by many sources of pollution. Besides classical pollutants like nutrients and organic matter that lead to eutrophication, micropollutants from various point- and non-point sources are getting more attention by water quality managers. For cost-effective management an integrated assessment is needed that takes into account all relevant pollutants and all sources of pollution within a catchment. Due to the difficulty of identifying and quantifying sources of pollution and the need for considering long-term changes in boundary conditions, typically substantial uncertainty exists about the consequences of potential management alternatives to improve surface water quality. We therefore need integrated assessment methods that are able to deal with multiple objectives and account for various sources of uncertainty. This paper aims to contribute to integrated, prospective water management by combining a) multi-criteria decision support methods to structure the decision process and quantify preferences, b) integrated water quality modelling to predict consequences of management alternatives accounting for uncertainty, and c) scenario planning to consider uncertainty from potential future climate and socio-economic developments, to evaluate the future cost-effectiveness of water quality management alternatives at the catchment scale. It aims to demonstrate the usefulness of multi-attribute value functions for water quality assessment to i) propagate uncertainties throughout the entire assessment procedure, ii) facilitate the aggregation of multiple objectives while avoiding discretization errors when using categories for sub-objectives, iii) transparently communicate the results. We show how to use such multi-attribute value functions for model-based decision support in water quality management. We showcase the procedure for the Mönchaltorfer Aa catchment on the Swiss Plateau. We evaluate ten different water quality management alternatives, including current practice, that tackle macro- and micropollutants from a wide spectrum of agricultural and urban sources. We evaluate costs and water quality effects of the alternatives under four different socio-economic scenarios for the horizon 2050 under present and future climate projections and visualize their uncertainty. While the performance of alternatives is catchment specific, the methods can be transferred to other places and other management situations. Results confirm the need for cross-sectoral coordination of different management actions and interdisciplinary collaboration to support the development of prospective strategies to improve water quality.

Project description:Background:Decision-making in public health and health policy is complex and requires careful deliberation of many and sometimes conflicting normative and technical criteria. Several approaches and tools, such as multi-criteria decision analysis, health technology assessments and evidence-to-decision (EtD) frameworks, have been proposed to guide decision-makers in selecting the criteria most relevant and appropriate for a transparent decision-making process. This study forms part of the development of the WHO-INTEGRATE EtD framework, a framework rooted in global health norms and values as reflected in key documents of the World Health Organization and the United Nations system. The objective of this study was to provide a comprehensive overview of criteria used in or proposed for real-world decision-making processes, including guideline development, health technology assessment, resource allocation and others. Methods:We conducted an overview of systematic reviews through a combination of systematic literature searches and extensive reference searches. Systematic reviews reporting criteria used for real-world health decision-making by governmental or non-governmental organization on a supranational, national, or programme level were included and their quality assessed through a bespoke critical appraisal tool. The criteria reported in the reviews were extracted, de-duplicated and sorted into first-level (i.e. criteria), second-level (i.e. sub-criteria) and third-level (i.e. decision aspects) categories. First-level categories were developed a priori using a normative approach; second- and third-level categories were developed inductively. Results:We included 36 systematic reviews providing criteria, of which one met all and another eleven met at least five of the items of our critical appraisal tool. The criteria were subsumed into 8 criteria, 45 sub-criteria and 200 decision aspects. The first-level of the category system comprised the following seven substantive criteria: "Health-related balance of benefits and harms"; "Human and individual rights"; "Acceptability considerations"; "Societal considerations"; "Considerations of equity, equality and fairness"; "Cost and financial considerations"; and "Feasibility and health system considerations". In addition, we identified an eight criterion "Evidence". Conclusion:This overview of systematic reviews provides a comprehensive overview of criteria used or suggested for real-world health decision-making. It also discusses key challenges in the selection of the most appropriate criteria and in seeking to implement a fair decision-making process.

Dataset Information

Does technique matter; a pilot study exploring weighting techniques for a multi-criteria decision support framework.

Background

Methods

Results

Conclusions

Publications

Does technique matter; a pilot study exploring weighting techniques for a multi-criteria decision support framework.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets