Dataset Information

Rapid methods including network meta-analysis to produce evidence in clinical decision support: a decision analysis.

ABSTRACT:

Background

Conducting systematic reviews is time-consuming but crucial to construct evidence-based patient decision aids, clinical practice guidelines and decision analyses. New methods might enable developers to produce a knowledge base more rapidly. However, trading off scientific rigour for speed when creating a knowledge base is controversial, and the consequences are insufficiently known. We developed and applied faster methods including systematic reviews and network meta-analyses, assessed their feasibility and compared them to a gold standard approach. We also assessed the feasibility of using decision analysis to perform this comparison.

Methods

Long-term treatment in bipolar disorder was our testing field. We developed two new methods: an empirically based, rapid network meta-analysis (NMA) and an expert NMA, and conducted a patient survey. We applied these methods to collect effect estimates for evidence-based treatments on outcomes important to patients. The relative importance of outcomes was obtained from patients using a stated preference method. We used multi-criteria decision analysis to compare a gold standard NMA with the rapid NMA in terms of the ability of the gold standard NMA to change the ranking and expected values of treatments for individual patients.

Results

Using rapid methods, it was feasible to identify evidence addressing outcomes important to patients. We found that replacing effect estimates from our rapid NMA with estimates from the gold standard NMA resulted in relatively small changes in the ranking and expected value of treatments. The rapid method sufficed to estimate the effects of nine out of ten options. To produce a ranking of treatments accurate for more than 95% of patients, it was necessary to supplement systematic with rapid methods and to use relative importance weights in the analysis. Integrating estimates of the outcome "treatment burden" had a larger impact on rankings than replacing rapid with gold standard methods. Using patients' importance weights only modestly affected results.

Conclusions

The transfer of knowledge to practice could benefit from faster systematic reviewing methods. The results in this preliminary assessment suggest that an improved rapid NMA approach might replace gold standard NMAs. Decision analysis could be used to compare evidence summarisation methods.

SUBMITTER: Eiring O

PROVIDER: S-EPMC6195718 | biostudies-literature | 2018 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Rapid methods including network meta-analysis to produce evidence in clinical decision support: a decision analysis.

Eiring Øystein Ø Brurberg Kjetil Gundro KG Nytrøen Kari K Nylenna Magne M

Systematic reviews 20181020 1

<h4>Background</h4>Conducting systematic reviews is time-consuming but crucial to construct evidence-based patient decision aids, clinical practice guidelines and decision analyses. New methods might enable developers to produce a knowledge base more rapidly. However, trading off scientific rigour for speed when creating a knowledge base is controversial, and the consequences are insufficiently known. We developed and applied faster methods including systematic reviews and network meta-analyses, ...[more]

PMID: 30342549

Similar Datasets

Project description:BackgroundDiagnostic errors pose significant health risks and contribute to patient mortality. With the growing accessibility of electronic health records, machine learning models offer a promising avenue for enhancing diagnosis quality. Current research has primarily focused on a limited set of diseases with ample training data, neglecting diagnostic scenarios with limited data availability.ObjectiveThis study aims to develop an information retrieval (IR)-based framework that accommodates data sparsity to facilitate broader diagnostic decision support.MethodsWe introduced an IR-based diagnostic decision support framework called CliniqIR. It uses clinical text records, the Unified Medical Language System Metathesaurus, and 33 million PubMed abstracts to classify a broad spectrum of diagnoses independent of training data availability. CliniqIR is designed to be compatible with any IR framework. Therefore, we implemented it using both dense and sparse retrieval approaches. We compared CliniqIR's performance to that of pretrained clinical transformer models such as Clinical Bidirectional Encoder Representations from Transformers (ClinicalBERT) in supervised and zero-shot settings. Subsequently, we combined the strength of supervised fine-tuned ClinicalBERT and CliniqIR to build an ensemble framework that delivers state-of-the-art diagnostic predictions.ResultsOn a complex diagnosis data set (DC3) without any training data, CliniqIR models returned the correct diagnosis within their top 3 predictions. On the Medical Information Mart for Intensive Care III data set, CliniqIR models surpassed ClinicalBERT in predicting diagnoses with <5 training samples by an average difference in mean reciprocal rank of 0.10. In a zero-shot setting where models received no disease-specific training, CliniqIR still outperformed the pretrained transformer models with a greater mean reciprocal rank of at least 0.10. Furthermore, in most conditions, our ensemble framework surpassed the performance of its individual components, demonstrating its enhanced ability to make precise diagnostic predictions.ConclusionsOur experiments highlight the importance of IR in leveraging unstructured knowledge resources to identify infrequently encountered diagnoses. In addition, our ensemble framework benefits from combining the complementary strengths of the supervised and retrieval-based models to diagnose a broad spectrum of diseases.

Project description:BackgroundDuring the COVID-19 pandemic, a variety of clinical decision support systems (CDSS) were developed to aid patient triage. However, research focusing on the interaction between decision support systems and human experts is lacking.MethodsThirty-two physicians were recruited to rate the survival probability of 59 critically ill patients by means of chart review. Subsequently, one of two artificial intelligence systems advised the physician of a computed survival probability. However, only one of these systems explained the reasons behind its decision-making. In the third step, physicians reviewed the chart once again to determine the final survival probability rating. We hypothesized that an explaining system would exhibit a higher impact on the physicians' second rating (i.e., higher weight-on-advice).ResultsThe survival probability rating given by the physician after receiving advice from the clinical decision support system was a median of 4 percentage points closer to the advice than the initial rating. Weight-on-advice was not significantly different (p = 0.115) between the two systems (with vs without explanation for its decision). Additionally, weight-on-advice showed no difference according to time of day or between board-qualified and not yet board-qualified physicians. Self-reported post-experiment overall trust was awarded a median of 4 out of 10 points. When asked after the conclusion of the experiment, overall trust was 5.5/10 (non-explaining median 4 (IQR 3.5-5.5), explaining median 7 (IQR 5.5-7.5), p = 0.007).ConclusionsAlthough overall trust in the models was low, the median (IQR) weight-on-advice was high (0.33 (0.0-0.56)) and in line with published literature on expert advice. In contrast to the hypothesis, weight-on-advice was comparable between the explaining and non-explaining systems. In 30% of cases, weight-on-advice was 0, meaning the physician did not change their rating. The median of the remaining weight-on-advice values was 50%, suggesting that physicians either dismissed the recommendation or employed a "meeting halfway" approach. Newer technologies, such as clinical reasoning systems, may be able to augment the decision process rather than simply presenting unexplained bias.

Project description:Pressure ulcers are a prevalent and global issue and support surfaces are widely used for preventing ulceration. However, the diversity of available support surfaces and the lack of direct comparisons in RCTs make decision-making difficult.To determine, using network meta-analysis, the relative effects of different support surfaces in reducing pressure ulcer incidence and comfort and to rank these support surfaces in order of their effectiveness.We conducted a systematic review, using a literature search up to November 2016, to identify randomised trials comparing support surfaces for pressure ulcer prevention. Two reviewers independently performed study selection, risk of bias assessment and data extraction. We grouped the support surfaces according to their characteristics and formed evidence networks using these groups. We used network meta-analysis to estimate the relative effects and effectiveness ranking of the groups for the outcomes of pressure ulcer incidence and participant comfort. GRADE was used to assess the certainty of evidence.We included 65 studies in the review. The network for assessing pressure ulcer incidence comprised evidence of low or very low certainty for most network contrasts. There was moderate-certainty evidence that powered active air surfaces and powered hybrid air surfaces probably reduce pressure ulcer incidence compared with standard hospital surfaces (risk ratios (RR) 0.42, 95% confidence intervals (CI) 0.29 to 0.63; 0.22, 0.07 to 0.66, respectively). The network for comfort suggested that powered active air-surfaces are probably slightly less comfortable than standard hospital mattresses (RR 0.80, 95% CI 0.69 to 0.94; moderate-certainty evidence).This is the first network meta-analysis of the effects of support surfaces for pressure ulcer prevention. Powered active air-surfaces probably reduce pressure ulcer incidence, but are probably less comfortable than standard hospital surfaces. Most prevention evidence was of low or very low certainty, and more research is required to reduce these uncertainties.

Project description:The objective of this work was to inform the design of a rapid response program to support evidence-informed decision-making in health policy and practice for the Americas region. Specifically, we focus on the following: (1) What are the best methodological approaches for rapid reviews of the research evidence? (2) What other strategies are needed to facilitate evidence-informed decision-making in health policy and practice? and (3) How best to operationalize a rapid response program?The evidence used to inform the design of a rapid response program included (i) two rapid reviews of methodological approaches for rapid reviews of the research evidence and strategies to facilitate evidence-informed decision-making, (ii) supplementary literature in relation to the "shortcuts" that could be considered to reduce the time needed to complete rapid reviews, (iii) four case studies, and (iv) supplementary literature to identify additional operational issues for the design of the program.There is no agreed definition of rapid reviews in the literature and no agreed methodology for conducting them. Better reporting of rapid review methods is needed. The literature found in relation to shortcuts will be helpful in choosing shortcuts that maximize timeliness while minimizing the impact on quality. Evidence for other strategies that can be used concurrently to facilitate the uptake of research evidence, including evidence drawn from rapid reviews, is presented. Operational issues that need to be considered in designing a rapid response program include the implications of a "user-pays" model, the importance of recruiting staff with the right mix of skills and qualifications, and ensuring that the impact of the model on research use in decision-making is formally evaluated.When designing a new rapid response program, greater attention needs to be given to specifying the rapid review methods and reporting these in sufficient detail to allow a quality assessment. It will also be important to engage in other strategies to facilitate the uptake of the rapid reviews and to evaluate the chosen model in order to make refinements and add to the evidence base for evidence-informed decision-making.

Project description:BackgroundPublic health professionals are increasingly expected to engage in evidence-informed decision making to inform practice and policy decisions. Evidence-informed decision making involves the use of research evidence along with expertise, existing public health resources, knowledge about community health issues, the local context and community, and the political climate. The National Collaborating Centre for Methods and Tools has identified a seven step process for evidence-informed decision making. Tools have been developed to support public health professionals as they work through each of these steps. This paper provides an overview of tools used in three Canadian public health departments involved in a study to develop capacity for evidence-informed decision making.MethodsAs part of a knowledge translation and exchange intervention, a Knowledge Broker worked with public health professionals to identify and apply tools for use with each of the steps of evidence-informed decision making. The Knowledge Broker maintained a reflective journal and interviews were conducted with a purposive sample of decision makers and public health professionals. This paper presents qualitative analysis of the perceived usefulness and usability of the tools.ResultsTools were used in the health departments to assist in: question identification and clarification; searching for the best available research evidence; assessing the research evidence for quality through critical appraisal; deciphering the 'actionable message(s)' from the research evidence; tailoring messages to the local context to ensure their relevance and suitability; deciding whether and planning how to implement research evidence in the local context; and evaluating the effectiveness of implementation efforts. Decision makers provided descriptions of how the tools were used within the health departments and made suggestions for improvement. Overall, the tools were perceived as valuable for advancing and sustaining evidence-informed decision making.ConclusionTools are available to support the process of evidence-informed decision making among public health professionals. The usability and usefulness of these tools for advancing and sustaining evidence-informed decision making are discussed, including recommendations for the tools' application in other public health settings beyond this study. Knowledge and awareness of these tools may assist other health professionals in their efforts to implement evidence-informed practice.

Project description:BackgroundMinistries of health in low- and middle-income countries often lack timely quality data for data-driven decision making in healthcare networks. We describe the design and implementation of decision-support electronic tools by the Ministry of Health of the State of Chiapas, in Mexico, as part of Salud Mesoamerica Initiative.MethodsThree electronic decision-support tools were designed through an iterative process focused on streamlined implementation: 1) to collect and report health facility data at health facilities; 2) to compile and analyze data at health district and central level; and, 3) to support stratified sampling of health facilities. Data was collected for five composite indicators measuring availability of equipment, medicines, and supplies for maternal and child health. Quality Assurance Teams collected data, evaluated results and supported quality improvement. Data was also analyzed at the central level and health districts for decision-making.ResultsData from 300 health facilities in four health districts was collected and analyzed (November 2014-June 2015). The first wave revealed gaps on availability of equipment and supplies in more than half of health facilities. Electronic tools provided the ministry of health officers new ways to visualize data, identify patterns and make hypothesis on root-causes. Between the first and second measurement, the number of missing items decreased, and actions performed by quality improvement teams became more proactive. In the final measurement, 89.7-100% of all health facilities achieved all the required items for each indicator.ConclusionsOur experience could help guide others seeking to implement electronic decision-support tools in low- and middle-income countries. Electronic decision-support tools supported data-driven decision-making by identifying gaps on heatmaps and graphs at the health facility, subdistrict, district or state level. Through a rapid improvement process, the Ministry of Health met targets of externally verified indicators. Using available information technology resources facilitated prompt implementation and adoption of technology.

Dataset Information

Rapid methods including network meta-analysis to produce evidence in clinical decision support: a decision analysis.

Background

Methods

Results

Conclusions

Publications

Rapid methods including network meta-analysis to produce evidence in clinical decision support: a decision analysis.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets