Dataset Information

Applying A/B Testing to Clinical Decision Support: Rapid Randomized Controlled Trials.

ABSTRACT:

Background

Clinical decision support (CDS) is a valuable feature of electronic health records (EHRs) designed to improve quality and safety. However, due to the complexities of system design and inconsistent results, CDS tools may inadvertently increase alert fatigue and contribute to physician burnout. A/B testing, or rapid-cycle randomized tests, is a useful method that can be applied to the EHR in order to rapidly understand and iteratively improve design choices embedded within CDS tools.

Objective

This paper describes how rapid randomized controlled trials (RCTs) embedded within EHRs can be used to quickly ascertain the superiority of potential CDS design changes to improve their usability, reduce alert fatigue, and promote quality of care.

Methods

A multistep process combining tools from user-centered design, A/B testing, and implementation science was used to understand, ideate, prototype, test, analyze, and improve each candidate CDS. CDS engagement metrics (alert views, acceptance rates) were used to evaluate which CDS version is superior.

Results

To demonstrate the impact of the process, 2 experiments are highlighted. First, after multiple rounds of usability testing, a revised CDS influenza alert was tested against usual care CDS in a rapid (~6 weeks) RCT. The new alert text resulted in minimal impact on reducing firings per patients per day, but this failure triggered another round of review that identified key technical improvements (ie, removal of dismissal button and firings in procedural areas) that led to a dramatic decrease in firings per patient per day (23.1 to 7.3). In the second experiment, the process was used to test 3 versions (financial, quality, regulatory) of text supporting tobacco cessation alerts as well as 3 supporting images. Based on 3 rounds of RCTs, there was no significant difference in acceptance rates based on the framing of the messages or addition of images.

Conclusions

These experiments support the potential for this new process to rapidly develop, deploy, and rigorously evaluate CDS within an EHR. We also identified important considerations in applying these methods. This approach may be an important tool for improving the impact of and experience with CDS.

Trial registration

Flu alert trial: ClinicalTrials.gov NCT03415425; https://clinicaltrials.gov/ct2/show/NCT03415425. Tobacco alert trial: ClinicalTrials.gov NCT03714191; https://clinicaltrials.gov/ct2/show/NCT03714191.

SUBMITTER: Austrian J

PROVIDER: S-EPMC8065554 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundLimited consideration of clinical decision support (CDS) design best practices, such as a user-centered design, is often cited as a key barrier to CDS adoption and effectiveness. The application of CDS best practices is resource intensive; thus, institutions often rely on commercially available CDS tools that are created to meet the generalized needs of many institutions and are not user centered. Beyond resource availability, insufficient guidance on how to address key aspects of implementation, such as contextual factors, may also limit the application of CDS best practices. An implementation science (IS) framework could provide needed guidance and increase the reproducibility of CDS implementations.ObjectiveThis study aims to compare the effectiveness of an enhanced CDS tool informed by CDS best practices and an IS framework with a generic, commercially available CDS tool.MethodsWe conducted an explanatory sequential mixed methods study. An IS-enhanced and commercial CDS alert were compared in a cluster randomized trial across 28 primary care clinics. Both alerts aimed to improve beta-blocker prescribing for heart failure. The enhanced alert was informed by CDS best practices and the Practical, Robust, Implementation, and Sustainability Model (PRISM) IS framework, whereas the commercial alert followed vendor-supplied specifications. Following PRISM, the enhanced alert was informed by iterative, multilevel stakeholder input and the dynamic interactions of the internal and external environment. Outcomes aligned with PRISM's evaluation measures, including patient reach, clinician adoption, and changes in prescribing behavior. Clinicians exposed to each alert were interviewed to identify design features that might influence adoption. The interviews were analyzed using a thematic approach.ResultsBetween March 15 and August 23, 2019, the enhanced alert fired for 61 patients (106 alerts, 87 clinicians) and the commercial alert fired for 26 patients (59 alerts, 31 clinicians). The adoption and effectiveness of the enhanced alert were significantly higher than those of the commercial alert (62% vs 29% alerts adopted, P<.001; 14% vs 0% changed prescribing, P=.006). Of the 21 clinicians interviewed, most stated that they preferred the enhanced alert.ConclusionsThe results of this study suggest that applying CDS best practices with an IS framework to create CDS tools improves implementation success compared with a commercially available tool.Trial registrationClinicalTrials.gov NCT04028557; http://clinicaltrials.gov/ct2/show/NCT04028557.

Project description:BackgroundCancer patients often do not make informed decisions regarding clinical trial participation. This study evaluated whether a web-based decision aid (DA) could support trial decisions compared with our cancer center's website.MethodsAdults diagnosed with cancer in the past 6 months who had not previously participated in a cancer clinical trial were eligible. Participants were randomized to view the DA or our cancer center's website (enhanced usual care [UC]). Controlling for whether participants had heard of cancer clinical trials and educational attainment, multivariable linear regression examined group on knowledge, self-efficacy for finding trial information, decisional conflict (values clarity and uncertainty), intent to participate, decision readiness, and trial perceptions.ResultsTwo hundred patients (86%) consented between May 2014 and April 2015. One hundred were randomized to each group. Surveys were completed by 87 in the DA group and 90 in the UC group. DA group participants reported clearer values regarding trial participation than UC group participants reported (least squares [LS] mean = 15.8 vs. 32, p < .0001) and less uncertainty (LS mean = 24.3 vs. 36.4, p = .025). The DA group had higher objective knowledge than the UC group's (LS mean = 69.8 vs. 55.8, p < .0001). There were no differences between groups in intent to participate.ConclusionsImprovements on key decision outcomes including knowledge, self-efficacy, certainty about choice, and values clarity among participants who viewed the DA suggest web-based DAs can support informed decisions about trial participation among cancer patients facing this preference-sensitive choice. Although better informing patients before trial participation could improve retention, more work is needed to examine DA impact on enrollment and retention.Implications for practiceThis paper describes evidence regarding a decision tool to support patients' decisions about trial participation. By improving knowledge, helping patients clarify preferences for participation, and facilitating conversations about trials, decision aids could lead to decisions about participation that better match patients' preferences, promoting patient-centered care and the ethical conduct of clinical research.

Project description:ObjectiveTo report the improvements achieved with clinical decision support systems and examine the heterogeneity from pooling effects across diverse clinical settings and intervention targets.DesignSystematic review and meta-analysis.Data sourcesMedline up to August 2019.Eligibility criteria for selecting studies and methodsRandomised or quasi-randomised controlled trials reporting absolute improvements in the percentage of patients receiving care recommended by clinical decision support systems. Multilevel meta-analysis accounted for within study clustering. Meta-regression was used to assess the degree to which the features of clinical decision support systems and study characteristics reduced heterogeneity in effect sizes. Where reported, clinical endpoints were also captured.ResultsIn 108 studies (94 randomised, 14 quasi-randomised), reporting 122 trials that provided analysable data from 1 203 053 patients and 10 790 providers, clinical decision support systems increased the proportion of patients receiving desired care by 5.8% (95% confidence interval 4.0% to 7.6%). This pooled effect exhibited substantial heterogeneity (I2=76%), with the top quartile of reported improvements ranging from 10% to 62%. In 30 trials reporting clinical endpoints, clinical decision support systems increased the proportion of patients achieving guideline based targets (eg, blood pressure or lipid control) by a median of 0.3% (interquartile range -0.7% to 1.9%). Two study characteristics (low baseline adherence and paediatric settings) were associated with significantly larger effects. Inclusion of these covariates in the multivariable meta-regression, however, did not reduce heterogeneity.ConclusionsMost interventions with clinical decision support systems appear to achieve small to moderate improvements in targeted processes of care, a finding confirmed by the small changes in clinical endpoints found in studies that reported them. A minority of studies achieved substantial increases in the delivery of recommended care, but predictors of these more meaningful improvements remain undefined.