Dataset Information

OrderRex clinical user testing: a randomized trial of recommender system decision support on simulated cases.

ABSTRACT: Objective: To assess usability and usefulness of a machine learning-based order recommender system applied to simulated clinical cases.

Materials and methods: 43 physicians entered orders for 5 simulated clinical cases using a clinical order entry interface with or without access to a previously developed automated order recommender system. Cases were randomly allocated to the recommender system in a 3:2 ratio. A panel of clinicians scored whether the orders placed were clinically appropriate. Our primary outcome included the difference in clinical appropriateness scores. Secondary outcomes included total number of orders, case time, and survey responses.

Results: Clinical appropriateness scores per order were comparable for cases randomized to the order recommender system (mean difference -0.11 order per score, 95% CI: [-0.41, 0.20]). Physicians using the recommender placed more orders (median 16 vs 15 orders, incidence rate ratio 1.09, 95%CI: [1.01-1.17]). Case times were comparable with the recommender system. Order suggestions generated from the recommender system were more likely to match physician needs than standard manual search options. Physicians used recommender suggestions in 98% of available cases. Approximately 95% of participants agreed the system would be useful for their workflows.

Discussion: User testing with a simulated electronic medical record interface can assess the value of machine learning and clinical decision support tools for clinician usability and acceptance before live deployments.

Conclusions: Clinicians can use and accept machine learned clinical order recommendations integrated into an electronic order entry interface in a simulated setting. The clinical appropriateness of orders entered was comparable even when supported by automated recommendations.

SUBMITTER: Kumar A

PROVIDER: S-EPMC7727352 | biostudies-literature | 2020 Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

OrderRex clinical user testing: a randomized trial of recommender system decision support on simulated cases.

Kumar Andre A Aikens Rachael C RC Hom Jason J Shieh Lisa L Chiang Jonathan J Morales David D Saini Divya D Musen Mark M Baiocchi Michael M Altman Russ R Goldstein Mary K MK Asch Steven S Chen Jonathan H JH

Journal of the American Medical Informatics Association : JAMIA 20201201 12

<h4>Objective</h4>To assess usability and usefulness of a machine learning-based order recommender system applied to simulated clinical cases.<h4>Materials and methods</h4>43 physicians entered orders for 5 simulated clinical cases using a clinical order entry interface with or without access to a previously developed automated order recommender system. Cases were randomly allocated to the recommender system in a 3:2 ratio. A panel of clinicians scored whether the orders placed were clinically a ...[more]

PMID: 33106874

Similar Datasets

Project description:BackgroundPharmacogenomic clinical decision support systems (CDSS) have the potential to help overcome some of the barriers for translating pharmacogenomic knowledge into clinical routine. Before developing a prototype it is crucial for developers to know which pharmacogenomic CDSS features and user-system interactions have yet been developed, implemented and tested in previous pharmacogenomic CDSS efforts and if they have been successfully applied. We address this issue by providing an overview of the designs of user-system interactions of recently developed pharmacogenomic CDSS.MethodsWe searched PubMed for pharmacogenomic CDSS published between January 1, 2012 and November 15, 2016. Thirty-two out of 118 identified articles were summarized and included in the final analysis. We then compared the designs of user-system interactions of the 20 pharmacogenomic CDSS we had identified.ResultsAlerts are the most widespread tools for physician-system interactions, but need to be implemented carefully to prevent alert fatigue and avoid liabilities. Pharmacogenomic test results and override reasons stored in the local EHR might help communicate pharmacogenomic information to other internal care providers. Integrating patients into user-system interactions through patient letters and online portals might be crucial for transferring pharmacogenomic data to external health care providers. Inbox messages inform physicians about new pharmacogenomic test results and enable them to request pharmacogenomic consultations. Search engines enable physicians to compare medical treatment options based on a patient's genotype.ConclusionsWithin the last 5 years, several pharmacogenomic CDSS have been developed. However, most of the included articles are solely describing prototypes of pharmacogenomic CDSS rather than evaluating them. To support the development of prototypes further evaluation efforts will be necessary. In the future, pharmacogenomic CDSS will likely include prediction models to identify patients who are suitable for preemptive genotyping.

Project description:Rationale & objectiveThe study aimed to develop, implement, and evaluate a clinical decision support (CDS) system for chronic kidney disease (CKD) in a primary care setting, with the goal of improving CKD care in adults.Study designThis was a cluster randomized trial.Setting & participantsA total of 32 Midwestern primary care clinics were randomly assigned to either receive usual care or CKD-CDS intervention. Between April 2019 and March 2020, we enrolled 6,420 patients aged 18-75 years with laboratory-defined glomerular filtration rate categories of CKD Stage G3 and G4, and 1 or more of 6 CKD care gaps: absence of a CKD diagnosis, suboptimal blood pressure or glycated hemoglobin levels, indication for angiotensin-converting enzyme inhibitor or angiotensin receptor blocker but not prescribed, a nonsteroidal anti-inflammatory agent on the active medication list, or indication for a nephrology referral.InterventionThe CKD-CDS provided personalized suggestions for CKD care improvement opportunities directed to both patients and clinicians at primary care encounters.OutcomesWe assessed the proportion of patients meeting each of 6 CKD-CDS quality metrics representing care gap resolution after 18 months.ResultsThe adjusted proportions of patients meeting quality metrics in CKD-CDS versus usual care were as follows: CKD diagnosis documented (26.6% vs 21.8%; risk ratio [RR], 1.17; 95% CI, 0.91-1.51); angiotensin-converting enzyme inhibitor or angiotensin receptor blocker prescribed (15.9% vs 16.1%; RR, 0.95; 95% CI, 0.76-1.18); blood pressure control (20.4% vs 20.2%; RR, 0.98; 95% CI, 0.84-1.15); glycated hemoglobin level control (21.4% vs 22.1%; RR, 1.00; 95% CI, 0.80-1.24); nonsteroidal anti-inflammatory agent not on the active medication list (51.5% vs 50.4%; RR, 1.03; 95% CI, 0.90-1.17); and referral or visit to a nephrologist (38.7% vs 36.1%; RR, 1.02; 95% CI, 0.79-1.32).LimitationsWe encountered an overall reduction in expected primary care encounters and obstacles to point-of-care CKD-CDS utilization because of the coronavirus disease 2019 pandemic.ConclusionsThe CKD-CDS intervention did not lead to a significant improvement in CKD quality metrics. The challenges to CDS use during the coronavirus disease 2019 pandemic likely influenced these results.FundingNational Institute of Diabetes and Digestive and Kidney Diseases (R18DK118463).Trial registrationclinicaltrials.gov Identifier: NCT03890588.

Project description:BackgroundA clinical diagnostic support system (CDSS) can support medical students and physicians in providing evidence-based care. In this study, we investigate diagnostic accuracy based on the history of present illness between groups of medical students using a CDSS, Google, and neither (control). Further, the degree of diagnostic accuracy of medical students using a CDSS is compared with that of residents using neither a CDSS nor Google.MethodsThis study is a randomized educational trial. The participants comprised 64 medical students and 13 residents who rotated in the Department of General Medicine at Chiba University Hospital from May to December 2020. The medical students were randomly divided into the CDSS group (n = 22), Google group (n = 22), and control group (n = 20). Participants were asked to provide the three most likely diagnoses for 20 cases, mainly a history of a present illness (10 common and 10 emergent diseases). Each correct diagnosis was awarded 1 point (maximum 20 points). The mean scores of the three medical student groups were compared using a one-way analysis of variance. Furthermore, the mean scores of the CDSS, Google, and residents' (without CDSS or Google) groups were compared.ResultsThe mean scores of the CDSS (12.0 ± 1.3) and Google (11.9 ± 1.1) groups were significantly higher than those of the control group (9.5 ± 1.7; p = 0.02 and p = 0.03, respectively). The residents' group's mean score (14.7 ± 1.4) was higher than the mean scores of the CDSS and Google groups (p = 0.01). Regarding common disease cases, the mean scores were 7.4 ± 0.7, 7.1 ± 0.7, and 8.2 ± 0.7 for the CDSS, Google, and residents' groups, respectively. There were no significant differences in mean scores (p = 0.1).ConclusionsMedical students who used the CDSS and Google were able to list differential diagnoses more accurately than those using neither. Furthermore, they could make the same level of differential diagnoses as residents in the context of common diseases.Trial registrationThis study was retrospectively registered with the University Hospital Medical Information Network Clinical Trials Registry on 24/12/2020 (unique trial number: UMIN000042831).

Project description:BackgroundClinical decision support (CDS) is a valuable feature of electronic health records (EHRs) designed to improve quality and safety. However, due to the complexities of system design and inconsistent results, CDS tools may inadvertently increase alert fatigue and contribute to physician burnout. A/B testing, or rapid-cycle randomized tests, is a useful method that can be applied to the EHR in order to rapidly understand and iteratively improve design choices embedded within CDS tools.ObjectiveThis paper describes how rapid randomized controlled trials (RCTs) embedded within EHRs can be used to quickly ascertain the superiority of potential CDS design changes to improve their usability, reduce alert fatigue, and promote quality of care.MethodsA multistep process combining tools from user-centered design, A/B testing, and implementation science was used to understand, ideate, prototype, test, analyze, and improve each candidate CDS. CDS engagement metrics (alert views, acceptance rates) were used to evaluate which CDS version is superior.ResultsTo demonstrate the impact of the process, 2 experiments are highlighted. First, after multiple rounds of usability testing, a revised CDS influenza alert was tested against usual care CDS in a rapid (~6 weeks) RCT. The new alert text resulted in minimal impact on reducing firings per patients per day, but this failure triggered another round of review that identified key technical improvements (ie, removal of dismissal button and firings in procedural areas) that led to a dramatic decrease in firings per patient per day (23.1 to 7.3). In the second experiment, the process was used to test 3 versions (financial, quality, regulatory) of text supporting tobacco cessation alerts as well as 3 supporting images. Based on 3 rounds of RCTs, there was no significant difference in acceptance rates based on the framing of the messages or addition of images.ConclusionsThese experiments support the potential for this new process to rapidly develop, deploy, and rigorously evaluate CDS within an EHR. We also identified important considerations in applying these methods. This approach may be an important tool for improving the impact of and experience with CDS.Trial registrationFlu alert trial: ClinicalTrials.gov NCT03415425; https://clinicaltrials.gov/ct2/show/NCT03415425. Tobacco alert trial: ClinicalTrials.gov NCT03714191; https://clinicaltrials.gov/ct2/show/NCT03714191.

Project description:OBJECTIVE: The aim of this study was to compare the clinical impact of computerized decision support with and without electronic access to clinical guidelines and laboratory data on antibiotic prescribing decisions. DESIGN: A crossover trial was conducted of four levels of computerized decision support-no support, antibiotic guidelines, laboratory reports, and laboratory reports plus a decision support system (DSS), randomly allocated to eight simulated clinical cases accessed by the Web. MEASUREMENTS: Rate of intervention adoption was measured by frequency of accessing information support, cost of use was measured by time taken to complete each case, and effectiveness of decision was measured by correctness of and self-reported confidence in individual prescribing decisions. Clinical impact score was measured by adoption rate and decision effectiveness. RESULTS: Thirty-one intensive care and infectious disease specialist physicians (ICPs and IDPs) participated in the study. Ventilator-associated pneumonia treatment guidelines were used in 24 (39%) of the 62 case scenarios for which they were available, microbiology reports in 36 (58%), and the DSS in 37 (60%). The use of all forms of information support did not affect clinicians' confidence in their decisions. Their use of the DSS plus microbiology report improved the agreement of decisions with those of an expert panel from 65% to 97% (p=0.0002), or to 67% (p=0.002) when antibiotic guidelines only were accessed. Significantly fewer IDPs than ICPs accessed information support in making treatment decisions. On average, it took 245 seconds to make a decision using the DSS compared with 113 seconds for unaided prescribing (p<0.001). The DSS plus microbiology reports had the highest clinical impact score (0.58), greater than that of electronic guidelines (0.26) and electronic laboratory reports (0.45). CONCLUSION: When used, computer-based decision support significantly improved decision quality. In measuring the impact of decision support systems, both their effectiveness in improving decisions and their likely rate of adoption in the clinical environment need to be considered. Clinicians chose to use antibiotic guidelines for one third and microbiology reports or the DSS for about two thirds of cases when they were available to assist their prescribing decisions.

Project description:BackgroundClinical decision support systems are implemented in many hospitals to prevent medication errors and associated harm. They are however associated with a high burden of false positive alerts and alert fatigue. The aim of this study was to evaluate a drug-drug interaction (DDI) clinical decision support system in terms of its performance, uptake and user satisfaction and to identify barriers and opportunities for improvement.MethodsA quantitative evaluation and end-user survey were performed in a large teaching hospital. First, very severe DDI alerts generated between 2019 and 2021 were evaluated retrospectively. Data collection comprised alert burden, override rates, the number of alert overrides reviewed by pharmacists and the resulting pharmacist recommendations as well as their acceptance rate. Second, an e-survey was carried out among prescribers to assess satisfaction, usefulness and relevance of DDI alerts as well as reasons for overriding.ResultsA total of 38,409 very severe DDI alerts were generated, of which 88.2% were overridden by the prescriber. In 3.2% of reviewed overrides, a recommendation by the pharmacist was provided, of which 79.2% was accepted. False positive alerts were caused by a too broad screening interval and lack of incorporation of patient-specific characteristics, such as QTc values. Co-prescribing of a non-vitamin K oral anticoagulant and a low molecular weight heparin accounted for 49.8% of alerts, of which 92.2% were overridden. In 88 (1.1%) of these overridden alerts, concurrent therapy was still present. Despite the high override rate, the e-survey revealed that the DDI clinical decision support system was found useful by prescribers.ConclusionsIdentified barriers were the lack of DDI-specific screening intervals and inclusion of patient-specific characteristics, both leading to a high number of false positive alerts and risk for alert fatigue. Despite these barriers, the added value of the DDI clinical decision support system was recognized by prescribers. Hence, integration of DDI-specific screening intervals and patient-specific characteristics is warranted to improve the performance of the DDI software.

Project description:BACKGROUND:Emergency departments (EDs) frequently care for individuals with opioid use disorder (OUD). Buprenorphine (BUP) is an effective treatment option for patients with OUD that can safely be initiated in the ED. At present, BUP is rarely initiated as a part of routine ED care. Clinical decision support (CDS) could accelerate adoption of ED-initiated BUP into routine emergency care. OBJECTIVE:This study aimed to design and formatively evaluate a user-centered decision support tool for ED initiation of BUP for patients with OUD. METHODS:User-centered design with iterative prototype development was used. Initial observations and interviews identified workflows and information needs. The design team and key stakeholders reviewed prototype designs to ensure accuracy. A total of 5 prototypes were evaluated and iteratively refined based on input from 26 attending and resident physicians. RESULTS:Early feedback identified concerns with the initial CDS design: an alert with several screens. The timing of the alert led to quick dismissal without using the tool. User feedback on subsequent iterations informed the development of a flexible tool to support clinicians with varied levels of experience with the intervention by providing both one-click options for direct activation of care pathways and user-activated support for critical decision points. The final design resolved challenging navigation issues through targeted placement, color, and design of the decision support modules and care pathways. In final testing, users expressed that the tool could be easily learned without training and was reasonable for use during routine emergency care. CONCLUSIONS:A user-centered design process helped designers to better understand users' needs for a Web-based clinical decision tool to support ED initiation of BUP for OUD. The process identified varying needs across user experience and familiarity with the protocol, leading to a flexible design supporting both direct care pathways and user-initiated decision support.

Dataset Information

OrderRex clinical user testing: a randomized trial of recommender system decision support on simulated cases.

Publications

OrderRex clinical user testing: a randomized trial of recommender system decision support on simulated cases.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets