Clinician involvement in research on machine learning-based predictive clinical decision support for the hospital setting: A scoping review.
Ontology highlight
ABSTRACT: Objective
The study sought to describe the prevalence and nature of clinical expert involvement in the development, evaluation, and implementation of clinical decision support systems (CDSSs) that utilize machine learning to analyze electronic health record data to assist nurses and physicians in prognostic and treatment decision making (ie, predictive CDSSs) in the hospital.Materials and methods
A systematic search of PubMed, CINAHL, and IEEE Xplore and hand-searching of relevant conference proceedings were conducted to identify eligible articles. Empirical studies of predictive CDSSs using electronic health record data for nurses or physicians in the hospital setting published in the last 5 years in peer-reviewed journals or conference proceedings were eligible for synthesis. Data from eligible studies regarding clinician involvement, stage in system design, predictive CDSS intention, and target clinician were charted and summarized.Results
Eighty studies met eligibility criteria. Clinical expert involvement was most prevalent at the beginning and late stages of system design. Most articles (95%) described developing and evaluating machine learning models, 28% of which described involving clinical experts, with nearly half functioning to verify the clinical correctness or relevance of the model (47%).Discussion
Involvement of clinical experts in predictive CDSS design should be explicitly reported in publications and evaluated for the potential to overcome predictive CDSS adoption challenges.Conclusions
If present, clinical expert involvement is most prevalent when predictive CDSS specifications are made or when system implementations are evaluated. However, clinical experts are less prevalent in developmental stages to verify clinical correctness, select model features, preprocess data, or serve as a gold standard.
SUBMITTER: Schwartz JM
PROVIDER: S-EPMC7936403 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BackgroundClinician trust in machine learning-based clinical decision support systems (CDSSs) for predicting in-hospital deterioration (a type of predictive CDSS) is essential for adoption. Evidence shows that clinician trust in predictive CDSSs is influenced by perceived understandability and perceived accuracy.ObjectiveThe aim of this study was to explore the phenomenon of clinician trust in predictive CDSSs for in-hospital deterioration by confirming and characterizing factors known to influence trust (understandability and accuracy), uncovering and describing other influencing factors, and comparing nurses' and prescribing providers' trust in predictive CDSSs.MethodsWe followed a qualitative descriptive methodology conducting directed deductive and inductive content analysis of interview data. Directed deductive analyses were guided by the human-computer trust conceptual framework. Semistructured interviews were conducted with nurses and prescribing providers (physicians, physician assistants, or nurse practitioners) working with a predictive CDSS at 2 hospitals in Mass General Brigham.ResultsA total of 17 clinicians were interviewed. Concepts from the human-computer trust conceptual framework-perceived understandability and perceived technical competence (ie, perceived accuracy)-were found to influence clinician trust in predictive CDSSs for in-hospital deterioration. The concordance between clinicians' impressions of patients' clinical status and system predictions influenced clinicians' perceptions of system accuracy. Understandability was influenced by system explanations, both global and local, as well as training. In total, 3 additional themes emerged from the inductive analysis. The first, perceived actionability, captured the variation in clinicians' desires for predictive CDSSs to recommend a discrete action. The second, evidence, described the importance of both macro- (scientific) and micro- (anecdotal) evidence for fostering trust. The final theme, equitability, described fairness in system predictions. The findings were largely similar between nurses and prescribing providers.ConclusionsAlthough there is a perceived trade-off between machine learning-based CDSS accuracy and understandability, our findings confirm that both are important for fostering clinician trust in predictive CDSSs for in-hospital deterioration. We found that reliance on the predictive CDSS in the clinical workflow may influence clinicians' requirements for trust. Future research should explore the impact of reliance, the optimal explanation design for enhancing understandability, and the role of perceived actionability in driving trust.
| S-EPMC9136656 | biostudies-literature
Project description:ImportanceAn increasing number of machine learning (ML)-based clinical decision support systems (CDSSs) are described in the medical literature, but this research focuses almost entirely on comparing CDSS directly with clinicians (human vs computer). Little is known about the outcomes of these systems when used as adjuncts to human decision-making (human vs human with computer).ObjectivesTo conduct a systematic review to investigate the association between the interactive use of ML-based diagnostic CDSSs and clinician performance and to examine the extent of the CDSSs' human factors evaluation.Evidence reviewA search of MEDLINE, Embase, PsycINFO, and grey literature was conducted for the period between January 1, 2010, and May 31, 2019. Peer-reviewed studies published in English comparing human clinician performance with and without interactive use of an ML-based diagnostic CDSSs were included. All metrics used to assess human performance were considered as outcomes. The risk of bias was assessed using Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) and Risk of Bias in Non-Randomised Studies-Intervention (ROBINS-I). Narrative summaries were produced for the main outcomes. Given the heterogeneity of medical conditions, outcomes of interest, and evaluation metrics, no meta-analysis was performed.FindingsA total of 8112 studies were initially retrieved and 5154 abstracts were screened; of these, 37 studies met the inclusion criteria. The median number of participating clinicians was 4 (interquartile range, 3-8). Of the 107 results that reported statistical significance, 54 (50%) were increased by the use of CDSSs, 4 (4%) were decreased, and 49 (46%) showed no change or an unclear change. In the subgroup of studies carried out in representative clinical settings, no association between the use of ML-based diagnostic CDSSs and improved clinician performance could be observed. Interobserver agreement was the commonly reported outcome whose change was the most strongly associated with CDSS use. Four studies (11%) reported on user feedback, and, in all but 1 case, clinicians decided to override at least some of the algorithms' recommendations. Twenty-eight studies (76%) were rated as having a high risk of bias in at least 1 of the 4 QUADAS-2 core domains, and 6 studies (16%) were considered to be at serious or critical risk of bias using ROBINS-I.Conclusions and relevanceThis systematic review found only sparse evidence that the use of ML-based CDSSs is associated with improved clinician diagnostic performance. Most studies had a low number of participants, were at high or unclear risk of bias, and showed little or no consideration for human factors. Caution should be exercised when estimating the current potential of ML to improve human diagnostic performance, and more comprehensive evaluation should be conducted before deploying ML-based CDSSs in clinical settings. The results highlight the importance of considering supported human decisions as end points rather than merely the stand-alone CDSSs outputs.
| S-EPMC7953308 | biostudies-literature
Project description:ObjectivesUsing theoretical frameworks from implementation science, we aimed to systematically explore the barriers and enablers to research active allied health professionals (AHP) participating and leading research in the hospital setting.DesignA qualitative interview study informed by behaviour change theory.SettingSingle Australian tertiary hospital and health service.ParticipantsWe recruited a convenience sample of 21 AHPs working within a hospital who were seeking to actively participate in/or lead research within their workplace.Data collectionSemistructured interviews explored perceived barriers and enablers to research participation, informed by the 14 domains of the theoretical domains framework (TDF). Transcribed interviews were deductively coded and mapped to the TDF. A deeper level of inductive coding was used to identify emergent themes that influenced behaviour change, according to the three key constructs of: capability, opportunity and motivation (COM-B).ResultsBarriers and enablers to research participation were identified within nine predominant domains of the TDF. Most enablers to engaging in research related to the motivation or opportunity constructs of the COM-B. These enablers included positive beliefs about the consequences of research participation, enabling social influences, peer support and motivation for skill development and to inform practice. Predominant barriers related to environmental context and resources (eg, reduced funding or time), emotional responses of being overwhelmed and perceptions of reduced capability.ConclusionThis study identified key barriers and enablers to behaviour change related to AHPs participating and/or leading research. Motivation and opportunities to participate in research may be enabled by maximising social influence opportunities, reiterating beliefs about positive consequences of research and considering AHP's emotional responses. Implementation science frameworks may provide a more systematic and holistic understanding of factors which influence research participation including enhancing knowledge, motivation and opportunity.
| S-EPMC7443264 | biostudies-literature
Project description:ObjectiveWe describe the Clickbusters initiative implemented at Vanderbilt University Medical Center (VUMC), which was designed to improve safety and quality and reduce burnout through the optimization of clinical decision support (CDS) alerts.Materials and methodsWe developed a 10-step Clickbusting process and implemented a program that included a curriculum, CDS alert inventory, oversight process, and gamification. We carried out two 3-month rounds of the Clickbusters program at VUMC. We completed descriptive analyses of the changes made to alerts during the process, and of alert firing rates before and after the program.ResultsPrior to Clickbusters, VUMC had 419 CDS alerts in production, with 488 425 firings (42 982 interruptive) each week. After 2 rounds, the Clickbusters program resulted in detailed, comprehensive reviews of 84 CDS alerts and reduced the number of weekly alert firings by more than 70 000 (15.43%). In addition to the direct improvements in CDS, the initiative also increased user engagement and involvement in CDS.ConclusionsAt VUMC, the Clickbusters program was successful in optimizing CDS alerts by reducing alert firings and resulting clicks. The program also involved more users in the process of evaluating and improving CDS and helped build a culture of continuous evaluation and improvement of clinical content in the electronic health record.
| S-EPMC9093034 | biostudies-literature
Project description:Importance:Sophisticated evidence-based information resources can filter medical evidence from the literature, integrate it into electronic health records, and generate recommendations tailored to individual patients. Objective:To assess the effectiveness of a computerized clinical decision support system (CDSS) that preappraises evidence and provides health professionals with actionable, patient-specific recommendations at the point of care. Design, Setting, and Participants:Open-label, parallel-group, randomized clinical trial among internal medicine wards of a large Italian general hospital. All analyses in this randomized clinical trial followed the intent-to-treat principle. Between November 1, 2015, and December 31, 2016, patients were randomly assigned to the intervention group, in which CDSS-generated reminders were displayed to physicians, or to the control group, in which reminders were generated but not shown. Data were analyzed between February 1 and July 31, 2018. Interventions:Evidence-Based Medicine Electronic Decision Support (EBMEDS), a commercial CDSS covering a wide array of health conditions across specialties, was integrated into the hospital electronic health records to generate patient-specific recommendations. Main Outcomes and Measures:The primary outcome was the resolution rate, the rate at which medical problems identified and alerted by the CDSS were addressed by a change in practice. Secondary outcomes included the length of hospital stay and in-hospital all-cause mortality. Results:In this randomized clinical trial, 20 563 patients were admitted to the hospital. Of these, 6480 (31.5%) were admitted to the internal medicine wards (study population) and randomized (3242 to CDSS and 3238 to control). The mean (SD) age of patients was 70.5 (17.3) years, and 54.5% were men. In total, 28 394 reminders were generated throughout the course of the trial (median, 3 reminders per patient per hospital stay; interquartile range [IQR], 1-6). These messages led to a change in practice in approximately 4 of 100 patients. The resolution rate was 38.0% (95% CI, 37.2%-38.8%) in the intervention group and 33.7% (95% CI, 32.9%-34.4%) in the control group, corresponding to an odds ratio of 1.21 (95% CI, 1.11-1.32; P < .001). The length of hospital stay did not differ between the groups, with a median time of 8 days (IQR, 5-13 days) for the intervention group and a median time of 8 days (IQR, 5-14 days) for the control group (P = .36). In-hospital all-cause mortality also did not differ between groups (odds ratio, 0.95; 95% CI, 0.77-1.17; P = .59). Alert fatigue did not differ between early and late study periods. Conclusions and Relevance:An international commercial CDSS intervention marginally influenced routine practice in a general hospital, although the change did not statistically significantly affect patient outcomes. Trial Registration:ClinicalTrials.gov identifier: NCT02577198.
| S-EPMC6991299 | biostudies-literature
Project description:There is good evidence that 50% or more of red blood cell (RBC) transfusions are unnecessary. To curtail inappropriate RBC transfusions at our hospital, real-time clinical decision support was implemented in our electronic medical record (EMR) that alerts clinicians to the patient's most recent pretransfusion hemoglobin value upon order entry and provides Best Practice Advisory. This is a soft pop-up alert which is activated when the hemoglobin exceeds 7 g/dL. The ordering clinician can either honor (by cancelling the order) or override the alert. We studied the impact of the alert on blood utilization during a 3-month period (November 2016 to January 2017). For patients who were transfused despite the alert, a retrospective review of the medical chart was performed to determine whether or not the transfusion was clinically indicated. During the study period, 178 of the 895 RBC transfusion orders (20%) triggered the alert. After excluding duplicates, 144 orders were included in our analysis. Most of these orders (124/144, 86%) were carried out despite the alert. According to our chart review, 48% of the alert transfusions could be considered inappropriate, with hemodynamically stable, asymptomatic anemia being the leading indication. Of clinical services, orthopedic surgery had the highest rate of overriding the alert with no clinical justification (70%). The number of RBC transfusions dropped from 313.5 units per month (preintervention period) to 293.2 units per month (postintervention period)-a 6.5% decrease. Real-time clinical decision support may reduce the number of inappropriate RBC transfusions in a community hospital setting, though in our study, the decrease in blood utilization (6.5%) did not reach statistical significance.
| S-EPMC6164780 | biostudies-other
Project description:BackgroundReal-time clinical decision support (CDS) integrated into clinicians' workflow has the potential to profoundly affect the cost, quality, and safety of health care delivery. Recent reports have identified a surprisingly low acceptance rate for different types of CDS. We hypothesized that factors affecting CDS system acceptance could be categorized as relating to differences in patients, physicians, CDS-type, or environmental characteristics.MethodsWe conducted a survey of all adult primary care physicians (PCPs, n = 225) within our group model Health Maintenance Organization (HMO) to identify factors that affect their acceptance of CDS. We defined clinical decision support broadly as "clinical information" that is either provided to you or accessible by you, from the clinical workstation (e.g., enhanced flow sheet displays, health maintenance reminders, alternative medication suggestions, order sets, alerts, and access to any internet-based information resources).Results110 surveys were returned (49%). There were no differences in the age, gender, or years of service between those who returned the survey and the entire adult PCP population. Overall, clinicians stated that the CDS provided "helps them take better care of their patients" (3.6 on scale of 1:Never-5:Always), "is worth the time it takes" (3.5), and "reminds them of something they've forgotten" (3.2). There was no difference in the perceived acceptance rate of alerts based on their type (i.e., cost, safety, health maintenance). When asked about specific patient characteristics that would make the clinicians "more", "equally" or "less" likely to accept alerts: 41% stated that they were more (8% stated "less") likely to accept alerts on elderly patients (> 65 yrs); 38% were more (14% stated less) likely to accept alerts on patients with more than 5 current medications; and 38% were more (20% stated less) likely to accept alerts on patients with more than 5 chronic clinical conditions. Interestingly, 80% said they were less likely to accept alerts when they were behind schedule and 84% of clinicians admitted to being at least 20 minutes behind schedule "some", "most", or "all of the time".ConclusionEven though a majority of our clinical decision support suggestions are not explicitly followed, clinicians feel they are of benefit and would be even more beneficial if they had more time available to address them.
| S-EPMC1403751 | biostudies-literature
Project description:Thoughtful integration of interruptive clinical decision support (CDS) alerts within the electronic health record is essential to guide clinicians on the application of pharmacogenomic results at point of care. St. Jude Children's Research Hospital implemented a preemptive pharmacogenomic testing program in 2011 in a multidisciplinary effort involving extensive education to clinicians about pharmacogenomic implications. We conducted a retrospective analysis of clinicians' adherence to 4783 pharmacogenomically guided CDS alerts that triggered for 12 genes and 60 drugs. Clinicians adhered to the therapeutic recommendations provided in 4392 alerts (92%). In our population of pediatric patients with catastrophic illnesses, the most frequently presented gene/drug CDS alerts were TPMT/NUDT15 and thiopurines (n = 3850), CYP2D6 and ondansetron (n = 667), CYP2D6 and oxycodone (n = 99), G6PD and G6PD high-risk medications (n = 51), and CYP2C19 and proton pump inhibitors (omeprazole and pantoprazole; n = 50). The high adherence rate was facilitated by our team approach to prescribing and our collaborative CDS design and delivery.
| S-EPMC9748527 | biostudies-literature
Project description:AKI carries a significant mortality and morbidity risk. Use of a clinical decision support system (CDSS) might improve outcomes. We conducted a multicenter, sequential period analysis of 528,108 patients without ESRD before admission, from October of 2012 to September of 2015, to determine whether use of a CDSS reduces hospital length of stay and in-hospital mortality for patients with AKI. We compared patients treated 12 months before (181,696) and 24 months after (346,412) implementation of the CDSS. Coprimary outcomes were hospital mortality and length of stay adjusted by demographics and comorbidities. AKI was diagnosed in 64,512 patients (12.2%). Crude mortality rate fell from 10.2% before to 9.4% after CDSS implementation (odds ratio, 0.91; 95% confidence interval [95% CI], 0.86 to 0.96; P=0.001) for patients with AKI but did not change in patients without AKI (from 1.5% to 1.4%). Mean hospital duration decreased from 9.3 to 9.0 days (P<0.001) for patients with AKI, with no change for patients without AKI. In multivariate mixed-effects models, the adjusted odds ratio (95% CI) was 0.76 (0.70 to 0.83) for mortality and 0.66 (0.61 to 0.72) for dialysis (P<0.001). Change in adjusted hospital length of stay was also significant (incidence rate ratio, 0.91; 95% CI, 0.89 to 0.92), decreasing from 7.2 to 6.0 days for patients with AKI. Results were robust to sensitivity analyses and were sustained for the duration of follow-up. Hence, implementation of a CDSS for AKI resulted in a small but sustained decrease in hospital mortality, dialysis use, and length of stay.
| S-EPMC5791078 | biostudies-literature
Project description:BackgroundThis scoping review provides a thorough analysis of how stakeholders have so far been involved in research priority setting. The review describes, synthesizes, and evaluates research priority setting projects not only for the field of health-as previous reviews have done-but does so on a much broader scale for any research area.MethodsA comprehensive electronic literature search was conducted in the databases PubMed, Scopus, and Web of Science. Reflecting the importance of grey literature, Google Scholar and relevant websites were also screened for eligible publications. A computational approach was then used for the study selection. The final screening for inclusion was done manually.ResultsThe scoping review encompasses 731 research priority setting projects published until the end of 2020. Overall, the projects were conducted within the realm of 50 subject areas ranging from agriculture and environment over health to social work and technology. Key learnings include that nearly all priority setting projects aimed to identify research priorities for the field of health (93%), particularly for nursing and care, cancer, pediatrics, and mental, behavioral and neurodevelopmental disorders. Only 6% of the projects were not health-related and 1% identified research priorities at the interface between health and a non-health area. Over time, 30 different stakeholder groups took part in research priority setting. The stakeholders most frequently asked to identify research priorities were doctors, patients, academics/researchers, nurses, allied healthcare professionals, family members, friends, and carers. Nearly two thirds of all projects have been conducted in Europe and North America. Overall, only 9% of the projects emphasized the importance of stakeholders in their goals and rationales and actively involved them. In around a quarter of the projects, stakeholders deliberated on their research priorities throughout the entire process.ConclusionBy mapping out the complex landscape of stakeholder involvement in research priority setting, this review guides future efforts to involve stakeholders effectively, inclusively, and transparently, which in turn may increase the overall value of research for society. As a practical addition to this review, the first worldwide research priority setting database was created: https://ois.lbg.ac.at/en/project-database . The database contains all the projects analyzed for this review and is constantly updated with the latest published research priority setting projects.
| S-EPMC8555197 | biostudies-literature