Dataset Information

A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare.

ABSTRACT: BACKGROUND:There is an increasing interest in clinical prediction tools that can achieve high prediction accuracy and provide explanations of the factors leading to increased risk of adverse outcomes. However, approaches to explaining complex machine learning (ML) models are rarely informed by end-user needs and user evaluations of model interpretability are lacking in the healthcare domain. We used extended revisions of previously-published theoretical frameworks to propose a framework for the design of user-centered displays of explanations. This new framework served as the basis for qualitative inquiries and design review sessions with critical care nurses and physicians that informed the design of a user-centered explanation display for an ML-based prediction tool. METHODS:We used our framework to propose explanation displays for predictions from a pediatric intensive care unit (PICU) in-hospital mortality risk model. Proposed displays were based on a model-agnostic, instance-level explanation approach based on feature influence, as determined by Shapley values. Focus group sessions solicited critical care provider feedback on the proposed displays, which were then revised accordingly. RESULTS:The proposed displays were perceived as useful tools in assessing model predictions. However, specific explanation goals and information needs varied by clinical role and level of predictive modeling knowledge. Providers preferred explanation displays that required less information processing effort and could support the information needs of a variety of users. Providing supporting information to assist in interpretation was seen as critical for fostering provider understanding and acceptance of the predictions and explanations. The user-centered explanation display for the PICU in-hospital mortality risk model incorporated elements from the initial displays along with enhancements suggested by providers. CONCLUSIONS:We proposed a framework for the design of user-centered displays of explanations for ML models. We used the proposed framework to motivate the design of a user-centered display of an explanation for predictions from a PICU in-hospital mortality risk model. Positive feedback from focus group participants provides preliminary support for the use of model-agnostic, instance-level explanations of feature influence as an approach to understand ML model predictions in healthcare and advances the discussion on how to effectively communicate ML model information to healthcare providers.

SUBMITTER: Barda AJ

PROVIDER: S-EPMC7545557 | biostudies-literature | 2020 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare.

Barda Amie J AJ Horvat Christopher M CM Hochheiser Harry H

BMC medical informatics and decision making 20201008 1

<h4>Background</h4>There is an increasing interest in clinical prediction tools that can achieve high prediction accuracy and provide explanations of the factors leading to increased risk of adverse outcomes. However, approaches to explaining complex machine learning (ML) models are rarely informed by end-user needs and user evaluations of model interpretability are lacking in the healthcare domain. We used extended revisions of previously-published theoretical frameworks to propose a framework ...[more]

PMID: 33032582

Similar Datasets

Project description:BackgroundDespite the unprecedented performance of deep learning algorithms in clinical domains, full reviews of algorithmic predictions by human experts remain mandatory. Under these circumstances, artificial intelligence (AI) models are primarily designed as clinical decision support systems (CDSSs). However, from the perspective of clinical practitioners, the lack of clinical interpretability and user-centered interfaces hinders the adoption of these AI systems in practice.ObjectiveThis study aims to develop an AI-based CDSS for assisting polysomnographic technicians in reviewing AI-predicted sleep staging results. This study proposed and evaluated a CDSS that provides clinically sound explanations for AI predictions in a user-centered manner.MethodsOur study is based on a user-centered design framework for developing explanations in a CDSS that identifies why explanations are needed, what information should be contained in explanations, and how explanations can be provided in the CDSS. We conducted user interviews, user observation sessions, and an iterative design process to identify three key aspects for designing explanations in the CDSS. After constructing the CDSS, the tool was evaluated to investigate how the CDSS explanations helped technicians. We measured the accuracy of sleep staging and interrater reliability with macro-F1 and Cohen κ scores to assess quantitative improvements after our tool was adopted. We assessed qualitative improvements through participant interviews that established how participants perceived and used the tool.ResultsThe user study revealed that technicians desire explanations that are relevant to key electroencephalogram (EEG) patterns for sleep staging when assessing the correctness of AI predictions. Here, technicians wanted explanations that could be used to evaluate whether the AI models properly locate and use these patterns during prediction. On the basis of this, information that is closely related to sleep EEG patterns was formulated for the AI models. In the iterative design phase, we developed a different visualization strategy for each pattern based on how technicians interpreted the EEG recordings with these patterns during their workflows. Our evaluation study on 9 polysomnographic technicians quantitatively and qualitatively investigated the helpfulness of the tool. For technicians with <5 years of work experience, their quantitative sleep staging performance improved significantly from 56.75 to 60.59 with a P value of .05. Qualitatively, participants reported that the information provided effectively supported them, and they could develop notable adoption strategies for the tool.ConclusionsOur findings indicate that formulating clinical explanations for automated predictions using the information in the AI with a user-centered design process is an effective strategy for developing a CDSS for sleep staging.

Project description:Backgrounde-Consultations between primary care physicians and specialists are a valuable means of improving access to specialty care. Adolescents and young adults (AYAs) face unique challenges in accessing limited adolescent medicine specialty care resources, which contributes to delayed or forgone care. e-Consultations between general pediatricians and adolescent medicine specialists may alleviate these barriers to care. However, the optimal application of this model in adolescent medicine requires careful attention to the nuances of AYA care.ObjectiveThis study aims to qualitatively analyze feedback obtained during the iterative development of an e-consultation system for communication between general pediatricians and adolescent medicine specialists tailored to the specific health care needs of AYAs.MethodsWe conducted an iterative user-centered design and evaluation process in two phases. In the first phase, we created a static e-consultation prototype and storyboards and evaluated them with target users (general pediatricians and adolescent medicine specialists). In the second phase, we incorporated feedback to develop a functional prototype within the electronic health record and again evaluated this with general pediatricians and adolescent medicine specialists. In each phase, general pediatricians and adolescent medicine specialists provided think-aloud feedback during the use of the prototypes and semistructured exit interviews, which was qualitatively analyzed to identify perspectives related to the usefulness and usability of the e-consultation system.ResultsBoth general pediatricians (n=12) and adolescent medicine specialists (n=12) perceived the usefulness of e-consultations for AYA patients, with more varied perceptions of potential usefulness for generalist and adolescent medicine clinicians. General pediatricians and adolescent medicine specialists discussed ways to maximize the usability of e-consultations for AYAs, primarily by improving efficiency (eg, reducing documentation, emphasizing critical information, using autopopulated data fields, and balancing specificity and efficiency through text prompts) and reducing the potential for errors (eg, prompting a review of autopopulated data fields, requiring physician contact information, and prompting explicit discussion of patient communication and confidentiality expectations). Through iterative design, patient history documentation was streamlined, whereas documentation of communication and confidentiality expectations were enhanced.ConclusionsThrough an iterative user-centered design process, we identified user perspectives to guide the refinement of an e-consultation system based on general pediatrician and adolescent medicine specialist feedback on usefulness and usability related to the care of AYAs. Qualitative analysis of this feedback revealed both opportunities and risks related to confidentiality, communication, and the use of tailored documentation prompts that should be considered in the development and use of e-consultations with AYAs.

Project description:BackgroundContingency management (CM) is one of the only behavioral interventions shown to be effective for the treatment of opioid use disorders when delivered alone and in combination with pharmacotherapy. Despite extensive empirical support, uptake of CM in community settings remains abysmally low. The current study applied user-centered design principles to gather qualitative data on familiarity with CM, current clinical practice, and preferences regarding the implementation of CM in community-based opioid treatment programs.MethodsParticipants were 21 leaders and 22 front-line counselors from 11 community-based opioid treatment programs. Semi-structured interviews were about 45 min long. Transcripts from each interview were coded by independent raters and analyzed using a reflexive team approach. Frequencies of responses were tallied, and queries were run in NVivo to identify exemplar quotes for each code.ResultsResults indicated low familiarity with CM, with less than half of the respondents defining CM correctly and over 40% of respondents declining to answer/ did not know. Abstinence was the most commonly recommended CM target, yet over 70% of respondents indicated that urine screens only occurred monthly. Attendance was also a popular recommendation, with respondents suggesting a range of possible indices including counseling, dosing, and/or case management sessions. Regarding the ideal role to administer CM prizes, program directors and supervisors were most commonly recommended, closely followed by front-line counselors. The most commonly suggested strategies to afford CM incentives included soliciting community donations and offering non-financial incentives.ConclusionsUser design principles to understand workflow constraints, target user needs, and simplify the intervention guided this qualitative investigation of CM implementation in opioid treatment programs. Findings highlighted the potential value of flexible, organization-specific definitions of CM attendance and non-financial incentives, as well as active involvement of clinical leaders and supervisors to promote buy in among staff/patients. Respondents were generally optimistic about their ability to fundraise or solicit donations to overcome cost-related barriers of CM. Implications for CM implementation strategies, including the use of targeted leadership coaching focused on sustainability, are explored.

Project description:Background Continual improvements to health systems, products, and services are necessary for improvements in health. However, many of these improvements are not incorporated into everyday practice. When designing new health systems, products, and services, involving members of the healthcare community and the public with personal healthcare experience can help to make sure that improvements will be useful and relevant to others like them. Methods Together with healthcare workers and family members with healthcare experience, we developed and applied a step-by-step guide to involving those with personal experience in the design of health system improvements. Results Our guide has three phases- 'Pre-Design', 'Co-Design', and 'Post-Design'. This paper describes each of these phases and illustrates how we applied them to our own project, which is to use virtual healthcare methods to improve care for children with chronic healthcare conditions and their families. In our own work, we found that healthcare workers and family members with personal healthcare experiences were able to use their knowledge and creativity to help us imagine how to improve care for children with chronic healthcare conditions and their families. We have created action items from these family member- and healthcare worker-identified needs, which we will use to shape our virtual healthcare system. Conclusions This paper may be useful for those seeking to involve members of the healthcare community and the public in the creation of better healthcare systems, products, and services. Background Challenges with the adoption, scale, and spread of health innovations represent significant gaps in the evidence-to-practice cycle. In the health innovation design process, a lack of attention paid to the needs of end-users, and subsequent tailoring of innovations to meet these needs, is a possible reason for this deficit. In the creative field of health innovation, which includes the design of healthcare products, systems (governance and organization mechanisms), and services (delivery mechanisms), a framework for both soliciting the needs of end-users and translating these needs into the design of health innovations is needed. Methods To address this gap, our team developed and applied a seven-step methodological framework, called A Generative Co-Design Framework for Healthcare Innovation. This framework was developed by an interdisciplinary team that included patient partners. Results This manuscript contributes a framework and applied exemplar for those seeking to engage end-users in the creative process of healthcare innovation. Through the stages of 'Pre-Design', 'Co-Design', and 'Post-Design', we were able to harness the creative insights of end-users, drawing on their experiences to shape a future state of care. Using an expository example of our own work, the DigiComp Kids project, we illustrate the application of each stage of the Framework. Conclusions A Generative Co-Design Framework for Healthcare Innovation provides healthcare innovators, applied health science researchers, clinicians, and quality improvement specialists with a guide to eliciting and incorporating the viewpoints of end-users while distilling practical considerations for healthcare innovation and design.

Project description:BackgroundAsthma hospital encounters impose a heavy burden on the health care system. To improve preventive care and outcomes for patients with asthma, we recently developed a black-box machine learning model to predict whether a patient with asthma will have one or more asthma hospital encounters in the succeeding 12 months. Our model is more accurate than previous models. However, black-box machine learning models do not explain their predictions, which forms a barrier to widespread clinical adoption. To solve this issue, we previously developed a method to automatically provide rule-based explanations for the model's predictions and to suggest tailored interventions without sacrificing model performance. For an average patient correctly predicted by our model to have future asthma hospital encounters, our explanation method generated over 5000 rule-based explanations, if any. However, the user of the automated explanation function, often a busy clinician, will want to quickly obtain the most useful information for a patient by viewing only the top few explanations. Therefore, a methodology is required to appropriately rank the explanations generated for a patient. However, this is currently an open problem.ObjectiveThe aim of this study is to develop a method to appropriately rank the rule-based explanations that our automated explanation method generates for a patient.MethodsWe developed a ranking method that struck a balance among multiple factors. Through a secondary analysis of 82,888 data instances of adults with asthma from the University of Washington Medicine between 2011 and 2018, we demonstrated our ranking method on the test case of predicting asthma hospital encounters in patients with asthma.ResultsFor each patient predicted to have asthma hospital encounters in the succeeding 12 months, the top few explanations returned by our ranking method typically have high quality and low redundancy. Many top-ranked explanations provide useful insights on the various aspects of the patient's situation, which cannot be easily obtained by viewing the patient's data in the current electronic health record system.ConclusionsThe explanation ranking module is an essential component of the automated explanation function, and it addresses the interpretability issue that deters the widespread adoption of machine learning predictive models in clinical practice. In the next few years, we plan to test our explanation ranking method on predictive modeling problems addressing other diseases as well as on data from other health care systems.International registered report identifier (irrid)RR2-10.2196/5039.

Project description:BackgroundReal-world implementation of psychological interventions for psychosis is poor. Barriers include therapy being insufficiently usable and useful for a diverse range of people. User-centered, inclusive design approaches could improve the usability of therapy, which may increase uptake, adherence, and effectiveness.ObjectiveThis study aimed to optimize the usability of an existing psychological intervention, Thinking Well, which targets reasoning processes in paranoia using a basic digital interface.MethodsWe conducted inclusive, user-centered design research characterized by purposive sampling of extreme users from the margins of groups, ethnographic investigation of the problem context, and iterative prototyping of solutions. The UK Design Council's double diamond method was used. This consisted of 4 phases: discover, including a case series of Thinking Well, stakeholder interviews, desk research, user profiling, system mapping, and a mood board; define, consisting of workshops to synthesize findings and generate the design brief; develop, involving concept workshops and prototype testing; and deliver, in which the final minimal viable product was storyboarded and iteratively coded.ResultsConsistent with our previous work, the Thinking Well case series showed medium to large effects on paranoia and well-being and small effects on reasoning. These were maintained at follow-up despite some participants reporting difficulties with the therapy interface. Insights from the discover phase confirmed that usability was challenged by information complexity and poor accessibility. Participants were generally positive about the potential of technology to be enjoyable, help manage paranoia, and provide tailored interpersonal support from therapists and peers, although they reported privacy and security concerns. The define phase highlighted that the therapy redesign should support monitoring, simplify information processing, enhance enjoyment and trust, promote personalization and normalization, and offer flexible interpersonal support. During the develop phase over 60 concepts were created, with 2 key concepts of thoughts visualized as bubbles and therapy as a journey selected for storyboarding. The output of the deliver phase was a minimal viable product of an innovative digital therapy, SlowMo. SlowMo works by helping people to notice their worries and fast thinking habits, and encourages them to slow down for a moment to find ways of feeling safer. A Web app supports the delivery of 8 face-to-face sessions, which are synchronized to a native mobile app.ConclusionsSlowMo makes use of personalization, ambient information, and visual metaphors to tailor the appeal, engagement, and memorability of therapy to a diversity of needs. Feasibility testing has been promising, and the efficacy of SlowMo therapy is now being tested in a multicentered randomized controlled trial. The study demonstrates that developments in psychological theory and techniques can be enhanced by improving the usability of the therapy interface to optimize its impact in daily life.

Dataset Information

A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare.

Publications

A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets