A Situation Awareness-Based Framework for Design and Evaluation of Explainable AI
Ontology highlight
ABSTRACT:
SUBMITTER: Calvaresi D
PROVIDER: S-EPMC7338174 | biostudies-literature | 2020 Jun
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Studies have shown a link between colorectal cancer (CRC) and gut microbiome compositions. In these studies, machine learning is used to infer CRC biomarkers using global explanation methods. While these methods allow the identification of bacteria generally correlated with CRC, they fail to recognize species that are only influential for some individuals. In this study, we investigate the potential of Shapley Additive Explanations (SHAP) for a more personalized CRC biomarker identification. Analyses of five independent datasets show that this method can even separate CRC subjects into subgroups with distinct CRC probabilities and bacterial biomarkers.
| S-EPMC9912568 | biostudies-literature
Project description:Explainable artificial intelligence (XAI) elucidates the decision-making process of complex AI models and is important in building trust in model predictions. XAI explanations themselves require evaluation as to accuracy and reasonableness and in the context of use of the underlying AI model. This review details the evaluation of XAI in cardiac AI applications and has found that, of the studies examined, 37% evaluated XAI quality using literature results, 11% used clinicians as domain-experts, 11% used proxies or statistical analysis, with the remaining 43% not assessing the XAI used at all. We aim to inspire additional studies within healthcare, urging researchers not only to apply XAI methods but to systematically assess the resulting explanations, as a step towards developing trustworthy and safe models.Supplementary informationThe online version contains supplementary material available at 10.1007/s10462-024-10852-w.
| S-EPMC11315784 | biostudies-literature
Project description:The accuracy and flexibility of artificial intelligence (AI) systems often comes at the cost of a decreased ability to offer an intuitive explanation of their predictions. This hinders trust and discourage adoption of AI in healthcare, exacerbated by concerns over liabilities and risks to patients' health in case of misdiagnosis. Providing an explanation for a model's prediction is possible due to recent advances in the field of interpretable machine learning. We considered a data set of hospital admissions linked to records of antibiotic prescriptions and susceptibilities of bacterial isolates. An appropriately trained gradient boosted decision tree algorithm, supplemented by a Shapley explanation model, predicts the likely antimicrobial drug resistance, with the odds of resistance informed by characteristics of the patient, admission data, and historical drug treatments and culture test results. Applying this AI-based system, we found that it substantially reduces the risk of mismatched treatment compared with the observed prescriptions. The Shapley values provide an intuitive association between observations/data and outcomes; the associations identified are broadly consistent with expectations based on prior knowledge from health specialists. The results, and the ability to attribute confidence and explanations, support the wider adoption of AI in healthcare.
| S-EPMC9931350 | biostudies-literature
Project description:Indecipherable black boxes are common in machine learning (ML), but applications increasingly require explainable artificial intelligence (XAI). The core of XAI is to establish transparent and interpretable data-driven algorithms. This work provides concrete tools for XAI in situations where prior knowledge must be encoded and untrustworthy inferences flagged. We use the "learn to optimize" (L2O) methodology wherein each inference solves a data-driven optimization problem. Our L2O models are straightforward to implement, directly encode prior knowledge, and yield theoretical guarantees (e.g. satisfaction of constraints). We also propose use of interpretable certificates to verify whether model inferences are trustworthy. Numerical examples are provided in the applications of dictionary-based signal recovery, CT imaging, and arbitrage trading of cryptoassets. Code and additional documentation can be found at https://xai-l2o.research.typal.academy .
| S-EPMC10284861 | biostudies-literature
Project description:Transparency in Machine Learning (ML), often also referred to as interpretability or explainability, attempts to reveal the working mechanisms of complex models. From a human-centered design perspective, transparency is not a property of the ML model but an affordance, i.e., a relationship between algorithm and users. Thus, prototyping and user evaluations are critical to attaining solutions that afford transparency. Following human-centered design principles in highly specialized and high stakes domains, such as medical image analysis, is challenging due to the limited access to end users and the knowledge imbalance between those users and ML designers. To investigate the state of transparent ML in medical image analysis, we conducted a systematic review of the literature from 2012 to 2021 in PubMed, EMBASE, and Compendex databases. We identified 2508 records and 68 articles met the inclusion criteria. Current techniques in transparent ML are dominated by computational feasibility and barely consider end users, e.g. clinical stakeholders. Despite the different roles and knowledge of ML developers and end users, no study reported formative user research to inform the design and development of transparent ML models. Only a few studies validated transparency claims through empirical user evaluations. These shortcomings put contemporary research on transparent ML at risk of being incomprehensible to users, and thus, clinically irrelevant. To alleviate these shortcomings in forthcoming research, we introduce the INTRPRT guideline, a design directive for transparent ML systems in medical image analysis. The INTRPRT guideline suggests human-centered design principles, recommending formative user research as the first step to understand user needs and domain requirements. Following these guidelines increases the likelihood that the algorithms afford transparency and enable stakeholders to capitalize on the benefits of transparent ML.
| S-EPMC9581990 | biostudies-literature
Project description:Safe cycling requires situation awareness (SA), which is the basis for recognizing and anticipating hazards. Children have poorer SA than adults, which may put them at risk. This study investigates whether cyclists' SA can be trained with a video-based learning game. The effect of executive working memory on SA was also studied. Thirty-six children (9-10 years) and 22 adults (21-48 years) played the game. The game had 30 video clips filmed from a cyclist's perspective. Each clip was suddenly masked and two or three locations were presented. The player's task was to choose locations with a potential hazard and feedback was given for their answers. Working memory capacity (WMC) was tested with a counting span task. Children's and adults' performance improved while playing the game, which suggests that playing the game trains SA. Adults performed better than children, and they also glanced at hazards more while the video was playing. Children expectedly had a lower WMC than adults, but WMC did not predict performance within the groups. This indicates that SA does not depend on WMC when passively viewing videos.
| S-EPMC5383826 | biostudies-literature
Project description:Explainable artificial intelligence (XAI) is of paramount importance to various domains, including healthcare, fitness, skill assessment, and personal assistants, to understand and explain the decision-making process of the artificial intelligence (AI) model. Smart homes embedded with smart devices and sensors enabled many context-aware applications to recognize physical activities. This study presents XAI-HAR, a novel XAI-empowered human activity recognition (HAR) approach based on key features identified from the data collected from sensors located at different places in a smart home. XAI-HAR identifies a set of new features (i.e., the total number of sensors used in a specific activity), as physical key features selection (PKFS) based on weighting criteria. Next, it presents statistical key features selection (SKFS) (i.e., mean, standard deviation) to handle the outliers and higher class variance. The proposed XAI-HAR is evaluated using machine learning models, namely, random forest (RF), K-nearest neighbor (KNN), support vector machine (SVM), decision tree (DT), naive Bayes (NB) and deep learning models such as deep neural network (DNN), convolution neural network (CNN), and CNN-based long short-term memory (CNN-LSTM). Experiments demonstrate the superior performance of XAI-HAR using RF classifier over all other machine learning and deep learning models. For explainability, XAI-HAR uses Local Interpretable Model Agnostic (LIME) with an RF classifier. XAI-HAR achieves 0.96% of F-score for health and dementia classification and 0.95 and 0.97% for activity recognition of dementia and healthy individuals, respectively.
| S-EPMC10033697 | biostudies-literature
Project description:Neural networks (NNs) are emerging as a rapid and scalable method for quantifying metabolites directly from nuclear magnetic resonance (NMR) spectra, but the nonlinear nature of NNs precludes understanding of how a model makes predictions. This study implements an explainable artificial intelligence algorithm called integrated gradients (IG) to elucidate which regions of input spectra are the most important for the quantification of specific analytes. The approach is first validated in simulated mixture spectra of eight aqueous metabolites and then investigated in experimentally acquired lipid spectra of a reference standard mixture and a murine hepatic extract. The IG method revealed that, like a human spectroscopist, NNs recognize and quantify analytes based on an analyte's respective resonance line-shapes, amplitudes, and frequencies. NNs can compensate for peak overlap and prioritize specific resonances most important for concentration determination. Further, we show how modifying a NN training dataset can affect how a model makes decisions, and we provide examples of how this approach can be used to de-bug issues with model performance. Overall, results show that the IG technique facilitates a visual and quantitative understanding of how model inputs relate to model outputs, potentially making NNs a more attractive option for targeted and automated NMR-based metabolomics.
| S-EPMC11205398 | biostudies-literature
Project description:Explainable AI is deemed essential for clinical applications as it allows rationalizing model predictions, helping to build trust between clinicians and automated decision support tools. We developed an inherently explainable AI model for the classification of acute myeloid leukemia subtypes from blood smears and found that high-attention cells identified by the model coincide with those labeled as diagnostically relevant by human experts. Based on over 80,000 single white blood cell images from digitized blood smears of 129 patients diagnosed with one of four WHO-defined genetic AML subtypes and 60 healthy controls, we trained SCEMILA, a single-cell based explainable multiple instance learning algorithm. SCEMILA could perfectly discriminate between AML patients and healthy controls and detected the APL subtype with an F1 score of 0.86±0.05 (mean±s.d., 5-fold cross-validation). Analyzing a novel multi-attention module, we confirmed that our algorithm focused with high concordance on the same AML-specific cells as human experts do. Applied to classify single cells, it is able to highlight subtype specific cells and deconvolve the composition of a patient's blood smear without the need of single-cell annotation of the training data. Our large AML genetic subtype dataset is publicly available, and an interactive online tool facilitates the exploration of data and predictions. SCEMILA enables a comparison of algorithmic and expert decision criteria and can present a detailed analysis of individual patient data, paving the way to deploy AI in the routine diagnostics for identifying hematopoietic neoplasms.
| S-EPMC10016704 | biostudies-literature
Project description:BACKGROUND:Studies show that implementing huddles in healthcare can improve a variety of outcomes. Yet little is known about the mechanisms through which huddles exert their effects. To help remedy this gap, our study objectives were to explore hospital administrator and frontline staff perspectives on the benefits and challenges of implementing a tiered huddle system; and propose a model based on our findings depicting the mediating pathways through which implementing a huddle system may reduce patient harm. METHODS:Using qualitative methods, we conducted semi-structured interviews and focus groups to obtain a deeper understanding of the huddle system and its outcomes as implemented in an academic tertiary care children's hospital with 539 inpatient beds. We recruited healthcare providers representing all levels using a snowball sampling technique (10 interviews), and emails, flyers, and paper invitations (six focus groups). We transcribed recordings and analysed the data using established techniques. RESULTS:Five themes emerged and provided the foundational constructs of our model. Specifically we propose that huddle implementation leads to improved efficiencies and quality of information sharing, increased levels of accountability, empowerment, and sense of community, which together create a culture of collaboration and collegiality that increases the staff's quality of collective awareness and enhanced capacity for eliminating patient harm. CONCLUSIONS:While each construct in the proposed model is itself a beneficial outcome of implementing huddles, conceptualising the pathways by which they may work allows us to design ways to evaluate other huddle implementation efforts designed to help reduce failures and eliminate patient harm.
| S-EPMC6288816 | biostudies-other