Dataset Information

Assessing the Performance of a New Artificial Intelligence-Driven Diagnostic Support Tool Using Medical Board Exam Simulations: Clinical Vignette Study.

ABSTRACT:

Background

Diagnostic decision support systems (DDSS) are computer programs aimed to improve health care by supporting clinicians in the process of diagnostic decision-making. Previous studies on DDSS demonstrated their ability to enhance clinicians' diagnostic skills, prevent diagnostic errors, and reduce hospitalization costs. Despite the potential benefits, their utilization in clinical practice is limited, emphasizing the need for new and improved products.

Objective

The aim of this study was to conduct a preliminary analysis of the diagnostic performance of "Kahun," a new artificial intelligence-driven diagnostic tool.

Methods

Diagnostic performance was evaluated based on the program's ability to "solve" clinical cases from the United States Medical Licensing Examination Step 2 Clinical Skills board exam simulations that were drawn from the case banks of 3 leading preparation companies. Each case included 3 expected differential diagnoses. The cases were entered into the Kahun platform by 3 blinded junior physicians. For each case, the presence and the rank of the correct diagnoses within the generated differential diagnoses list were recorded. Each diagnostic performance was measured in two ways: first, as diagnostic sensitivity, and second, as case-specific success rates that represent diagnostic comprehensiveness.

Results

The study included 91 clinical cases with 78 different chief complaints and a mean number of 38 (SD 8) findings for each case. The total number of expected diagnoses was 272, of which 174 were different (some appeared more than once). Of the 272 expected diagnoses, 231 (87.5%; 95% CI 76-99) diagnoses were suggested within the top 20 listed diagnoses, 209 (76.8%; 95% CI 66-87) were suggested within the top 10, and 168 (61.8%; 95% CI 52-71) within the top 5. The median rank of correct diagnoses was 3 (IQR 2-6). Of the 91 expected diagnoses, 62 (68%; 95% CI 59-78) of the cases were suggested within the top 20 listed diagnoses, 44 (48%; 95% CI 38-59) within the top 10, and 24 (26%; 95% CI 17-35) within the top 5. Of the 91 expected diagnoses, in 87 (96%; 95% CI 91-100), at least 2 out of 3 of the cases' expected diagnoses were suggested within the top 20 listed diagnoses; 78 (86%; 95% CI 79-93) were suggested within the top 10; and 61 (67%; 95% CI 57-77) within the top 5.

Conclusions

The diagnostic support tool evaluated in this study demonstrated good diagnostic accuracy and comprehensiveness; it also had the ability to manage a wide range of clinical findings.

SUBMITTER: Ben-Shabat N

PROVIDER: S-EPMC8672291 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:Antimalarial drugs are becoming less effective due to the emergence of drug resistance. Resistance has been reported for all available malaria drugs, including artemisinin, thus creating a perpetual need for alternative drug candidates. The traditional drug discovery approach of high throughput screening (HTS) of large compound libraries for identification of new drug leads is time-consuming and resource intensive. While virtual in silico screening is a solution to this problem, however, the generalization of the models is not ideal. Artificial intelligence (AI), utilizing either structure-based or ligand-based approaches, has demonstrated highly accurate performances in the field of chemical property prediction. Leveraging the existing data, AI would be a suitable alternative to blind-search HTS or fingerprint-based virtual screening. The AI model would learn patterns within the data and help to search for hit compounds efficiently. In this work, we introduce DeepMalaria, a deep-learning based process capable of predicting the anti-Plasmodium falciparum inhibitory properties of compounds using their SMILES. A graph-based model is trained on 13,446 publicly available antiplasmodial hit compounds from GlaxoSmithKline (GSK) dataset that are currently being used to find novel drug candidates for malaria. We validated this model by predicting hit compounds from a macrocyclic compound library and already approved drugs that are used for repurposing. We have chosen macrocyclic compounds as these ligand-binding structures are underexplored in malaria drug discovery. The in silico pipeline for this process also consists of additional validation of an in-house independent dataset consisting mostly of natural product compounds. Transfer learning from a large dataset was leveraged to improve the performance of the deep learning model. To validate the DeepMalaria generated hits, we used a commonly used SYBR Green I fluorescence assay based phenotypic screening. DeepMalaria was able to detect all the compounds with nanomolar activity and 87.5% of the compounds with greater than 50% inhibition. Further experiments to reveal the compounds' mechanism of action have shown that not only does one of the hit compounds, DC-9237, inhibits all asexual stages of Plasmodium falciparum, but is a fast-acting compound which makes it a strong candidate for further optimization.

Project description:BackgroundArtificial intelligence (AI)-driven serious games have been used in health care to offer a customizable and immersive experience. Summarizing the features of the current AI-driven serious games is very important to explore how they have been developed and used and their current state to plan on how to leverage them in the current and future health care needs.ObjectiveThis study aimed to explore the features of AI-driven serious games in health care as reported by previous research.MethodsWe conducted a scoping review to achieve the abovementioned objective. The most popular databases in the information technology and health fields (ie, MEDLINE, PsycInfo, Embase, CINAHL, IEEE Xplore, ACM Digital Library, and Google Scholar) were searched using keywords related to serious games and AI. Two reviewers independently performed the study selection process. Three reviewers independently extracted data from the included studies. A narrative approach was used for data synthesis.ResultsThe search process returned 1470 records. Of these 1470 records, 46 (31.29%) met all eligibility criteria. A total of 64 different serious games were found in the included studies. Motor impairment was the most common health condition targeted by these serious games. Serious games were used for rehabilitation in most of the studies. The most common genres of serious games were role-playing games, puzzle games, and platform games. Unity was the most prominent game engine used to develop serious games. PCs were the most common platform used to play serious games. The most common algorithm used in the included studies was support vector machine. The most common purposes of AI were the detection of disease and the evaluation of user performance. The size of the data set ranged from 36 to 795,600. The most common validation techniques used in the included studies were k-fold cross-validation and training-test split validation. Accuracy was the most commonly used metric for evaluating the performance of AI models.ConclusionsThe last decade witnessed an increase in the development of AI-driven serious games for health care purposes, targeting various health conditions, and leveraging multiple AI algorithms; this rising trend is expected to continue for years to come. Although the evidence uncovered in this study shows promising applications of AI-driven serious games, larger and more rigorous, diverse, and robust studies may be needed to examine the efficacy and effectiveness of AI-driven serious games in different populations with different health conditions.