Dataset Information

Efficacy of Artificial-Intelligence-Driven Differential-Diagnosis List on the Diagnostic Accuracy of Physicians: An Open-Label Randomized Controlled Study.

ABSTRACT:

Background

The efficacy of artificial intelligence (AI)-driven automated medical-history-taking systems with AI-driven differential-diagnosis lists on physicians' diagnostic accuracy was shown. However, considering the negative effects of AI-driven differential-diagnosis lists such as omission (physicians reject a correct diagnosis suggested by AI) and commission (physicians accept an incorrect diagnosis suggested by AI) errors, the efficacy of AI-driven automated medical-history-taking systems without AI-driven differential-diagnosis lists on physicians' diagnostic accuracy should be evaluated.

Objective

The present study was conducted to evaluate the efficacy of AI-driven automated medical-history-taking systems with or without AI-driven differential-diagnosis lists on physicians' diagnostic accuracy.

Methods

This randomized controlled study was conducted in January 2021 and included 22 physicians working at a university hospital. Participants were required to read 16 clinical vignettes in which the AI-driven medical history of real patients generated up to three differential diagnoses per case. Participants were divided into two groups: with and without an AI-driven differential-diagnosis list.

Results

There was no significant difference in diagnostic accuracy between the two groups (57.4% vs. 56.3%, respectively; p = 0.91). Vignettes that included a correct diagnosis in the AI-generated list showed the greatest positive effect on physicians' diagnostic accuracy (adjusted odds ratio 7.68; 95% CI 4.68-12.58; p < 0.001). In the group with AI-driven differential-diagnosis lists, 15.9% of diagnoses were omission errors and 14.8% were commission errors.

Conclusions

Physicians' diagnostic accuracy using AI-driven automated medical history did not differ between the groups with and without AI-driven differential-diagnosis lists.

SUBMITTER: Harada Y

PROVIDER: S-EPMC7924871 | biostudies-literature | 2021 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Efficacy of Artificial-Intelligence-Driven Differential-Diagnosis List on the Diagnostic Accuracy of Physicians: An Open-Label Randomized Controlled Study.

Harada Yukinori Y Katsukura Shinichi S Kawamura Ren R Shimizu Taro T

International journal of environmental research and public health 20210221 4

<h4>Background</h4>The efficacy of artificial intelligence (AI)-driven automated medical-history-taking systems with AI-driven differential-diagnosis lists on physicians' diagnostic accuracy was shown. However, considering the negative effects of AI-driven differential-diagnosis lists such as omission (physicians reject a correct diagnosis suggested by AI) and commission (physicians accept an incorrect diagnosis suggested by AI) errors, the efficacy of AI-driven automated medical-history-taking ...[more]

PMID: 33669930

Similar Datasets

Project description:Background Although artificial intelligence (AI) shows promise across many aspects of radiology, the use of AI to create differential diagnoses for rare and common diseases at brain MRI has not been demonstrated. Purpose To evaluate an AI system for generation of differential diagnoses at brain MRI compared with radiologists. Materials and Methods This retrospective study tested performance of an AI system for probabilistic diagnosis in patients with 19 common and rare diagnoses at brain MRI acquired between January 2008 and January 2018. The AI system combines data-driven and domain-expertise methodologies, including deep learning and Bayesian networks. First, lesions were detected by using deep learning. Then, 18 quantitative imaging features were extracted by using atlas-based coregistration and segmentation. Third, these image features were combined with five clinical features by using Bayesian inference to develop probability-ranked differential diagnoses. Quantitative feature extraction algorithms and conditional probabilities were fine-tuned on a training set of 86 patients (mean age, 49 years ± 16 [standard deviation]; 53 women). Accuracy was compared with radiology residents, general radiologists, neuroradiology fellows, and academic neuroradiologists by using accuracy of top one, top two, and top three differential diagnoses in 92 independent test set patients (mean age, 47 years ± 18; 52 women). Results For accuracy of top three differential diagnoses, the AI system (91% correct) performed similarly to academic neuroradiologists (86% correct; P = .20), and better than radiology residents (56%; P < .001), general radiologists (57%; P < .001), and neuroradiology fellows (77%; P = .003). The performance of the AI system was not affected by disease prevalence (93% accuracy for common vs 85% for rare diseases; P = .26). Radiologists were more accurate at diagnosing common versus rare diagnoses (78% vs 47% across all radiologists; P < .001). Conclusion An artificial intelligence system for brain MRI approached overall top one, top two, and top three differential diagnoses accuracy of neuroradiologists and exceeded that of less-specialized radiologists. © RSNA, 2020 Online supplemental material is available for this article. See also the editorial by Zaharchuk in this issue.

Project description:ObjectiveThis study assesses the concordance in migraine diagnosis between an online, self-administered, Computer-based, Diagnostic Engine (CDE) and semi-structured interview (SSI) by a headache specialist, both using International Classification of Headache Disorders, 3rd edition (ICHD-3) criteria.BackgroundDelay in accurate diagnosis is a major barrier to headache care. Accurate computer-based algorithms may help reduce the need for SSI-based encounters to arrive at correct ICHD-3 diagnosis.MethodsBetween March 2018 and August 2019, adult participants were recruited from three academic headache centers and the community via advertising to our cross-sectional study. Participants completed two evaluations: phone interview conducted by headache specialists using the SSI and a web-based expert questionnaire and analytics, CDE. Participants were randomly assigned to either the SSI followed by the web-based questionnaire or the web-based questionnaire followed by the SSI. Participants completed protocols a few minutes apart. The concordance in migraine/probable migraine (M/PM) diagnosis between SSI and CDE was measured using Cohen's kappa statistics. The diagnostic accuracy of CDE was assessed using the SSI as reference standard.ResultsOf the 276 participants consented, 212 completed both SSI and CDE (study completion rate = 77%; median age = 32 years [interquartile range: 28-40], female:male ratio = 3:1). Concordance in M/PM diagnosis between SSI and CDE was: κ = 0.83 (95% confidence interval [CI]: 0.75-0.91). CDE diagnostic accuracy: sensitivity = 90.1% (118/131), 95% CI: 83.6%-94.6%; specificity = 95.8% (68/71), 95% CI: 88.1%-99.1%. Positive and negative predictive values = 97.0% (95% CI: 91.3%-99.0%) and 86.6% (95% CI: 79.3%-91.5%), respectively, using identified migraine prevalence of 60%. Assuming a general migraine population prevalence of 10%, positive and negative predictive values were 70.3% (95% CI: 43.9%-87.8%) and 98.9% (95% CI: 98.1%-99.3%), respectively.ConclusionThe SSI and CDE have excellent concordance in diagnosing M/PM. Positive CDE helps rule in M/PM, through high specificity and positive likelihood ratio. A negative CDE helps rule out M/PM through high sensitivity and low negative likelihood ratio. CDE that mimics SSI logic is a valid tool for migraine diagnosis.

Project description:IntroductionStandards for Reporting of Diagnostic Accuracy Study (STARD) was developed to improve the completeness and transparency of reporting in studies investigating diagnostic test accuracy. However, its current form, STARD 2015 does not address the issues and challenges raised by artificial intelligence (AI)-centred interventions. As such, we propose an AI-specific version of the STARD checklist (STARD-AI), which focuses on the reporting of AI diagnostic test accuracy studies. This paper describes the methods that will be used to develop STARD-AI.Methods and analysisThe development of the STARD-AI checklist can be distilled into six stages. (1) A project organisation phase has been undertaken, during which a Project Team and a Steering Committee were established; (2) An item generation process has been completed following a literature review, a patient and public involvement and engagement exercise and an online scoping survey of international experts; (3) A three-round modified Delphi consensus methodology is underway, which will culminate in a teleconference consensus meeting of experts; (4) Thereafter, the Project Team will draft the initial STARD-AI checklist and the accompanying documents; (5) A piloting phase among expert users will be undertaken to identify items which are either unclear or missing. This process, consisting of surveys and semistructured interviews, will contribute towards the explanation and elaboration document and (6) On finalisation of the manuscripts, the group's efforts turn towards an organised dissemination and implementation strategy to maximise end-user adoption.Ethics and disseminationEthical approval has been granted by the Joint Research Compliance Office at Imperial College London (reference number: 19IC5679). A dissemination strategy will be aimed towards five groups of stakeholders: (1) academia, (2) policy, (3) guidelines and regulation, (4) industry and (5) public and non-specific stakeholders. We anticipate that dissemination will take place in Q3 of 2021.

Project description:BACKGROUND. Potential approaches for abbreviated knee MRI, including prospective acceleration with deep learning, have achieved limited clinical implementation. OBJECTIVE. The objective of this study was to evaluate the interreader agreement between conventional knee MRI and a 5-minute 3D quantitative double-echo steady-state (qDESS) sequence with automatic T2 mapping and deep learning super-resolutionaugmentation and to compare the diagnostic performance of the two methods regarding findings from arthroscopic surgery. METHODS. Fifty-one patients with knee pain underwent knee MRI that included an additional 3D qDESS sequence with automatic T2 mapping. Fourier interpolation was followed by prospective deep learning super resolution to enhance qDESS slice resolution twofold. A musculoskeletal radiologist and a radiology resident performed independent retrospective evaluations of articular cartilage, menisci, ligaments, bones, extensor mechanism, and synovium using conventional MRI. Following a 2-month washout period, readers reviewed qDESS images alone followed by qDESS with the automatic T2 maps. Interreader agreement between conventional MRI and qDESS was computed using percentage agreement and Cohen kappa. The sensitivity and specificity of conventional MRI, qDESS alone, and qDESS plus T2 mapping were compared with arthroscopic findings using exact McNemar tests. RESULTS. Conventional MRI and qDESS showed 92% agreement in evaluating all tissues. Kappa was 0.79 (95% CI, 0.76-0.81) across all imaging findings. In 43 patients who underwent arthroscopy, sensitivity and specificity were not significantly different (p = .23 to > .99) between conventional MRI (sensitivity, 58-93%; specificity, 27-87%) and qDESS alone (sensitivity, 54-90%; specificity, 23-91%) for cartilage, menisci, ligaments, and synovium. For grade 1 cartilage lesions, sensitivity and specificity were 33% and 56%, respectively, for conventional MRI; 23% and 53% for qDESS (p = .81); and 46% and 39% for qDESS with T2 mapping (p = .80). For grade 2A lesions, values were 27% and 53% for conventional MRI, 26% and 52% for qDESS (p = .02), and 58% and 40% for qDESS with T2 mapping (p < .001). CONCLUSION. The qDESS method prospectively augmented with deep learning showed strong interreader agreement with conventional knee MRI and near-equivalent diagnostic performance regarding arthroscopy. The ability of qDESS to automatically generate T2 maps increases sensitivity for cartilage abnormalities. CLINICAL IMPACT. Using prospective artificial intelligence to enhance qDESS image quality may facilitate an abbreviated knee MRI protocol while generating quantitative T2 maps.

Project description:BackgroundEndocytoscopy (EC) is a nuclei and micro-vessels visualization in real-time and can facilitate "optical biopsy" and "virtual histology" of colorectal lesions. This study aimed to investigate the significance of employing artificial intelligence (AI) in the field of endoscopy, specifically in diagnosing colorectal lesions. The research was conducted under the supervision of experienced professionals and trainees.MethodsEMBASE, PubMed, Cochrane Library, Web of Science, Chinese National Knowledge Infrastructure (CNKI) database, and other potential databases were surveyed for articles related to the EC with AI published before September 2023. RevMan (5.40), Stata (14.0), and R software (4.1.0) were used for statistical assessment. Studies that measured the accuracy of EC using AI for colorectal lesions were included. Two authors independently assessed the selected studies and their extracted data. This included information such as the country, literature, total study population, study design, characteristics of the fundamental study and control groups, sensitivity, number of samples, assay methodology, specificity, true positives or negatives, and false positives or negatives. The diagnostic accuracy of EC by AI was determined by a bivariate random-effects model, avoiding a high heterogeneity effect. The ANOVA model was employed to determine the more effective approach.ResultsA total of 223 studies were reviewed; 8 articles were selected that included 2984 patients (4241 lesions) for systematic review and meta-analysis. AI assessed 4069 lesions; experts diagnosed 3165 and 5014 by trainees. AI demonstrated high accuracy, sensitivity, and specificity levels in detecting colorectal lesions, with values of 0.93 (95% CI: 0.90, 0.95) and 0.94 (95% CI: 0.73, 0.99). Expert diagnosis was 0.90 (95% CI: 0.85, 0.94), 0.87 (95% CI: 0.78, 0.93), and trainee diagnosis was 0.74 (95% CI: 0.67, 0.79), 0.72 (95% CI: 0.62, 0.80). With the EC by AI, the AUC from SROC was 0.95 (95% CI: 0.93, 0.97), therefore classified as excellent category, expert showed 0.95 (95% CI: 0.93, 0.97), and the trainee had 0.79 (95% CI: 0.75, 0.82). The superior index from the ANOVA model was 4.00 (1.15,5.00), 2.00 (1.15,5.00), and 0.20 (0.20,0.20), respectively. The examiners conducted meta-regression and subgroup analyses to evaluate the presence of heterogeneity. The findings of these investigations suggest that the utilization of NBI technology was correlated with variability in sensitivity and specificity. There was a lack of solid evidence indicating the presence of publishing bias.ConclusionsThe present findings indicate that using AI in EC can potentially enhance the efficiency of diagnosing colorectal abnormalities. As a valuable instrument, it can enhance prognostic outcomes in ordinary EC procedures, exhibiting superior diagnostic accuracy compared to trainee-level endoscopists and demonstrating comparability to expert endoscopists. The research is subject to certain constraints, namely a limited number of clinical investigations and variations in the methodologies used for identification. Consequently, it is imperative to conduct comprehensive and extensive research to enhance the precision of diagnostic procedures.

Project description:Prostate cancer is a leading cause of morbidity and mortality for adult males in the US. The diagnosis of prostate carcinoma is usually made on prostate core needle biopsies obtained through a transrectal approach. These biopsies may account for a significant portion of the pathologists' workload, yet variability in the experience and expertise, as well as fatigue of the pathologist may adversely affect the reliability of cancer detection. Machine-learning algorithms are increasingly being developed as tools to aid and improve diagnostic accuracy in anatomic pathology. The Paige Prostate AI-based digital diagnostic is one such tool trained on the digital slide archive of New York's Memorial Sloan Kettering Cancer Center (MSKCC) that categorizes a prostate biopsy whole-slide image as either "Suspicious" or "Not Suspicious" for prostatic adenocarcinoma. To evaluate the performance of this program on prostate biopsies secured, processed, and independently diagnosed at an unrelated institution, we used Paige Prostate to review 1876 prostate core biopsy whole-slide images (WSIs) from our practice at Yale Medicine. Paige Prostate categorizations were compared to the pathology diagnosis originally rendered on the glass slides for each core biopsy. Discrepancies between the rendered diagnosis and categorization by Paige Prostate were each manually reviewed by pathologists with specialized genitourinary pathology expertise. Paige Prostate showed a sensitivity of 97.7% and positive predictive value of 97.9%, and a specificity of 99.3% and negative predictive value of 99.2% in identifying core biopsies with cancer in a data set derived from an independent institution. Areas for improvement were identified in Paige Prostate's handling of poor quality scans. Overall, these results demonstrate the feasibility of porting a machine-learning algorithm to an institution remote from its training set, and highlight the potential of such algorithms as a powerful workflow tool for the evaluation of prostate core biopsies in surgical pathology practices.

Dataset Information

Efficacy of Artificial-Intelligence-Driven Differential-Diagnosis List on the Diagnostic Accuracy of Physicians: An Open-Label Randomized Controlled Study.

Background

Objective

Methods

Results

Conclusions

Publications

Efficacy of Artificial-Intelligence-Driven Differential-Diagnosis List on the Diagnostic Accuracy of Physicians: An Open-Label Randomized Controlled Study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets