Dataset Information

A practical artificial intelligence system to diagnose COVID-19 using computed tomography: A multinational external validation study.

ABSTRACT: Computed tomography has gained an important role in the early diagnosis of COVID-19 pneumonia. However, the ever-increasing number of patients has overwhelmed radiology departments and has caused a reduction in quality of services. Artificial intelligence (AI) systems are the remedy to the current situation. However, the lack of application in real-world conditions has limited their consideration in clinical settings. This study validated a clinical AI system, COVIDiag, to aid radiologists in accurate and rapid evaluation of COVID-19 cases. 50 COVID-19 and 50 non-COVID-19 pneumonia cases were included from each of five centers: Argentina, Turkey, Iran, Netherlands, and Italy. The Dutch database included only 50 COVID-19 cases. The performance parameters namely sensitivity, specificity, accuracy, and area under the ROC curve (AUC) were computed for each database using COVIDiag model. The most common pattern of involvement among COVID-19 cases in all databases were bilateral involvement of upper and lower lobes with ground-glass opacities. The best sensitivity of 92.0% was recorded for the Italian database. The system achieved an AUC of 0.983, 0.914, 0.910, and 0.882 for Argentina, Turkey, Iran, and Italy, respectively. The model obtained a sensitivity of 86.0% for the Dutch database. COVIDiag model could diagnose COVID-19 pneumonia in all of cohorts with AUC of 0.921 (sensitivity, specificity, and accuracy of 88.8%, 87.0%, and 88.0%, respectively). Our study confirmed the accuracy of our proposed AI model (COVIDiag) in the diagnosis of COVID-19 cases. Furthermore, the system demonstrated consistent optimal diagnostic performance on multinational databases, which is critical to determine the generalizability and objectivity of the proposed COVIDiag model. Our results are significant as they provide real-world evidence regarding the applicability of AI systems in clinical medicine.

SUBMITTER: Ardakani AA

PROVIDER: S-EPMC8457921 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundThe use of an artificial intelligence (AI)-based diagnostic system can significantly aid in analyzing the histogram of pulmonary nodules. The aim of our study was to evaluate the value of computed tomography (CT) histogram indicators analyzed by AI in predicting the tumor invasiveness of ground-glass nodules (GGNs) and to determine the added value of contrast-enhanced CT (CECT) compared with nonenhanced CT (NECT) in this prediction.MethodsThis study enrolled patients with persistent GGNs who underwent preoperative NECT and CECT scanning. AI-based histogram analysis was performed for pathologically confirmed GGNs, which was followed by screening invasiveness-related factors via univariable analysis. Multivariable logistic models were developed based on candidate CT histogram indicators measured on either NECT or CECT. Receiver operating characteristic (ROC) curve and precision-recall (PR) curve were used to evaluate the models' performance.ResultsA total of 116 patients comprising 121 GGNs were included and divided into the precancerous lesion and adenocarcinoma groups based on invasiveness. In the AI-based histogram analysis, the mean CT value [NECT: odds ratio (OR) =1.009; 95% confidence interval (CI): 1.004-1.013; P<0.001] and solid component volume (NECT: OR =1.005; 95% CI: 1.000-1.010; P=0.032) were associated with the adenocarcinoma and used for multivariable logistic modeling. The area under ROC curve (AUC) and PR curve (AUPR) were not significantly different between the NECT model (AUC =0.765, 95% CI: 0.679-0.837; AUPR =0.907, 95% CI: 0.825-0.953) and the optimal CECT model (delayed phase: AUC =0.772, 95% CI: 0.687-0.843; AUPR =0.895, 95% CI: 0.812-0.944). No significantly different metrics were observed between the NECT and CECT models (precision: 0.707 vs. 0.742; P=0.616).ConclusionsThe AI diagnostic system can help in the diagnosis of GGNs. The system displayed decent performance in GGN detection and alert to malignancy. Mean CT value and solid component volume were independent predictors of tumor invasiveness. CECT provided no additional improvement in diagnostic performance as compared with NECT.

Project description:ObjectivesCoronary artery calcium (CAC) scores derived from computed tomography (CT) scans are used for cardiovascular risk stratification. Artificial intelligence (AI) can assist in CAC quantification and potentially reduce the time required for human analysis. This study aimed to develop and evaluate a fully automated model that identifies and quantifies CAC.MethodsFully convolutional neural networks for automated CAC scoring were developed and trained on 2439 cardiac CT scans and validated using 771 scans. The model was tested on an independent set of 1849 cardiac CT scans. Agatston CAC scores were further categorised into five risk categories (0, 1-10, 11-100, 101-400, and > 400). Automated scores were compared to the manual reference standard (level 3 expert readers).ResultsOf 1849 scans used for model testing (mean age 55.7 ± 10.5 years, 49% males), the automated model detected the presence of CAC in 867 (47%) scans compared with 815 (44%) by human readers (p = 0.09). CAC scores from the model correlated very strongly with the manual score (Spearman's r = 0.90, 95% confidence interval [CI] 0.89-0.91, p < 0.001 and intraclass correlation coefficient = 0.98, 95% CI 0.98-0.99, p < 0.001). The model classified 1646 (89%) into the same risk category as human observers. The Bland-Altman analysis demonstrated little difference (1.69, 95% limits of agreement: -41.22, 44.60) and there was almost excellent agreement (Cohen's κ = 0.90, 95% CI 0.88-0.91, p < 0.001). Model analysis time was 13.1 ± 3.2 s/scan.ConclusionsThis artificial intelligence-based fully automated CAC scoring model shows high accuracy and low analysis times. Its potential to optimise clinical workflow efficiency and patient outcomes requires evaluation.Key points• Coronary artery calcium (CAC) scores are traditionally assessed using cardiac computed tomography and require manual input by human operators to identify calcified lesions. • A novel artificial intelligence (AI)-based model for fully automated CAC scoring was developed and tested on an independent dataset of computed tomography scans, showing very high levels of correlation and agreement with manual measurements as a reference standard. • AI has the potential to assist in the identification and quantification of CAC, thereby reducing the time required for human analysis.

Project description:ImportanceWith a shortfall in fellowship-trained breast radiologists, mammography screening programs are looking toward artificial intelligence (AI) to increase efficiency and diagnostic accuracy. External validation studies provide an initial assessment of how promising AI algorithms perform in different practice settings.ObjectiveTo externally validate an ensemble deep-learning model using data from a high-volume, distributed screening program of an academic health system with a diverse patient population.Design, setting, and participantsIn this diagnostic study, an ensemble learning method, which reweights outputs of the 11 highest-performing individual AI models from the Digital Mammography Dialogue on Reverse Engineering Assessment and Methods (DREAM) Mammography Challenge, was used to predict the cancer status of an individual using a standard set of screening mammography images. This study was conducted using retrospective patient data collected between 2010 and 2020 from women aged 40 years and older who underwent a routine breast screening examination and participated in the Athena Breast Health Network at the University of California, Los Angeles (UCLA).Main outcomes and measuresPerformance of the challenge ensemble method (CEM) and the CEM combined with radiologist assessment (CEM+R) were compared with diagnosed ductal carcinoma in situ and invasive cancers within a year of the screening examination using performance metrics, such as sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC).ResultsEvaluated on 37 317 examinations from 26 817 women (mean [SD] age, 58.4 [11.5] years), individual model AUROC estimates ranged from 0.77 (95% CI, 0.75-0.79) to 0.83 (95% CI, 0.81-0.85). The CEM model achieved an AUROC of 0.85 (95% CI, 0.84-0.87) in the UCLA cohort, lower than the performance achieved in the Kaiser Permanente Washington (AUROC, 0.90) and Karolinska Institute (AUROC, 0.92) cohorts. The CEM+R model achieved a sensitivity (0.813 [95% CI, 0.781-0.843] vs 0.826 [95% CI, 0.795-0.856]; P = .20) and specificity (0.925 [95% CI, 0.916-0.934] vs 0.930 [95% CI, 0.929-0.932]; P = .18) similar to the radiologist performance. The CEM+R model had significantly lower sensitivity (0.596 [95% CI, 0.466-0.717] vs 0.850 [95% CI, 0.766-0.923]; P < .001) and specificity (0.803 [95% CI, 0.734-0.861] vs 0.945 [95% CI, 0.936-0.954]; P < .001) than the radiologist in women with a prior history of breast cancer and Hispanic women (0.894 [95% CI, 0.873-0.910] vs 0.926 [95% CI, 0.919-0.933]; P = .004).Conclusions and relevanceThis study found that the high performance of an ensemble deep-learning model for automated screening mammography interpretation did not generalize to a more diverse screening cohort, suggesting that the model experienced underspecification. This study suggests the need for model transparency and fine-tuning of AI models for specific target populations prior to their clinical adoption.

Dataset Information

A practical artificial intelligence system to diagnose COVID-19 using computed tomography: A multinational external validation study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets