Dataset Information

COVID-19 pneumonia on chest X-rays: Performance of a deep learning-based computer-aided detection system.

ABSTRACT: Chest X-rays (CXRs) can help triage for Coronavirus disease (COVID-19) patients in resource-constrained environments, and a computer-aided detection system (CAD) that can identify pneumonia on CXR may help the triage of patients in those environment where expert radiologists are not available. However, the performance of existing CAD for identifying COVID-19 and associated pneumonia on CXRs has been scarcely investigated. In this study, CXRs of patients with and without COVID-19 confirmed by reverse transcriptase polymerase chain reaction (RT-PCR) were retrospectively collected from four and one institution, respectively, and a commercialized, regulatory-approved CAD that can identify various abnormalities including pneumonia was used to analyze each CXR. Performance of the CAD was evaluated using area under the receiver operating characteristic curves (AUCs), with reference standards of the RT-PCR results and the presence of findings of pneumonia on chest CTs obtained within 24 hours from the CXR. For comparison, 5 thoracic radiologists and 5 non-radiologist physicians independently interpreted the CXRs. Afterward, they re-interpreted the CXRs with corresponding CAD results. The performance of CAD (AUCs, 0.714 and 0.790 against RT-PCR and chest CT, respectively hereinafter) were similar with those of thoracic radiologists (AUCs, 0.701 and 0.784), and higher than those of non-radiologist physicians (AUCs, 0.584 and 0.650). Non-radiologist physicians showed significantly improved performance when assisted with the CAD (AUCs, 0.584 to 0.664 and 0.650 to 0.738). In addition, inter-reader agreement among physicians was also improved in the CAD-assisted interpretation (Fleiss' kappa coefficient, 0.209 to 0.322). In conclusion, radiologist-level performance of the CAD in identifying COVID-19 and associated pneumonia on CXR and enhanced performance of non-radiologist physicians with the CAD assistance suggest that the CAD can support physicians in interpreting CXRs and helping image-based triage of COVID-19 patients in resource-constrained environment.

SUBMITTER: Hwang EJ

PROVIDER: S-EPMC8184006 | biostudies-literature | 2021

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

COVID-19 pneumonia on chest X-rays: Performance of a deep learning-based computer-aided detection system.

Hwang Eui Jin EJ Kim Ki Beom KB Kim Jin Young JY Lim Jae-Kwang JK Nam Ju Gang JG Choi Hyewon H Kim Hyungjin H Yoon Soon Ho SH Goo Jin Mo JM Park Chang Min CM

PloS one 20210607 6

Chest X-rays (CXRs) can help triage for Coronavirus disease (COVID-19) patients in resource-constrained environments, and a computer-aided detection system (CAD) that can identify pneumonia on CXR may help the triage of patients in those environment where expert radiologists are not available. However, the performance of existing CAD for identifying COVID-19 and associated pneumonia on CXRs has been scarcely investigated. In this study, CXRs of patients with and without COVID-19 confirmed by rev ...[more]

PMID: 34097708

Similar Datasets

Project description:BackgroundDiagnosis of pneumonia is critical in managing patients with febrile neutropenia (FN), however, chest X-ray (CXR) has limited performance in the detection of pneumonia. We aimed to evaluate the performance of a deep learning-based computer-aided detection (CAD) system in pneumonia detection in the CXRs of consecutive FN patients and investigated whether CAD could improve radiologists' diagnostic performance when used as a second reader.MethodsCXRs of patients with FN (a body temperature ≥ 38.3 °C, or a sustained body temperature ≥ 38.0 °C for an hour; absolute neutrophil count < 500/mm3) obtained between January and December 2017 were consecutively included, from a single tertiary referral hospital. Reference standards for the diagnosis of pneumonia were defined by consensus of two thoracic radiologists after reviewing medical records and CXRs. A commercialized, deep learning-based CAD system was retrospectively applied to detect pulmonary infiltrates on CXRs. For comparing performance, five radiologists independently interpreted CXRs initially without the CAD results (radiologist-alone interpretation), followed by the interpretation with CAD. The sensitivities and specificities for detection of pneumonia were compared between radiologist-alone interpretation and interpretation with CAD. The standalone performance of the CAD was also evaluated, using area under the receiver operating characteristic curve (AUC), sensitivity, and specificity. Moreover, sensitivity and specificity of standalone CAD were compared with those of radiologist-alone interpretation.ResultsAmong 525 CXRs from 413 patients (52.3% men; median age 59 years), pneumonia was diagnosed in 128 (24.4%) CXRs. In the interpretation with CAD, average sensitivity of radiologists was significantly improved (75.4% to 79.4%, P = 0.003) while their specificity remained similar (75.4% to 76.8%, P = 0.101), compared to radiologist-alone interpretation. The CAD exhibited AUC, sensitivity, and specificity of 0.895, 88.3%, and 68.3%, respectively. The standalone CAD exhibited higher sensitivity (86.6% vs. 75.2%, P < 0.001) and lower specificity (64.8% vs. 75.4%, P < 0.001) compared to radiologist-alone interpretation.ConclusionsIn patients with FN, the deep learning-based CAD system exhibited radiologist-level performance in detecting pneumonia on CXRs and enhanced radiologists' performance.

Project description:ObjectivesTo investigate the effectiveness of BMAX, a deep learning-based computer-aided detection system for detecting fibrosing interstitial lung disease (ILD) on chest radiographs among non-expert and expert physicians in the real-world clinical setting.DesignRetrospective, observational study.SettingThis study used chest radiograph images consecutively taken in three medical facilities with various degrees of referral. Three expert ILD physicians interpreted each image and determined whether it was a fibrosing ILD-suspected image (fibrosing ILD positive) or not (fibrosing ILD negative). Interpreters, including non-experts and experts, classified each of 120 images extracted from the pooled data for the reading test into positive or negative for fibrosing ILD without and with the assistance of BMAX.ParticipantsChest radiographs of patients aged 20 years or older with two or more visits that were taken during consecutive periods were accumulated. 1251 chest radiograph images were collected, from which 120 images (24 positive and 96 negative images) were randomly extracted for the reading test. The interpreters for the reading test were 20 non-expert physicians and 5 expert physicians (3 pulmonologists and 2 radiologists).Primary and secondary outcome measuresThe primary outcome was the comparison of area under the receiver-operating characteristic curve (ROC-AUC) for identifying fibrosing ILD-positive images by non-experts without versus with BMAX. The secondary outcome was the comparison of sensitivity, specificity and accuracy by non-experts and experts without versus with BMAX.ResultsThe mean ROC-AUC of non-expert interpreters was 0.795 (95% CI; 0.765 to 0.825) without BMAX and 0.825 (95% CI; 0.799 to 0.850) with BMAX (p=0.005). After using BMAX, sensitivity was improved from 0.744 (95% CI; 0.697 to 0.791) to 0.802 (95% CI; 0.754 to 0.850) among non-experts (p=0.003), but not among experts (p=0.285). Specificity and accuracy were not changed after using BMAX among either non-expert or expert interpreters.ConclusionBMAX was useful for detecting fibrosing ILD-suspected chest radiographs for non-expert physicians.Trial registration numberjRCT1032220090.

Project description:Computer-aided detection (CAD) was recently recommended by the WHO for TB screening and triage based on several evaluations, but unlike traditional diagnostic tests, software versions are updated frequently and require constant evaluation. Since then, newer versions of two of the evaluated products have already been released. We used a case control sample of 12,890 chest X-rays to compare performance and model the programmatic effect of upgrading to newer versions of CAD4TB and qXR. We compared the area under the receiver operating characteristic curve (AUC), overall, and with data stratified by age, TB history, gender, and patient source. All versions were compared against radiologist readings and WHO’s Target Product Profile (TPP) for a TB triage test. Both newer versions significantly outperformed their predecessors in terms of AUC: CAD4TB version 6 (0.823 [0.816–0.830]), version 7 (0.903 [0.897–0.908]) and qXR version 2 (0.872 [0.866–0.878]), version 3 (0.906 [0.901–0.911]). Newer versions met WHO TPP values, older versions did not. All products equalled or surpassed the human radiologist performance with improvements in triage ability in newer versions. Humans and CAD performed worse in older age groups and among those with TB history. New versions of CAD outperform their predecessors. Prior to implementation CAD should be evaluated using local data because underlying neural networks can differ significantly. An independent rapid evaluation centre is necessitated to provide implementers with performance data on new versions of CAD products as they are developed. Author summary The World Health Organization recommended the use of artificial intelligence (AI)-powered computer-aided detection (CAD) for TB screening and triage in 2021. One year on, we comprehensively compare the performance of the newest versions of two CAD (CAD4TB and qXR) to their WHO-evaluated predecessors. We found that both newer versions significantly improved upon their predecessor’s ability to detect TB, performing better than the human readers. We also showed that the AI underlying new software versions can differ remarkably from the old and resemble an entirely new product altogether. We further demonstrate that, unlike laboratory diagnostic tools, CAD software updates could significantly impact the selection of appropriate threshold scores, the number of people with TB detected and cost-effectiveness. With newer CAD versions being rolled out almost annually, our results therefore underscore the need for rapid evidence generation to evaluate newer CAD versions in the fast-growing medical AI industry.

Project description:BackgroundNowadays doctors and radiologists are overwhelmed with a huge amount of work. This led to the effort to design different Computer-Aided Diagnosis systems (CAD system), with the aim of accomplishing a faster and more accurate diagnosis. The current development of deep learning is a big opportunity for the development of new CADs. In this paper, we propose a novel architecture for a convolutional neural network (CNN) ensemble for classifying chest X-ray (CRX) images into four classes: viral Pneumonia, Tuberculosis, COVID-19, and Healthy. Although Computed tomography (CT) is the best way to detect and diagnoses pulmonary issues, CT is more expensive than CRX. Furthermore, CRX is commonly the first step in the diagnosis, so it's very important to be accurate in the early stages of diagnosis and treatment.ResultsWe applied the transfer learning technique and data augmentation to all CNNs for obtaining better performance. We have designed and evaluated two different CNN-ensembles: Stacking and Voting. This system is ready to be applied in a CAD system to automated diagnosis such a second or previous opinion before the doctors or radiology's. Our results show a great improvement, 99% accuracy of the Stacking Ensemble and 98% of accuracy for the the Voting Ensemble.ConclusionsTo minimize missclassifications, we included six different base CNN models in our architecture (VGG16, VGG19, InceptionV3, ResNet101V2, DenseNet121 and CheXnet) and it could be extended to any number as well as we expect extend the number of diseases to detected. The proposed method has been validated using a large dataset created by mixing several public datasets with different image sizes and quality. As we demonstrate in the evaluation carried out, we reach better results and generalization compared with previous works. In addition, we make a first approach to explainable deep learning with the objective of providing professionals more information that may be valuable when evaluating CRXs.

Project description:The COVID-19 pandemic has caused a worldwide catastrophe and widespread devastation that reeled almost all countries. The pandemic has mounted pressure on the existing healthcare system and caused panic and desperation. The gold testing standard for COVID-19 detection, reverse transcription-polymerase chain reaction (RT-PCR), has shown its limitations with 70% accuracy, contributing to the incorrect diagnosis that exaggerated the complexities and increased the fatalities. The new variations further pose unseen challenges in terms of their diagnosis and subsequent treatment. The COVID-19 virus heavily impacts the lungs and fills the air sacs with fluid causing pneumonia. Thus, chest X-ray inspection is a viable option if the inspection detects COVID-19-induced pneumonia, hence confirming the exposure of COVID-19. Artificial intelligence and machine learning techniques are capable of examining chest X-rays in order to detect patterns that can confirm the presence of COVID-19-induced pneumonia. This research used CNN and deep learning techniques to detect COVID-19-induced pneumonia from chest X-rays. Transfer learning with fine-tuning ensures that the proposed work successfully classifies COVID-19-induced pneumonia, regular pneumonia, and normal conditions. Xception, Visual Geometry Group 16, and Visual Geometry Group 19 are used to realize transfer learning. The experimental results were promising in terms of precision, recall, F1 score, specificity, false omission rate, false negative rate, false positive rate, and false discovery rate with a COVID-19-induced pneumonia detection accuracy of 98%. Experimental results also revealed that the proposed work has not only correctly identified COVID-19 exposure but also made a distinction between COVID-19-induced pneumonia and regular pneumonia, as the latter is a very common disease, while COVID-19 is more lethal. These results mitigated the concern and overlap in the diagnosis of COVID-19-induced pneumonia and regular pneumonia. With further integrations, it can be employed as a potential standard model in differentiating the various lung-related infections, including COVID-19.

Project description:BackgroundColposcopy is widely used to detect cervical cancer, but developing countries lack the experienced colposcopists necessary for accurate diagnosis. Artificial intelligence (AI) is being widely used in computer-aided diagnosis (CAD) systems. In this study, we developed and validated a CAD model based on deep learning to classify cervical lesions on colposcopy images.MethodsPatient data, including clinical information, colposcopy images, and pathological results, were collected from Qilu Hospital. The study included 15,276 images from 7,530 patients. We performed two tasks in this study: normal cervix (NC) vs. low grade squamous intraepithelial lesion or worse (LSIL+) and high-grade squamous intraepithelial lesion (HSIL)- vs. HSIL+. The residual neural network (ResNet) probability was calculated for each patient to reflect the probability of lesions through a ResNet model. Next, a combination model was constructed by incorporating the ResNet probability and clinical features. We divided the dataset into a training set, validation set, and testing set at a ratio of 7:1:2. Finally, we randomly selected 300 patients from the testing set and compared the results with the diagnosis of a senior colposcopist and a junior colposcopist.ResultsThe model that combines ResNet and clinical features performs better than ResNet alone. In the classification of NC and LSIL+, the area under the receiver operating characteristic curve (AUC), accuracy, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) were 0.953, 0.886, 0.932, 0.846, 0.838, and 0.936, respectively. In the classification of HSIL- and HSIL+, the AUC, accuracy, sensitivity, specificity, PPV, and NPV were 0.900, 0.807, 0.823, 0.800, 0.618, and 0.920, respectively. In the two classification tasks, the diagnostic performance of the model was determined to be comparable to that of the senior colposcopist and exhibited a stronger diagnostic performance than the junior colposcopist.ConclusionsThe CAD system for cervical lesion diagnosis based on deep learning performs well in the classification of cervical lesions and can provide an objective diagnostic basis for colposcopists.

Project description:BackgroundTuberculosis (TB) was the leading infectious cause of mortality globally prior to COVID-19 and chest radiography has an important role in the detection, and subsequent diagnosis, of patients with this disease. The conventional experts reading has substantial within- and between-observer variability, indicating poor reliability of human readers. Substantial efforts have been made in utilizing various artificial intelligence-based algorithms to address the limitations of human reading of chest radiographs for diagnosing TB.ObjectiveThis systematic literature review (SLR) aims to assess the performance of machine learning (ML) and deep learning (DL) in the detection of TB using chest radiography (chest x-ray [CXR]).MethodsIn conducting and reporting the SLR, we followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. A total of 309 records were identified from Scopus, PubMed, and IEEE (Institute of Electrical and Electronics Engineers) databases. We independently screened, reviewed, and assessed all available records and included 47 studies that met the inclusion criteria in this SLR. We also performed the risk of bias assessment using Quality Assessment of Diagnostic Accuracy Studies version 2 (QUADAS-2) and meta-analysis of 10 included studies that provided confusion matrix results.ResultsVarious CXR data sets have been used in the included studies, with 2 of the most popular ones being Montgomery County (n=29) and Shenzhen (n=36) data sets. DL (n=34) was more commonly used than ML (n=7) in the included studies. Most studies used human radiologist's report as the reference standard. Support vector machine (n=5), k-nearest neighbors (n=3), and random forest (n=2) were the most popular ML approaches. Meanwhile, convolutional neural networks were the most commonly used DL techniques, with the 4 most popular applications being ResNet-50 (n=11), VGG-16 (n=8), VGG-19 (n=7), and AlexNet (n=6). Four performance metrics were popularly used, namely, accuracy (n=35), area under the curve (AUC; n=34), sensitivity (n=27), and specificity (n=23). In terms of the performance results, ML showed higher accuracy (mean ~93.71%) and sensitivity (mean ~92.55%), while on average DL models achieved better AUC (mean ~92.12%) and specificity (mean ~91.54%). Based on data from 10 studies that provided confusion matrix results, we estimated the pooled sensitivity and specificity of ML and DL methods to be 0.9857 (95% CI 0.9477-1.00) and 0.9805 (95% CI 0.9255-1.00), respectively. From the risk of bias assessment, 17 studies were regarded as having unclear risks for the reference standard aspect and 6 studies were regarded as having unclear risks for the flow and timing aspect. Only 2 included studies had built applications based on the proposed solutions.ConclusionsFindings from this SLR confirm the high potential of both ML and DL for TB detection using CXR. Future studies need to pay a close attention on 2 aspects of risk of bias, namely, the reference standard and the flow and timing aspects.Trial registrationPROSPERO CRD42021277155; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=277155.

Project description:BackgroundAirspace disease as seen on chest X-rays is an important point in triage for patients initially presenting to the emergency department with suspected COVID-19 infection. The purpose of this study is to evaluate a previously trained interpretable deep learning algorithm for the diagnosis and prognosis of COVID-19 pneumonia from chest X-rays obtained in the ED.MethodsThis retrospective study included 2456 (50% RT-PCR positive for COVID-19) adult patients who received both a chest X-ray and SARS-CoV-2 RT-PCR test from January 2020 to March of 2021 in the emergency department at a single U.S.InstitutionA total of 2000 patients were included as an additional training cohort and 456 patients in the randomized internal holdout testing cohort for a previously trained Siemens AI-Radiology Companion deep learning convolutional neural network algorithm. Three cardiothoracic fellowship-trained radiologists systematically evaluated each chest X-ray and generated an airspace disease area-based severity score which was compared against the same score produced by artificial intelligence. The interobserver agreement, diagnostic accuracy, and predictive capability for inpatient outcomes were assessed. Principal statistical tests used in this study include both univariate and multivariate logistic regression.ResultsOverall ICC was 0.820 (95% CI 0.790-0.840). The diagnostic AUC for SARS-CoV-2 RT-PCR positivity was 0.890 (95% CI 0.861-0.920) for the neural network and 0.936 (95% CI 0.918-0.960) for radiologists. Airspace opacities score by AI alone predicted ICU admission (AUC = 0.870) and mortality (0.829) in all patients. Addition of age and BMI into a multivariate log model improved mortality prediction (AUC = 0.906).ConclusionThe deep learning algorithm provides an accurate and interpretable assessment of the disease burden in COVID-19 pneumonia on chest radiographs. The reported severity scores correlate with expert assessment and accurately predicts important clinical outcomes. The algorithm contributes additional prognostic information not currently incorporated into patient management.

Dataset Information

COVID-19 pneumonia on chest X-rays: Performance of a deep learning-based computer-aided detection system.

Publications

COVID-19 pneumonia on chest X-rays: Performance of a deep learning-based computer-aided detection system.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets