Dataset Information

Quantifying the incremental value of deep learning: Application to lung nodule detection.

ABSTRACT: We present a case study for implementing a machine learning algorithm with an incremental value framework in the domain of lung cancer research. Machine learning methods have often been shown to be competitive with prediction models in some domains; however, implementation of these methods is in early development. Often these methods are only directly compared to existing methods; here we present a framework for assessing the value of a machine learning model by assessing the incremental value. We developed a machine learning model to identify and classify lung nodules and assessed the incremental value added to existing risk prediction models. Multiple external datasets were used for validation. We found that our image model, trained on a dataset from The Cancer Imaging Archive (TCIA), improves upon existing models that are restricted to patient characteristics, but it was inconclusive about whether it improves on models that consider nodule features. Another interesting finding is the variable performance on different datasets, suggesting population generalization with machine learning models may be more challenging than is often considered.

SUBMITTER: Warsavage T

PROVIDER: S-EPMC7156089 | biostudies-literature | 2020

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Quantifying the incremental value of deep learning: Application to lung nodule detection.

Warsavage Theodore T Xing Fuyong F Barón Anna E AE Feser William J WJ Hirsch Erin E Miller York E YE Malkoski Stephen S Wolf Holly J HJ Wilson David O DO Ghosh Debashis D

PloS one 20200414 4

We present a case study for implementing a machine learning algorithm with an incremental value framework in the domain of lung cancer research. Machine learning methods have often been shown to be competitive with prediction models in some domains; however, implementation of these methods is in early development. Often these methods are only directly compared to existing methods; here we present a framework for assessing the value of a machine learning model by assessing the incremental value. ...[more]

PMID: 32287288

Similar Datasets

Project description:ObjectiveIn this study, we evaluated a commercially available computer assisted diagnosis system (CAD). The deep learning algorithm of the CAD was trained with a lung cancer screening cohort and developed for detection, classification, quantification, and growth of actionable pulmonary nodules on chest CT scans. Here, we evaluated the CAD in a retrospective cohort of a routine clinical population.Materials and methodsIn total, a number of 337 scans of 314 different subjects with reported nodules of 3-30 mm in size were included into the evaluation. Two independent thoracic radiologists alternately reviewed scans with or without CAD assistance to detect, classify, segment, and register pulmonary nodules. A third, more experienced, radiologist served as an adjudicator. In addition, the cohort was analyzed by the CAD alone. The study cohort was divided into five different groups: 1) 178 CT studies without reported pulmonary nodules, 2) 95 studies with 1-10 pulmonary nodules, 23 studies from the same patients with 3) baseline and 4) follow-up studies, and 5) 18 CT studies with subsolid nodules. A reference standard for nodules was based on majority consensus with the third thoracic radiologist as required. Sensitivity, false positive (FP) rate and Dice inter-reader coefficient were calculated.ResultsAfter analysis of 470 pulmonary nodules, the sensitivity readings for radiologists without CAD and radiologist with CAD, were 71.9% (95% CI: 66.0%, 77.0%) and 80.3% (95% CI: 75.2%, 85.0%) (p < 0.01), with average FP rate of 0.11 and 0.16 per CT scan, respectively. Accuracy and kappa of CAD for classifying solid vs sub-solid nodules was 94.2% and 0.77, respectively. Average inter-reader Dice coefficient for nodule segmentation was 0.83 (95% CI: 0.39, 0.96) and 0.86 (95% CI: 0.51, 0.95) for CAD versus readers. Mean growth percentage discrepancy of readers and CAD alone was 1.30 (95% CI: 1.02, 2.21) and 1.35 (95% CI: 1.01, 4.99), respectively.ConclusionThe applied CAD significantly increased radiologist's detection of actionable nodules yet also minimally increasing the false positive rate. The CAD can automatically classify and quantify nodules and calculate nodule growth rate in a cohort of a routine clinical population. Results suggest this Deep Learning software has the potential to assist chest radiologists in the tasks of pulmonary nodule detection and management within their routine clinical practice.

Project description:BackgroundThe COVID-19 pandemic has exposed the vulnerability of healthcare services worldwide, especially in underdeveloped countries. There is a clear need to develop novel computer-assisted diagnosis tools to provide rapid and cost-effective screening in places where massive traditional testing is not feasible. Lung ultrasound is a portable, easy to disinfect, low cost and non-invasive tool that can be used to identify lung diseases. Computer-assisted analysis of lung ultrasound imagery is a relatively recent approach that has shown great potential for diagnosing pulmonary conditions, being a viable alternative for screening and diagnosing COVID-19.ObjectiveTo evaluate and compare the performance of deep-learning techniques for detecting COVID-19 infections from lung ultrasound imagery.MethodsWe adapted different pre-trained deep learning architectures, including VGG19, InceptionV3, Xception, and ResNet50. We used the publicly available POCUS dataset comprising 3326 lung ultrasound frames of healthy, COVID-19, and pneumonia patients for training and fine-tuning. We conducted two experiments considering three classes (COVID-19, pneumonia, and healthy) and two classes (COVID-19 versus pneumonia and COVID-19 versus non-COVID-19) of predictive models. The obtained results were also compared with the POCOVID-net model. For performance evaluation, we calculated per-class classification metrics (Precision, Recall, and F1-score) and overall metrics (Accuracy, Balanced Accuracy, and Area Under the Receiver Operating Characteristic Curve). Lastly, we performed a statistical analysis of performance results using ANOVA and Friedman tests followed by post-hoc analysis using the Wilcoxon signed-rank test with the Holm's step-down correction.ResultsInceptionV3 network achieved the best average accuracy (89.1%), balanced accuracy (89.3%), and area under the receiver operating curve (97.1%) for COVID-19 detection from bacterial pneumonia and healthy lung ultrasound data. The ANOVA and Friedman tests found statistically significant performance differences between models for accuracy, balanced accuracy and area under the receiver operating curve. Post-hoc analysis showed statistically significant differences between the performance obtained with the InceptionV3-based model and POCOVID-net, VGG19-, and ResNet50-based models. No statistically significant differences were found in the performance obtained with InceptionV3- and Xception-based models.ConclusionsDeep learning techniques for computer-assisted analysis of lung ultrasound imagery provide a promising avenue for COVID-19 screening and diagnosis. Particularly, we found that the InceptionV3 network provides the most promising predictive results from all AI-based techniques evaluated in this work. InceptionV3- and Xception-based models can be used to further develop a viable computer-assisted screening tool for COVID-19 based on ultrasound imagery.

Project description:ObjectiveThe purpose of this study was to explore the auxiliary diagnostic value of volumetric CT value in quantifying the activity of a pulmonary tuberculoma.MethodsChest CT image data of 112 patients with pulmonary tuberculomas who were diagnosed clinically between October 16, 2013 and March 21, 2023 were selected. With the shortest diameter axis>5 mm on the mediastinal window serving as the inclusion criterion, 108 active tuberculomas and 64 non-active tuberculomas were selected. The focused image was manually segmented using ITK-SNAP software, the volumetric CT value of the focus was calculated, and the ROC curve was analyzed. Using the final clinical diagnosis as the reference standard, the auxiliary diagnostic efficacy and consistency of the conventional CT film reading method and volumetric CT value in determining the activity of a pulmonary tuberculoma were compared.ResultsThe volumetric CT value of 108 active pulmonary tuberculoma lesions (33.39 [28.17,36.23] HU) was significantly less than 64 inactive pulmonary tuberculoma lesions (78.91 [57.81,120.31] HU); the difference was statistically significant (Z = -10.888. P < 0.001). ROC curve analysis showed that at a maximum Yoden index value of 0.963, the optimal volumetric CT threshold value was 45.32 HU, the sensitivity and specificity of the volumetric CT value in determining the activity of a pulmonary tuberculoma were 97.2% and 100.0%, respectively, and the maximum area under the ROC curve was 0.998. Taking the final clinical diagnosis as the reference standard, the sensitivity, specificity, consistency, and kappa value of the conventional CT film reading method for determining the activity of a pulmonary tuberculoma were 72.2% (78/108), 70.3% (45/64), 71.5% (123/172), and 0.413, respectively, while the corresponding volumetric CT values were 97.2% (105/108), 100.0% (64/64), 98.3% (168/172), and 0.951, respectively.ConclusionAccurately quantifying the volumetric CT value of a pulmonary tuberculoma focus determines the activity of a pulmonary tuberculoma, which has very important auxiliary diagnostic value.

Dataset Information

Quantifying the incremental value of deep learning: Application to lung nodule detection.

Publications

Quantifying the incremental value of deep learning: Application to lung nodule detection.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets