Evaluating a Fully Automated Pulmonary Nodule Detection Approach and Its Impact on Radiologist Performance.
Ontology highlight
ABSTRACT: Purpose
To compare sensitivity in the detection of lung nodules between the deep learning (DL) model and radiologists using various patient population and scanning parameters and to assess whether the radiologists' detection performance could be enhanced when using the DL model for assistance.Materials and methods
A total of 12 754 thin-section chest CT scans from January 2012 to June 2017 were retrospectively collected for DL model training, validation, and testing. Pulmonary nodules from these scans were categorized into four types: solid, subsolid, calcified, and pleural. The testing dataset was divided into three cohorts based on radiation dose, patient age, and CT manufacturer. Detection performance of the DL model was analyzed by using a free-response receiver operating characteristic curve. Sensitivities of the DL model and radiologists were compared by using exploratory data analysis. False-positive detection rates of the DL model were compared within each cohort. Detection performance of the same radiologist with and without the DL model were compared by using nodule-level sensitivity and patient-level localization receiver operating characteristic curves.Results
The DL model showed elevated overall sensitivity compared with manual review of pulmonary nodules. No significant dependence regarding radiation dose, patient age range, or CT manufacturer was observed. The sensitivity of the junior radiologist was significantly dependent on patient age. When radiologists used the DL model for assistance, their performance improved and reading time was reduced.Conclusion
DL shows promise to enhance the identification of pulmonary nodules and benefit nodule management.© RSNA, 2019Supplemental material is available for this article.
SUBMITTER: Liu K
PROVIDER: S-EPMC8017422 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:This study aimed at implementing practice to build a standardized protocol to test the performance of computer-aided detection (CAD) algorithms for pulmonary nodules. A test dataset was established according to a standardized procedure, including data collection, curation and annotation. Six types of pulmonary nodules were manually annotated as reference standard. Three specific rules to match algorithm output with reference standard were applied and compared. These rules included: (1) "center hit" [whether the center of algorithm highlighted region of interest (ROI) hit the ROI of reference standard]; (2) "center distance" (whether the distance between algorithm highlighted ROI center and reference standard center was below a certain threshold); (3) "area overlap" (whether the overlap between algorithm highlighted ROI and reference standard was above a certain threshold). Performance metrics were calculated and the results were compared among ten algorithms under test (AUTs). The test set currently consisted of CT sequences from 593 patients. Under "center hit" rule, the average recall rate, average precision, and average F1 score of ten algorithms under test were 54.68, 38.19, and 42.39%, respectively. Correspondingly, the results under "center distance" rule were 55.43, 38.69, and 42.96%, and the results under "area overlap" rule were 40.35, 27.75, and 31.13%. Among the six types of pulmonary nodules, the AUTs showed the highest miss rate for pure ground-glass nodules, with an average of 59.32%, followed by pleural nodules and solid nodules, with an average of 49.80 and 42.21%, respectively. The algorithm testing results changed along with specific matching methods adopted in the testing process. The AUTs showed uneven performance on different types of pulmonary nodules. This centralized testing protocol supports the comparison between algorithms with similar intended use, and helps evaluate algorithm performance.
| S-EPMC9768365 | biostudies-literature
Project description:BackgroundThe deep learning-based nodule detection (DLD) system improves nodule detection performance of observers on chest radiographs (CXRs). However, its performance in different pulmonary nodule (PN) locations remains unknown.MethodsWe divided the CXR intrathoracic region into non-danger zone (NDZ) and danger zone (DZ). The DZ included the lung apices, paramediastinal areas, and retrodiaphragmatic areas, where nodules could be missed. We used a dataset of 300 CXRs (100 normal and 200 abnormal images with 216 PNs [107 NDZ and 109 DZ nodules]). Eight observers (two thoracic radiologists [TRs], two non-thoracic radiologists [NTRs], and four radiology residents [RRs]) interpreted each radiograph with and without the DLD system. The metric of lesion localization fraction (LLF; the number of correctly localized lesions divided by the total number of true lesions) was used to evaluate the diagnostic performance according to the nodule location.ResultsThe DLD system demonstrated a lower LLF for the detection of DZ nodules (64.2) than that of NDZ nodules (83.2, p = 0.008). For DZ nodule detection, the LLF of the DLD system (64.2) was lower than that of TRs (81.7, p < 0.001), which was comparable to that of NTRs (56.4, p = 0.531) and RRs (56.7, p = 0.459). Nonetheless, the LLF of RRs significantly improved from 56.7 to 65.6 using the DLD system (p = 0.021) for DZ nodule detection.ConclusionThe performance of the DLD system was lower in the detection of DZ nodules compared to that of NDZ nodules. Nonetheless, RR performance in detecting DZ nodules improved upon using the DLD system.Critical relevance statementDespite the deep learning-based nodule detection system's limitations in detecting danger zone nodules, it proves beneficial for less-experienced observers by providing valuable assistance in identifying these nodules, thereby advancing nodule detection in clinical practice.Key points• The deep learning-based nodule detection (DLD) system can improve the diagnostic performance of observers in nodule detection. • The DLD system shows poor diagnostic performance in detecting danger zone nodules. • For less-experienced observers, the DLD system is helpful in detecting danger zone nodules.
| S-EPMC10509107 | biostudies-literature
Project description:Nodules form on plant roots through the symbiotic relationship between soybean (Glycine max L. Merr.) roots and bacteria (Bradyrhizobium japonicum) and are an important structure where atmospheric nitrogen (N2) is fixed into bioavailable ammonia (NH3) for plant growth and development. Nodule quantification on soybean roots is a laborious and tedious task; therefore, assessment is frequently done on a numerical scale that allows for rapid phenotyping, but is less informative and suffers from subjectivity. We report the Soybean Nodule Acquisition Pipeline (SNAP) for nodule quantification that combines RetinaNet and UNet deep learning architectures for object (i.e., nodule) detection and segmentation. SNAP was built using data from 691 unique roots from diverse soybean genotypes, vegetative growth stages, and field locations and has a good model fit (R 2 = 0.99). SNAP reduces the human labor and inconsistencies of counting nodules, while acquiring quantifiable traits related to nodule growth, location, and distribution on roots. The ability of SNAP to phenotype nodules on soybean roots at a higher throughput enables researchers to assess the genetic and environmental factors, and their interactions on nodulation from an early development stage. The application of SNAP in research and breeding pipelines may lead to more nitrogen use efficiency for soybean and other legume species cultivars, as well as enhanced insight into the plant-Bradyrhizobium relationship.
| S-EPMC8343430 | biostudies-literature
Project description:Understanding the detailed dynamics of neuronal networks will require the simultaneous measurement of spike trains from hundreds of neurons (or more). Currently, approaches to extracting spike times and labels from raw data are time consuming, lack standardization, and involve manual intervention, making it difficult to maintain data provenance and assess the quality of scientific results. Here, we describe an automated clustering approach and associated software package that addresses these problems and provides novel cluster quality metrics. We show that our approach has accuracy comparable to or exceeding that achieved using manual or semi-manual techniques with desktop central processing unit (CPU) runtimes faster than acquisition time for up to hundreds of electrodes. Moreover, a single choice of parameters in the algorithm is effective for a variety of electrode geometries and across multiple brain regions. This algorithm has the potential to enable reproducible and automated spike sorting of larger scale recordings than is currently possible.
| S-EPMC5743236 | biostudies-literature
Project description:ObjectiveInvestigate the performance of multiparametric MRI radiomic features, alone or combined with current standard-of-care methods, for pulmonary nodule classification. Assess the impact of segmentation variability on feature reproducibility and reliability.MethodsRadiomic features were extracted from 74 pulmonary nodules of 68 patients who underwent nodule resection or biopsy after MRI exam. The MRI features were compared with histopathology and conventional quantitative imaging values (maximum standardized uptake value [SUVmax] and mean Hounsfield unit [HU]) to determine whether MRI radiomic features can differentiate types of nodules and associate with SUVmax and HU using Wilcoxon rank sum test and linear regression. Diagnostic performance of features and four machine learning (ML) models were evaluated with area under the receiver operating characteristic curve (AUC) and 95% confidence intervals (CIs). Concordance correlation coefficient (CCC) assessed the segmentation variation impact on feature reproducibility and reliability.ResultsElevn diffusion-weighted features distinguished malignant from benign nodules (adjusted p < 0.05, AUC: 0.73-0.81). No features differentiated cancer types. Sixty-seven multiparametric features associated with mean CT HU and 14 correlated with SUVmax. All significant MRI features outperformed traditional imaging parameters (SUVmax, mean HU, apparent diffusion coefficient [ADC], T1, T2, dynamic contrast-enhanced imaging values) in distinguishing malignant from benign nodules with some achieving statistical significance (p < 0.05). Adding ADC and smoking history improved feature performance. Machine learning models demonstrated strong performance in nodule classification, with extreme gradient boosting (XGBoost) having the highest discrimination (AUC = 0.83, CI=[0.727, 0.932]). We found good to excellent inter- and intrareader feature reproducibility and reliability (CCC≥0.80).ConclusionEleven MRI radiomic features differentiated malignant from benign lung nodules, outperforming traditional quantitative methods. MRI radiomic ML models demonstrated good nodule classification performances with XGBoost superior to three others. There was good to excellent inter- and intrareader feature reproducibility and reliability.Advances in knowledgeOur study identified MRI radiomic features that successfully differentiated malignant from benign lung nodules and demonstrated high performance of our MR radiomic feature-based ML models for nodule classification. These new findings could help further establish thoracic MRI as a non-invasive and radiation-free alternative to standard practice for pulmonary nodule assessment.
| S-EPMC9733623 | biostudies-literature
Project description:Sleep spindles are discrete, intermittent patterns of brain activity observed in human electroencephalographic data. Increasingly, these oscillations are of biological and clinical interest because of their role in development, learning and neurological disorders. We used an Internet interface to crowdsource spindle identification by human experts and non-experts, and we compared their performance with that of automated detection algorithms in data from middle- to older-aged subjects from the general population. We also refined methods for forming group consensus and evaluating the performance of event detectors in physiological data such as electroencephalographic recordings from polysomnography. Compared to the expert group consensus gold standard, the highest performance was by individual experts and the non-expert group consensus, followed by automated spindle detectors. This analysis showed that crowdsourcing the scoring of sleep data is an efficient method to collect large data sets, even for difficult tasks such as spindle identification. Further refinements to spindle detection algorithms are needed for middle- to older-aged subjects.
| S-EPMC3972193 | biostudies-literature
Project description:An ever-growing collection of commercial biostimulants is becoming available in a wide variety of forms and compositions to improve crop performance. Given the intricate nature of deciphering the underlying mechanisms of commercial products, which typically comprise various biological components, it is crucial for research in this area to have robust tools to demonstrate their effectiveness in field trials. Here, we took a multi-attribute approach to evaluating the impact of biostimulants on crop performance. First, we assessed the impact of a biostimulant on the soil and rhizosphere microbiomes associated to crops in eight reference farms, including corn (3 farms), soybean (2), cotton (2) and sugarcane (1), in different biomes and production contexts in Brazil and Paraguay. Second, we modeled a set of integrated indicators to measure crop responses to biostimulant application, including five analytical themes as follows: i) crop development and production (9 indicators), ii) soil chemistry (9), iii) soil physics (5), iv) soil biology (6) and v) plant health (10). Amplicon 16S rRNA and ITS sequencing revealed that the use of the biostimulant consistently changes the structure of bacterial and fungal communities associated with the production system for all evaluated crops. In the rhizosphere samples, the most responsive bacterial taxa to biostimulant application were Prevotella in cotton; Prauserella and Methylovirgula in corn; and Methylocapsa in sugar cane. The most responsive fungal taxa to biostimulant use were Arachnomyces in soybean and cotton; and Rhizophlyctis in corn. The proposed integrated indicators yielded highly favorable positive impact indices (averaging at 0.80), indicating that biostimulant-treated fields correlate with better plant development and crop performance. Prominent indices were observed for indicators in four themes: soil biology (average index 0.84), crop production (0.81), soil physics (compaction reduction 0.81), and chemical fertility (0.75). The multi-attribute approach employed in this study offers an effective strategy for assessing the efficacy of biostimulant products across a wide range of crops and production systems.
| S-EPMC10449575 | biostudies-literature
Project description:ObjectivesEarly identification of lung cancer on chest radiographs improves patient outcomes. Artificial intelligence (AI) tools may increase diagnostic accuracy and streamline this pathway. This study evaluated the performance of commercially available AI-based software trained to identify cancerous lung nodules on chest radiographs.DesignThis retrospective study included primary care chest radiographs acquired in a UK centre. The software evaluated each radiograph independently and outputs were compared with two reference standards: (1) the radiologist report and (2) the diagnosis of cancer by multidisciplinary team decision. Failure analysis was performed by interrogating the software marker locations on radiographs.Participants5722 consecutive chest radiographs were included from 5592 patients (median age 59 years, 53.8% women, 1.6% prevalence of cancer).ResultsCompared with radiologist reports for nodule detection, the software demonstrated sensitivity 54.5% (95% CI 44.2% to 64.4%), specificity 83.2% (82.2% to 84.1%), positive predictive value (PPV) 5.5% (4.6% to 6.6%) and negative predictive value (NPV) 99.0% (98.8% to 99.2%). Compared with cancer diagnosis, the software demonstrated sensitivity 60.9% (50.1% to 70.9%), specificity 83.3% (82.3% to 84.2%), PPV 5.6% (4.8% to 6.6%) and NPV 99.2% (99.0% to 99.4%). Normal or variant anatomy was misidentified as an abnormality in 69.9% of the 943 false positive cases.ConclusionsThe software demonstrated considerable underperformance in this real-world patient cohort. Failure analysis suggested a lack of generalisability in the training and testing datasets as a potential factor. The low PPV carries the risk of over-investigation and limits the translation of the software to clinical practice. Our findings highlight the importance of training and testing software in representative datasets, with broader implications for the implementation of AI tools in imaging.
| S-EPMC10632826 | biostudies-literature
Project description:BackgroundLung cancer is a malignant tumor, for which pulmonary nodules are considered to be significant indicators. Early recognition and timely treatment of pulmonary nodules can contribute to improving the survival rate of patients with cancer. Positron emission tomography-computed tomography (PET/CT) is a noninvasive, fusion imaging technique that can obtain both functional and structural information of lung regions. However, studies of pulmonary nodules based on computer-aided diagnosis have primarily focused on the nodule level due to a reliance on the annotation of nodules, which is superficial and unable to contribute to the actual clinical diagnosis. The aim of this study was thus to develop a fully automated classification framework for a more comprehensive assessment of pulmonary nodules in PET/CT imaging data.MethodsWe developed a two-stage multimodal learning framework for the diagnosis of pulmonary nodules in PET/CT imaging. In this framework, Stage I focuses on pulmonary parenchyma segmentation using a pretrained U-Net and PET/CT registration. Stage II aims to extract, integrate, and recognize image-level and feature-level features by employing the three-dimensional (3D) Inception-residual net (ResNet) convolutional block attention module architecture and a dense-voting fusion mechanism.ResultsIn the experiments, the proposed model's performance was comprehensively validated using a set of real clinical data, achieving mean scores of 89.98%, 89.21%, 84.75%, 93.38%, 86.83%, and 0.9227 for accuracy, precision, recall, specificity, F1 score, and area under curve values, respectively.ConclusionsThis paper presents a two-stage multimodal learning approach for the automatic diagnosis of pulmonary nodules. The findings reveal that the main reason for limiting model performance is the nonsolitary property of nodules in pulmonary nodule diagnosis, providing direction for future research.
| S-EPMC11320548 | biostudies-literature
Project description:Microbiome profiling holds great promise for the development of novel disease biomarkers and therapeutics. Next-generation sequencing is currently the preferred method for microbiome data collection and multiple standardized tools, packages, and pipelines have been developed for the purpose of raw data processing and microbial annotation. However, these currently available pipelines come with entry-level barriers such as high-performance hardware, software installation, and sequential command-line scripting that often deter end-users. We thus created Cloud Computing for Microbiome Profiling (CCMP, https://ccmp.usc.edu), a public cloud-based web tool which combines the analytical power of current microbiome analysis platforms with a user-friendly interface. CCMP is a free-of-charge software-as-a-service (SaaS) that simplifies user experience by enabling users to complete their analysis in a single step, uploading raw sequencing data files. Once users upload 16S ribosomal RNA gene sequence data, our pipeline performs taxonomic annotation, abundance profiling, and statistical tests to report microbiota signatures altered by diseases or experimental conditions. CCMP took a 125 gigabyte (GB) input of 16S ribosomal RNA gene sequence data from 1052 specimens in FASTQ format and reported figures and tables of taxonomic annotations, statistical tests, ? and ? diversity calculations, and principal coordinate analyses within 21 hours. CCMP is the first fully-automated web interface that integrates three key solutions for large-scale data analysis: cloud computing, fast file transfer technology, and microbiome analysis tools. As a reliable platform that supplies consistent microbiome analysis, CCMP will advance microbiome research by making effortful bioinformatics easily accessible to public.
| S-EPMC6719681 | biostudies-literature