Beyond Performance Metrics: Automatic Deep Learning Retinal OCT Analysis Reproduces Clinical Trial Outcome.
Ontology highlight
ABSTRACT: Purpose
To validate the efficacy of a fully automatic, deep learning-based segmentation algorithm beyond conventional performance metrics by measuring the primary outcome of a clinical trial for macular telangiectasia type 2 (MacTel2).Design
Evaluation of diagnostic test or technology.Participants
A total of 92 eyes from 62 participants with MacTel2 from a phase 2 clinical trial (NCT01949324) randomized to 1 of 2 treatment groups METHODS: The ellipsoid zone (EZ) defect areas were measured on spectral domain OCT images of each eye at 2 time points (baseline and month 24) by a fully automatic, deep learning-based segmentation algorithm. The change in EZ defect area from baseline to month 24 was calculated and analyzed according to the clinical trial protocol.Main outcome measure
Difference in the change in EZ defect area from baseline to month 24 between the 2 treatment groups.Results
The difference in the change in EZ defect area from baseline to month 24 between the 2 treatment groups measured by the fully automatic segmentation algorithm was 0.072±0.035 mm2 (P = 0.021). This was comparable to the outcome of the clinical trial using semiautomatic measurements by expert readers, 0.065±0.033 mm2 (P = 0.025).Conclusions
The fully automatic segmentation algorithm was as accurate as semiautomatic expert segmentation to assess EZ defect areas and was able to reliably reproduce the statistically significant primary outcome measure of the clinical trial. This approach, to validate the performance of an automatic segmentation algorithm on the primary clinical trial end point, provides a robust gauge of its clinical applicability.
SUBMITTER: Loo J
PROVIDER: S-EPMC7246171 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:PurposeRetinal toxicity resulting from hydroxychloroquine use manifests photoreceptor loss and disruption of the ellipsoid zone (EZ) reflectivity band detectable on spectral-domain (SD) OCT imaging. This study investigated whether an automatic deep learning-based algorithm can detect and quantitate EZ loss on SD OCT images with an accuracy comparable with that of human annotations.DesignRetrospective analysis of data acquired in a prospective, single-center, case-control study.ParticipantsEighty-five patients (168 eyes) who were long-term hydroxychloroquine users (average exposure time, 14 ± 7.2 years).MethodsA mask region-based convolutional neural network (M-RCNN) was implemented and trained on individual OCT B-scans. Scan-by-scan detections were aggregated to produce an en face map of EZ loss per 3-dimensional SD OCT volume image. To improve the accuracy and robustness of the EZ loss map, a dual network architecture was proposed that learns to detect EZ loss in parallel using horizontal (horizontal mask region-based convolutional neural network [M-RCNNH]) and vertical (vertical mask region-based convolutional neural network [M-RCNNV]) B-scans independently. To quantify accuracy, 10-fold cross-validation was performed.Main outcome measuresPrecision, recall, intersection over union (IOU), F1-score metrics, and measured total EZ loss area were compared against human grader annotations and with the determination of toxicity based on the recommended screening guidelines.ResultsThe combined projection network demonstrated the best overall performance: precision, 0.90 ± 0.09; recall, 0.88 ± 0.08; and F1 score, 0.89 ± 0.07. The combined model performed superiorly to the M-RCNNH only model (precision, 0.79 ± 0.17; recall, 0.96 ± 0.04; IOU, 0.78 ± 0.15; and F1 score, 0.86 ± 0.12) and M-RCNNV only model (precision, 0.71 ± 0.21; recall, 0.94 ± 0.06; IOU, 0.69 ± 0.21; and F1 score, 0.79 ± 0.16). The accuracy was comparable with the variability of human experts: precision, 0.85 ± 0.09; recall, 0.98 ± 0.01; IOU, 0.82 ± 0.12; and F1 score, 0.91 ± 0.06. Automatically generated en face EZ loss maps provide quantitative SD OCT metrics for accurate toxicity determination combined with other functional testing.ConclusionsThe algorithm can provide a fast, objective, automatic method for measuring areas with EZ loss and can serve as a quantitative assistance tool to screen patients for the presence and extent of toxicity.
| S-EPMC9560656 | biostudies-literature
Project description:The analysis of the choroid in the eye is crucial for our understanding of a range of ocular diseases and physiological processes. Optical coherence tomography (OCT) imaging provides the ability to capture highly detailed cross-sectional images of the choroid yet only a very limited number of commercial OCT instruments provide methods for automatic segmentation of choroidal tissue. Manual annotation of the choroidal boundaries is often performed but this is impractical due to the lengthy time taken to analyse large volumes of images. Therefore, there is a pressing need for reliable and accurate methods to automatically segment choroidal tissue boundaries in OCT images. In this work, a variety of patch-based and fully-convolutional deep learning methods are proposed to accurately determine the location of the choroidal boundaries of interest. The effect of network architecture, patch-size and contrast enhancement methods was tested to better understand the optimal architecture and approach to maximize performance. The results are compared with manual boundary segmentation used as a ground-truth, as well as with a standard image analysis technique. Results of total retinal layer segmentation are also presented for comparison purposes. The findings presented here demonstrate the benefit of deep learning methods for segmentation of the chorio-retinal boundary analysis in OCT images.
| S-EPMC6746702 | biostudies-literature
Project description:PurposeTo evaluate the reliability of automated fluid detection in identifying retinal fluid activity in OCT scans of patients treated with anti-VEGF therapy for neovascular age-related macular degeneration by correlating human expert and automated measurements with central retinal subfield thickness (CSFT) and fluid volume values.MethodsWe utilized an automated deep learning approach to quantify macular fluid in SD-OCT volumes (Cirrus, Spectralis, Topcon) from patients of HAWK and HARRIER Studies. Three-dimensional volumes for IRF and SRF were measured at baseline and under therapy in the central millimeter and compared to fluid gradings, CSFT and foveal centerpoint thickness (CPT) values measured by the Vienna Reading Center.Results41.906 SD-OCT volume scans were included into the analysis. Concordance between human expert grading and automated algorithm performance reached AUC values of 0.93/0.85 for IRF and 0.87 for SRF in HARRIER/HAWK in the central millimeter. IRF volumes showed a moderate correlation with CSFT at baseline (HAWK: r = 0.54; HARRIER: r = 0.62) and weaker correlation under therapy (HAWK: r = 0.44; HARRIER: r = 0.34). SRF and CSFT correlations were low at baseline (HAWK: r = 0.29; HARRIER: r = 0.22) and under therapy (HAWK: r = 0.38; HARRIER: r = 0.45). The residual standard error (IRF: 75.90 µm; SRF: 95.26 µm) and marginal residual standard deviations (IRF: 46.35 µm; SRF: 44.19 µm) of fluid volume were high compared to the range of CSFT values.ConclusionDeep learning-based segmentation of retinal fluid performs reliably on OCT images. CSFT values are weak indicators for fluid activity in nAMD. Automated quantification of fluid types, highlight the potential of deep learning-based approaches to objectively monitor anti-VEGF therapy.
| S-EPMC10698046 | biostudies-literature
Project description:Deep neural networks have been increasingly proposed for automated screening and diagnosis of retinal diseases from optical coherence tomography (OCT), but often provide high-confidence predictions on out-of-distribution (OOD) cases, compromising their clinical usage. With this in mind, we performed an in-depth comparative analysis of the state-of-the-art uncertainty estimation methods for OOD detection in retinal OCT imaging. The analysis was performed within the use-case of automated screening and staging of age-related macular degeneration (AMD), one of the leading causes of blindness worldwide, where we achieved a macro-average area under the curve (AUC) of 0.981 for AMD classification. We focus on a few-shot Outlier Exposure (OE) method and the detection of near-OOD cases that share pathomorphological characteristics with the inlier AMD classes. Scoring the OOD case based on the Cosine distance in the feature space from the penultimate network layer proved to be a robust approach for OOD detection, especially in combination with the OE. Using Cosine distance and only 8 outliers exposed per class, we were able to improve the near-OOD detection performance of the OE with Reject Bucket method by [Formula: see text] 10% compared to without OE, reaching an AUC of 0.937. The Cosine distance served as a robust metric for OOD detection of both known and unknown classes and should thus be considered as an alternative to the reject bucket class probability in OE approaches, especially in the few-shot scenario. The inclusion of these methodologies did not come at the expense of classification performance, and can substantially improve the reliability and trustworthiness of the resulting deep learning-based diagnostic systems in the context of retinal OCT.
| S-EPMC10533534 | biostudies-literature
Project description:Deep learning (DL) has been successfully applied to the diagnosis of ophthalmic diseases. However, rare diseases are commonly neglected due to insufficient data. Here, we demonstrate that few-shot learning (FSL) using a generative adversarial network (GAN) can improve the applicability of DL in the optical coherence tomography (OCT) diagnosis of rare diseases. Four major classes with a large number of datasets and five rare disease classes with a few-shot dataset are included in this study. Before training the classifier, we constructed GAN models to generate pathological OCT images of each rare disease from normal OCT images. The Inception-v3 architecture was trained using an augmented training dataset, and the final model was validated using an independent test dataset. The synthetic images helped in the extraction of the characteristic features of each rare disease. The proposed DL model demonstrated a significant improvement in the accuracy of the OCT diagnosis of rare retinal diseases and outperformed the traditional DL models, Siamese network, and prototypical network. By increasing the accuracy of diagnosing rare retinal diseases through FSL, clinicians can avoid neglecting rare diseases with DL assistance, thereby reducing diagnosis delay and patient burden.
| S-EPMC7829497 | biostudies-literature
Project description:PurposeThe purpose of this study was to investigate the effect of device and scan size on quantitative optical coherence tomography angiography (OCT-A) metrics.MethodsThe 3 × 3 mm scans from Optovue AngioVue and Zeiss AngioPlex systems were included for 18 eyes of 18 subjects without ocular pathology. The foveal avascular zone (FAZ) was segmented manually by two observers, from which estimates of FAZ area (using both the nominal image scale and the axial length corrected image scale) and acircularity were derived. Three scan sizes (3 mm, 6 mm HD, and 8 mm) from the AngioVue system were included for 15 eyes of 15 subjects without ocular pathology. For each subject, larger image sizes were resized to the same resolution as 3 × 3 mm scans, aligned, then cropped to a common area. FAZ area, FAZ acircularity, average and total parafoveal intercapillary area, vessel density, and vessel end points were computed.ResultsBetween the devices used here, there were no significant differences in FAZ acircularity (P = 0.88) or FAZ area using scaled (P = 0.11) or unscaled images (P = 0.069). Although there was no significant difference in FAZ area across scan sizes (P = 0.30), vessel morphometry metrics were all significantly influenced by scan size.ConclusionsThe scan devices and sizes used here do not affect FAZ area measures derived from manual segmentations. In contrast, vessel morphometry metrics are affected by scan size. As individual differences in axial length induce differences in absolute scan size, extreme care should be taken when interpreting metrics of vessel morphometry, both between and within OCT-A devices.Translational relevanceA better characterization of the confounds surrounding OCT-A retinal vasculature metrics can lead to improved application of these metrics as biomarkers for retinal and systemic diseases.
| S-EPMC7545061 | biostudies-literature
Project description:PurposeTo develop a fully-automatic hybrid algorithm to jointly segment and quantify biomarkers of polypoidal choroidal vasculopathy (PCV) on indocyanine green angiography (ICGA) and spectral domain-OCT (SD-OCT) images.DesignEvaluation of diagnostic test or technology.ParticipantsSeventy-two participants with PCV enrolled in clinical studies at Singapore National Eye Center.MethodsThe dataset consisted of 2-dimensional (2-D) ICGA and 3-dimensional (3-D) SD-OCT images which were spatially registered and manually segmented by clinicians. A deep learning-based hybrid algorithm called PCV-Net was developed for automatic joint segmentation of biomarkers. The PCV-Net consisted of a 2-D segmentation branch for ICGA and 3-D segmentation branch for SD-OCT. We developed fusion attention modules to connect the 2-D and 3-D branches for effective use of the spatial correspondence between the imaging modalities by sharing learned features. We also used self-supervised pretraining and ensembling to further enhance the performance of the algorithm without the need for additional datasets. We compared the proposed PCV-Net to several alternative model variants.Main outcome measuresThe PCV-Net was evaluated based on the Dice similarity coefficient (DSC) of the segmentations and the Pearson's correlation and absolute difference of the clinical measurements obtained from the segmentations. Manual grading was used as the gold standard.ResultsThe PCV-Net showed good performance compared to manual grading and alternative model variants based on both quantitative and qualitative analyses. Compared to the baseline variant, PCV-Net improved the DSC by 0.04 to 0.43 across the different biomarkers, increased the correlations, and decreased the absolute differences of clinical measurements of interest. Specifically, the largest average (mean ± standard error) DSC improvement was for intraretinal fluid, from 0.02 ± 0.00 (baseline variant) to 0.45 ± 0.06 (PCV-Net). In general, improving trends were observed across the model variants as more technical specifications were added, demonstrating the importance of each aspect of the proposed method.ConclusionThe PCV-Net has the potential to aid clinicians in disease assessment and research to improve clinical understanding and management of PCV.Financial disclosuresProprietary or commercial disclosure may be found after the references.
| S-EPMC10070921 | biostudies-literature
Project description:ObjectiveTo develop and validate a deep learning algorithm that could automate the annotation of scleral spur (SS) and segmentation of anterior chamber (AC) structures for measurements of AC, iris, and angle width parameters in anterior segment OCT (ASOCT) scans.DesignCross-sectional study.SubjectsData from 2 population-based studies (i.e., the Singapore Chinese Eye Study and Singapore Malay Eye Study) and 1 clinical study on angle-closure disease were included in algorithm development. A separate clinical study on angle-closure disease was used for external validation.MethodImage contrast of ASOCT scans were first enhanced with CycleGAN. We utilized a heat map regression approach with coarse-to-fine framework for SS annotation. Then, an ensemble network of U-Net, full resolution residual network, and full resolution U-Net was used for structure segmentation. Measurements obtained from predicted SSs and structure segmentation were measured and compared with measurements obtained from manual SS annotation and structure segmentation (i.e., ground truth).Main outcome measuresWe measured Euclidean distance and intraclass correlation coefficients (ICC) to evaluate SS annotation and Dice similarity coefficient for structure segmentation. The ICC, Bland-Altman plot, and repeatability coefficient were used to evaluate agreement and precision of measurements.ResultsFor SS annotation, our algorithm achieved a Euclidean distance of 124.7 μm, ICC ≥ 0.95, and a 3.3% error rate. For structure segmentation, we obtained Dice similarity coefficient ≥ 0.91 for cornea, iris, and AC segmentation. For angle width measurements, ≥ 95% of data points were within the 95% limits-of-agreement in Bland-Altman plot with insignificant systematic bias (all P > 0.12). The ICC ranged from 0.71-0.87 for angle width measurements, 0.54 for IT750, 0.83-0.85 for other iris measurements, and 0.89-0.99 for AC measurements. Using the same SS coordinates from a human expert, measurements obtained from our algorithm were generally less variable than measurements obtained from a semiautomated angle assessment program.ConclusionWe developed a deep learning algorithm that could automate SS annotation and structure segmentation in ASOCT scans like human experts, in both open-angle and angle-closure eyes. This algorithm reduces the time needed and subjectivity in obtaining ASOCT measurements.Financial disclosuresThe author(s) have no proprietary or commercial interest in any materials discussed in this article.
| S-EPMC10587633 | biostudies-literature
Project description:The article presents an AI-based fungi species recognition system for a citizen-science community. The system's real-time identification too - FungiVision - with a mobile application front-end, led to increased public interest in fungi, quadrupling the number of citizens collecting data. FungiVision, deployed with a human-in-the-loop, reaches nearly 93% accuracy. Using the collected data, we developed a novel fine-grained classification dataset - Danish Fungi 2020 (DF20) - with several unique characteristics: species-level labels, a small number of errors, and rich observation metadata. The dataset enables the testing of the ability to improve classification using metadata, e.g., time, location, habitat and substrate, facilitates classifier calibration testing and finally allows the study of the impact of the device settings on the classification performance. The continual flow of labelled data supports improvements of the online recognition system. Finally, we present a novel method for the fungi recognition service, based on a Vision Transformer architecture. Trained on DF20 and exploiting available metadata, it achieves a recognition error that is 46.75% lower than the current system. By providing a stream of labeled data in one direction, and an accuracy increase in the other, the collaboration creates a virtuous cycle helping both communities.
| S-EPMC8779018 | biostudies-literature
Project description:Rice disease has serious negative effects on crop yield, and the correct diagnosis of rice diseases is the key to avoid these effects. However, the existing disease diagnosis methods for rice are neither accurate nor efficient, and special equipment is often required. In this study, an automatic diagnosis method was developed and implemented in a smartphone app. The method was developed using deep learning based on a large dataset that contained 33,026 images of six types of rice diseases: leaf blast, false smut, neck blast, sheath blight, bacterial stripe disease, and brown spot. The core of the method was the Ensemble Model in which submodels were integrated. Finally, the Ensemble Model was validated using a separate set of images. Results showed that the three best submodels were DenseNet-121, SE-ResNet-50, and ResNeSt-50, in terms of several attributes, such as, learning rate, precision, recall, and disease recognition accuracy. Therefore, these three submodels were selected and integrated in the Ensemble Model. The Ensemble Model minimized confusion among the different types of disease, reducing misdiagnosis of the disease. Using the Ensemble Model to diagnose six types of rice diseases, an overall accuracy of 91% was achieved, which is considered to be reasonably good, considering the appearance similarities in some types of rice disease. The smartphone app allowed the client to use the Ensemble Model on the web server through a network, which was convenient and efficient for the field diagnosis of rice leaf blast, false smut, neck blast, sheath blight, bacterial stripe disease, and brown spot.
| S-EPMC8416767 | biostudies-literature