Deep learning versus human graders for classifying diabetic retinopathy severity in a nationwide screening program.
Ontology highlight
ABSTRACT:
SUBMITTER: Raumviboonsuk P
PROVIDER: S-EPMC6550283 | biostudies-literature | 2019
REPOSITORIES: biostudies-literature
Publications
Deep learning versus human graders for classifying diabetic retinopathy severity in a nationwide screening program.
Raumviboonsuk Paisan P Krause Jonathan J Chotcomwongse Peranut P Sayres Rory R Raman Rajiv R Widner Kasumi K Campana Bilson J L BJL Phene Sonia S Hemarat Kornwipa K Tadarati Mongkol M Silpa-Archa Sukhum S Limwattanayingyong Jirawut J Rao Chetan C Kuruvilla Oscar O Jung Jesse J Tan Jeffrey J Orprayoon Surapong S Kangwanwongpaisan Chawawat C Sukumalpaiboon Ramase R Luengchaichawang Chainarong C Fuangkaew Jitumporn J Kongsap Pipat P Chualinpha Lamyong L Saree Sarawuth S Kawinpanitan Srirut S Mitvongsa Korntip K Lawanasakol Siriporn S Thepchatri Chaiyasit C Wongpichedchai Lalita L Corrado Greg S GS Peng Lily L Webster Dale R DR
NPJ digital medicine 20190410
Deep learning algorithms have been used to detect diabetic retinopathy (DR) with specialist-level accuracy. This study aims to validate one such algorithm on a large-scale clinical population, and compare the algorithm performance with that of human graders. A total of 25,326 gradable retinal images of patients with diabetes from the community-based, nationwide screening program of DR in Thailand were analyzed for DR severity and referable diabetic macular edema (DME). Grades adjudicated by a pa ...[more]
Similar Datasets
Project description:ObjectiveTo evaluate diabetic retinopathy (DR) screening via deep learning (DL) and trained human graders (HG) in a longitudinal cohort, as case spectrum shifts based on treatment referral and new-onset DR.MethodsWe randomly selected patients with diabetes screened twice, two years apart within a nationwide screening program. The reference standard was established via adjudication by retina specialists. Each patient's color fundus photographs were graded, and a patient was considered as having sight-threatening DR (STDR) if the worse eye had severe nonproliferative DR, proliferative DR, or diabetic macular edema. We compared DR screening via two modalities: DL and HG. For each modality, we simulated treatment referral by excluding patients with detected STDR from the second screening using that modality.ResultsThere were 5,738 patients (12.3% STDR) in the first screening. DL and HG captured different numbers of STDR cases, and after simulated referral and excluding ungradable cases, 4,148 and 4,263 patients remained in the second screening, respectively. The STDR prevalence at the second screening was 5.1% and 6.8% for DL- and HG-based screening, respectively. Along with the prevalence decrease, the sensitivity for both modalities decreased from the first to the second screening (DL: from 95% to 90%, p = 0.008; HG: from 74% to 57%, p < 0.001). At both the first and second screenings, the rate of false negatives for the DL was a fifth that of HG (0.5-0.6% vs. 2.9-3.2%).ConclusionOn 2-year longitudinal follow-up of a DR screening cohort, STDR prevalence decreased for both DL- and HG-based screening. Follow-up screenings in longitudinal DR screening can be more difficult and induce lower sensitivity for both DL and HG, though the false negative rate was substantially lower for DL. Our data may be useful for health-economics analyses of longitudinal screening settings.
| S-EPMC7758133 | biostudies-literature
Project description:IntroductionDeep learning (DL) for screening diabetic retinopathy (DR) has the potential to address limited healthcare resources by enabling expanded access to healthcare. However, there is still limited health economic evaluation, particularly in low- and middle-income countries, on this subject to aid decision-making for DL adoption.MethodsIn the context of a middle-income country (MIC), using Thailand as a model, we constructed a decision tree-Markov hybrid model to estimate lifetime costs and outcomes of Thailand's national DR screening program via DL and trained human graders (HG). We calculated the incremental cost-effectiveness ratio (ICER) between the two strategies. Sensitivity analyses were performed to probe the influence of modeling parameters.ResultsFrom a societal perspective, screening with DL was associated with a reduction in costs of ~ US$ 2.70, similar quality-adjusted life-years (QALY) of + 0.0043, and an incremental net monetary benefit of ~ US$ 24.10 in the base case. In sensitivity analysis, DL remained cost-effective even with a price increase from US$ 1.00 to US$ 4.00 per patient at a Thai willingness-to-pay threshold of ~ US$ 4.997 per QALY gained. When further incorporating recent findings suggesting improved compliance to treatment referral with DL, our analysis models effectiveness benefits of ~ US$ 20 to US$ 50 depending on compliance.ConclusionDR screening using DL in an MIC using Thailand as a model may result in societal cost-savings and similar health outcomes compared with HG. This study may provide an economic rationale to expand DL-based DR screening in MICs as an alternative solution for limited availability of skilled human resources for primary screening, particularly in MICs with similar prevalence of diabetes and low compliance to referrals for treatment.
| S-EPMC10011252 | biostudies-literature
Project description:PurposeTo develop an automated diabetic retinopathy (DR) staging system using optical coherence tomography angiography (OCTA) images with a convolutional neural network (CNN) and to verify the feasibility of the system.MethodsIn this retrospective cross-sectional study, a total of 918 data sets of 3 × 3 mm2 OCTA images and 917 data sets of 6 × 6 mm2 OCTA images were obtained from 1118 eyes. A deep CNN and four traditional machine learning models were trained with annotations made by a retinal specialist based on ultra-widefield fluorescein angiography. Separately, the same images of the test data sets were independently graded by two human experts. The results of the CNN algorithm were compared with those of traditional machine learning-based classifiers and human experts.ResultsThe proposed CNN achieved an accuracy of 0.728, a sensitivity of 0.675, a specificity of 0.944, an F1 score of 0.683, and a quadratic weighted κ of 0.908 for a six-level staging task, which were far superior to the results of traditional machine learning methods or human experts. The CNN algorithm showed a better performance using 6 × 6 mm2 rather than 3 × 3 mm2 sized OCTA images and using combined data rather than a separate OCTA layer alone.ConclusionsCNN-based classification using OCTA images can provide reliable assistance to clinicians for DR classification.Translational relevanceThis CNN algorithm can guide the clinical decision for invasive angiography or referrals to ophthalmology specialists, helping to create more efficient diagnostic workflow in primary care settings.
| S-EPMC8899862 | biostudies-literature
Project description:BACKGROUND:Diabetic retinopathy (DR) is the leading cause of blindness among working-age adults worldwide. Early detection and treatment are necessary to forestall vision loss from DR. METHODS:A working group of ophthalmic and diabetes experts was established to develop a consensus on the key principles of an effective DR screening program. Recommendations are based on analysis of a structured literature review. RESULTS:The recommendations for implementing an effective DR screening program are: (1) Examination methods must be suitable for the screening region, and DR classification/grading systems must be systematic and uniformly applied. Two-field retinal imaging is sufficient for DR screening and is preferable to seven-field imaging, and referable DR should be well defined and reliably identifiable by qualified screening staff; (2) in many countries/regions, screening can and should take place outside the ophthalmology clinic; (3) screening staff should be accredited and show evidence of ongoing training; (4) screening programs should adhere to relevant national quality assurance standards; (5) studies that use uniform definitions of risk to determine optimum risk-based screening intervals are required; (6) technology infrastructure should be in place to ensure that high-quality images can be stored securely to protect patient information; (7) although screening for diabetic macular edema (DME) in conjunction with DR evaluations may have merit, there is currently insufficient evidence to support implementation of programs solely for DME screening. CONCLUSION:Use of these recommendations may yield more effective DR screening programs that reduce the risk of vision loss worldwide.
| S-EPMC7311555 | biostudies-literature
Project description:ObjectivesTo compare the accuracy of trained level 1 diabetic retinopathy (DR) graders (nurses, endocrinologists and one general practitioner), level 2 graders (midlevel ophthalmologists) and level 3 graders (senior ophthalmologists) in Vietnam against a reference standard from the UK and assess the impact of supplementary targeted grader training.DesignDiagnostic test accuracy study.SettingSecondary care hospitals in Southern Vietnam.ParticipantsDR training was delivered to Vietnamese graders in February 2018 by National Health Service (NHS) UK graders. Two-field retinal images (412 patient images) were graded by 14 trained graders in Vietnam between August and October 2018 and then regraded retrospectively by an NHS-certified reference standard UK optometrist (phase I). Further DR training based on phase I results was delivered to graders in November 2019. After training, a randomised subset of images from January to October 2020 (115 patient images) was graded by six of the original cohort (phase II). The reference grader regraded all images from phase I and II retrospectively in masked fashion.Primary and secondary outcome measuresSensitivity was calculated at the two different time points, and χ2 was used to test significance.ResultsIn phase I, the sensitivity for detecting any DR for all grader groups in Vietnam was low (41.8-42.2%) and improved in phase II after additional training was delivered (51.3-87.2%). The greatest improvement was seen among level 1 graders (p<0.001), and the lowest improvement was observed among level 3 graders (p=0.326). There was a statistically significant improvement in sensitivity for detecting referable DR and referable diabetic macular oedema between all grader levels. The post-training values ranged from 40.0 to 61.5% (including ungradable images) and 55.6%-90.0% (excluding ungradable images).ConclusionsThis study demonstrates that targeted training interventions can improve accuracy of DR grading. These findings have important implications for improving service delivery in DR screening programmes in low-resource settings.
| S-EPMC9472142 | biostudies-literature
Project description:Diabetic retinopathy occurs as a result of the harmful effects of diabetes on the eyes. Diabetic retinopathy is also a disease that should be diagnosed early. If not treated early, vision loss may occur. It is estimated that one third of more than half a million diabetic patients will have diabetic retinopathy by the 22nd century. Many effective methods have been proposed for disease detection with deep learning. In this study, unlike other studies, a deep learning-based method has been proposed in which diabetic retinopathy lesions are detected automatically and independently of datasets, and the detected lesions are classified. In the first stage of the proposed method, a data pool is created by collecting diabetic retinopathy data from different datasets. With Faster RCNN, lesions are detected, and the region of interests are marked. The images obtained in the second stage are classified using the transfer learning and attention mechanism. The method tested in Kaggle and MESSIDOR datasets reached 99.1% and 100% ACC and 99.9% and 100% AUC, respectively. When the obtained results are compared with other results in the literature, it is seen that more successful results are obtained.
| S-EPMC8184323 | biostudies-literature
Project description:Diabetic retinopathy (DR) screening images are heterogeneous and contain undesirable non-retinal, incorrect field and ungradable samples which require curation, a laborious task to perform manually. We developed and validated single and multi-output laterality, retinal presence, retinal field and gradability classification deep learning (DL) models for automated curation. The internal dataset comprised of 7743 images from DR screening (UK) with 1479 external test images (Portugal and Paraguay). Internal vs external multi-output laterality AUROC were right (0.994 vs 0.905), left (0.994 vs 0.911) and unidentifiable (0.996 vs 0.680). Retinal presence AUROC were (1.000 vs 1.000). Retinal field AUROC were macula (0.994 vs 0.955), nasal (0.995 vs 0.962) and other retinal field (0.997 vs 0.944). Gradability AUROC were (0.985 vs 0.918). DL effectively detects laterality, retinal presence, retinal field and gradability of DR screening images with generalisation between centres and populations. DL models could be used for automated image curation within DR screening.
| S-EPMC9249740 | biostudies-literature
Project description:BackgroundFor medical artificial intelligence (AI) training and validation, human expert labels are considered the gold standard that represents the correct answers or desired outputs for a given data set. These labels serve as a reference or benchmark against which the model's predictions are compared.ObjectiveThis study aimed to assess the accuracy of a custom deep learning (DL) algorithm on classifying diabetic retinopathy (DR) and further demonstrate how label errors may contribute to this assessment in a nationwide DR-screening program.MethodsFundus photographs from the Lifeline Express, a nationwide DR-screening program, were analyzed to identify the presence of referable DR using both (1) manual grading by National Health Service England-certificated graders and (2) a DL-based DR-screening algorithm with validated good lab performance. To assess the accuracy of labels, a random sample of images with disagreement between the DL algorithm and the labels was adjudicated by ophthalmologists who were masked to the previous grading results. The error rates of labels in this sample were then used to correct the number of negative and positive cases in the entire data set, serving as postcorrection labels. The DL algorithm's performance was evaluated against both pre- and postcorrection labels.ResultsThe analysis included 736,083 images from 237,824 participants. The DL algorithm exhibited a gap between the real-world performance and the lab-reported performance in this nationwide data set, with a sensitivity increase of 12.5% (from 79.6% to 92.5%, P<.001) and a specificity increase of 6.9% (from 91.6% to 98.5%, P<.001). In the random sample, 63.6% (560/880) of negative images and 5.2% (140/2710) of positive images were misclassified in the precorrection human labels. High myopia was the primary reason for misclassifying non-DR images as referable DR images, while laser spots were predominantly responsible for misclassified referable cases. The estimated label error rate for the entire data set was 1.2%. The label correction was estimated to bring about a 12.5% enhancement in the estimated sensitivity of the DL algorithm (P<.001).ConclusionsLabel errors based on human image grading, although in a small percentage, can significantly affect the performance evaluation of DL algorithms in real-world DR screening.
| S-EPMC11358665 | biostudies-literature
Project description:Purpose: To assess the accuracy and efficacy of a semi-automated deep learning algorithm (DLA) assisted approach to detect vision-threatening diabetic retinopathy (DR). Methods: We developed a two-step semi-automated DLA-assisted approach to grade fundus photographs for vision-threatening referable DR. Study images were obtained from the Lingtou Cohort Study, and captured at participant enrollment in 2009-2010 ("baseline images") and annual follow-up between 2011 and 2017. To begin, a validated DLA automatically graded baseline images for referable DR and classified them as positive, negative, or ungradable. Following, each positive image, all other available images from patients who had a positive image, and a 5% random sample of all negative images were selected and regraded by trained human graders. A reference standard diagnosis was assigned once all graders achieved consistent grading outcomes or with a senior ophthalmologist's final diagnosis. The semi-automated DLA assisted approach combined initial DLA screening and subsequent human grading for images identified as high-risk. This approach was further validated within the follow-up image datasets and its time and economic costs evaluated against fully human grading. Results: For evaluation of baseline images, a total of 33,115 images were included and automatically graded by the DLA. 2,604 images (480 positive results, 624 available other images from participants with a positive result, and 1500 random negative samples) were selected and regraded by graders. The DLA achieved an area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and accuracy of 0.953, 0.970, 0.879, and 88.6%, respectively. In further validation within the follow-up image datasets, a total of 88,363 images were graded using this semi-automated approach and human grading was performed on 8975 selected images. The DLA achieved an AUC, sensitivity, and specificity of 0.914, 0.852, 0.853, respectively. Compared against fully human grading, the semi-automated DLA-assisted approach achieved an estimated 75.6% time and 90.1% economic cost saving. Conclusions: The DLA described in this study was able to achieve high accuracy, sensitivity, and specificity in grading fundus images for referable DR. Validated against long-term follow-up datasets, a semi-automated DLA-assisted approach was able to accurately identify suspect cases, and minimize misdiagnosis whilst balancing safety, time, and economic cost.
| S-EPMC8656222 | biostudies-literature
Project description:BACKGROUND:Pay-for-Performance programs have shown improvement in indicators monitoring adequacy and target achievement in diabetic care. However, less is known regarding the impact of this program on the occurrence and long-term effects of diabetic retinopathy. The objective of this study was to determine the effect of pay-for-performance program on the development of treatment needed for diabetic retinopathy in type 2 diabetes patients. METHODS:We conducted a nationwide retrospective cohort study with a matching design using the Taiwan National Health Insurance Research Database from 2000 to 2012. The outcome was defined as the treatment needed diabetic retinopathy. We matched Pay-for-Performance and non-Pay-for-Performance groups for age, gender, year diabetes was diagnosed and study enrollment, and duration of follow-up. RESULTS:A total of 9311 patients entered the study cohort, of whom 2157 were registered in the Pay-for-Performance group and 7154 matched in the non-Pay-for-Performance group. The incidence of treatment needed diabetic retinopathy was not significantly different in two groups. However, the incidence of treatment needed diabetic retinopathy was significantly different if restricted the non-Pay-for-Performance group who had at least 1 eye examination or optical coherence tomography within 1 year (adjusted hazard ratio, 0.78; 95% confidence interval, 0.64-0.94). CONCLUSIONS:Pay-for-Performance is valuable in preventing the development of treatment needed diabetic retinopathy, which could be attributed to the routine eye examination required in the Pay-for-Performance program. We could improve our diabetic care by promoting eye health education and patient awareness on the importance of regular examinations.
| S-EPMC6094472 | biostudies-literature