Dataset Information

Performance of a Deep-Learning Algorithm vs Manual Grading for Detecting Diabetic Retinopathy in India.

ABSTRACT:

Importance

More than 60 million people in India have diabetes and are at risk for diabetic retinopathy (DR), a vision-threatening disease. Automated interpretation of retinal fundus photographs can help support and scale a robust screening program to detect DR.

Objective

To prospectively validate the performance of an automated DR system across 2 sites in India.

Design, setting, and participants

This prospective observational study was conducted at 2 eye care centers in India (Aravind Eye Hospital and Sankara Nethralaya) and included 3049 patients with diabetes. Data collection and patient enrollment took place between April 2016 and July 2016 at Aravind and May 2016 and April 2017 at Sankara Nethralaya. The model was trained and fixed in March 2016.

Interventions

Automated DR grading system compared with manual grading by 1 trained grader and 1 retina specialist from each site. Adjudication by a panel of 3 retinal specialists served as the reference standard in the cases of disagreement.

Main outcomes and measures

Sensitivity and specificity for moderate or worse DR or referable diabetic macula edema.

Results

Of 3049 patients, 1091 (35.8%) were women and the mean (SD) age for patients at Aravind and Sankara Nethralaya was 56.6 (9.0) years and 56.0 (10.0) years, respectively. For moderate or worse DR, the sensitivity and specificity for manual grading by individual nonadjudicator graders ranged from 73.4% to 89.8% and from 83.5% to 98.7%, respectively. The automated DR system's performance was equal to or exceeded manual grading, with an 88.9% sensitivity (95% CI, 85.8-91.5), 92.2% specificity (95% CI, 90.3-93.8), and an area under the curve of 0.963 on the data set from Aravind Eye Hospital and 92.1% sensitivity (95% CI, 90.1-93.8), 95.2% specificity (95% CI, 94.2-96.1), and an area under the curve of 0.980 on the data set from Sankara Nethralaya.

Conclusions and relevance

This study shows that the automated DR system generalizes to this population of Indian patients in a prospective setting and demonstrates the feasibility of using an automated DR grading system to expand screening programs.

SUBMITTER: Gulshan V

PROVIDER: S-EPMC6567842 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:Purpose: To assess the accuracy and efficacy of a semi-automated deep learning algorithm (DLA) assisted approach to detect vision-threatening diabetic retinopathy (DR). Methods: We developed a two-step semi-automated DLA-assisted approach to grade fundus photographs for vision-threatening referable DR. Study images were obtained from the Lingtou Cohort Study, and captured at participant enrollment in 2009-2010 ("baseline images") and annual follow-up between 2011 and 2017. To begin, a validated DLA automatically graded baseline images for referable DR and classified them as positive, negative, or ungradable. Following, each positive image, all other available images from patients who had a positive image, and a 5% random sample of all negative images were selected and regraded by trained human graders. A reference standard diagnosis was assigned once all graders achieved consistent grading outcomes or with a senior ophthalmologist's final diagnosis. The semi-automated DLA assisted approach combined initial DLA screening and subsequent human grading for images identified as high-risk. This approach was further validated within the follow-up image datasets and its time and economic costs evaluated against fully human grading. Results: For evaluation of baseline images, a total of 33,115 images were included and automatically graded by the DLA. 2,604 images (480 positive results, 624 available other images from participants with a positive result, and 1500 random negative samples) were selected and regraded by graders. The DLA achieved an area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and accuracy of 0.953, 0.970, 0.879, and 88.6%, respectively. In further validation within the follow-up image datasets, a total of 88,363 images were graded using this semi-automated approach and human grading was performed on 8975 selected images. The DLA achieved an AUC, sensitivity, and specificity of 0.914, 0.852, 0.853, respectively. Compared against fully human grading, the semi-automated DLA-assisted approach achieved an estimated 75.6% time and 90.1% economic cost saving. Conclusions: The DLA described in this study was able to achieve high accuracy, sensitivity, and specificity in grading fundus images for referable DR. Validated against long-term follow-up datasets, a semi-automated DLA-assisted approach was able to accurately identify suspect cases, and minimize misdiagnosis whilst balancing safety, time, and economic cost.

Project description:Summary We described a challenge named “Diabetic Retinopathy (DR)—Grading and Image Quality Estimation Challenge” in conjunction with ISBI 2020 to hold three sub-challenges and develop deep learning models for DR image assessment and grading. The scientific community responded positively to the challenge, with 34 submissions from 574 registrations. In the challenge, we provided the DeepDRiD dataset containing 2,000 regular DR images (500 patients) and 256 ultra-widefield images (128 patients), both having DR quality and grading annotations. We discussed details of the top 3 algorithms in each sub-challenges. The weighted kappa for DR grading ranged from 0.93 to 0.82, and the accuracy for image quality evaluation ranged from 0.70 to 0.65. The results showed that image quality assessment can be used as a further target for exploration. We also have released the DeepDRiD dataset on GitHub to help develop automatic systems and improve human judgment in DR screening and diagnosis. Graphical abstract Highlights • Provides the DeepDRiD dataset, performance evaluation, top methods and results• Presents deep learning approaches in DR image quality assessment and grading• Discusses the future work of DR automatic screening The bigger picture Diabetic retinopathy (DR) is the most common disease caused by diabetes. Challenges are held to address real-world issues encountered in the design of DR automated screening systems to advance the technology in this area. Thus, we described a challenge named "Diabetic Retinopathy (DR)—Grading and Image Quality Estimation Challenge" in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI 2020) for fundus image assessment and DR grading. The scientific community responded positively to the challenge. In the challenge, we provided a deep DR image dataset (DeepDRiD) containing regular DR images and ultra-widefield (UWF) DR images, both having image quality and DR grading diagnosis. We discussed details of the three best algorithms in each sub-challenges. The results by the top algorithms showed that image quality assessment can be used as a target for further exploration. In DeepDRiD challenge, organizers hold a real-world exploration in diabetic retinopathy (DR) auto-screening systems using regular fundus images from 500 participants and ultra-widefield fundus images from 128 participants. Among the 34 participating teams, we summarized the top 3 teams in the three sub-challenges involved in DR grading and image quality assessment. In addition to providing new insights into image quality assessment strategy, these models can enhance the judgment of healthcare workers in DR screening and bring precise screening results.