Dataset Information

Remote Tool-Based Adjudication for Grading Diabetic Retinopathy.

ABSTRACT:

Purpose

To present and evaluate a remote, tool-based system and structured grading rubric for adjudicating image-based diabetic retinopathy (DR) grades.

Methods

We compared three different procedures for adjudicating DR severity assessments among retina specialist panels, including (1) in-person adjudication based on a previously described procedure (Baseline), (2) remote, tool-based adjudication for assessing DR severity alone (TA), and (3) remote, tool-based adjudication using a feature-based rubric (TA-F). We developed a system allowing graders to review images remotely and asynchronously. For both TA and TA-F approaches, images with disagreement were reviewed by all graders in a round-robin fashion until disagreements were resolved. Five panels of three retina specialists each adjudicated a set of 499 retinal fundus images (1 panel using Baseline, 2 using TA, and 2 using TA-F adjudication). Reliability was measured as grade agreement among the panels using Cohen's quadratically weighted kappa. Efficiency was measured as the number of rounds needed to reach a consensus for tool-based adjudication.

Results

The grades from remote, tool-based adjudication showed high agreement with the Baseline procedure, with Cohen's kappa scores of 0.948 and 0.943 for the two TA panels, and 0.921 and 0.963 for the two TA-F panels. Cases adjudicated using TA-F were resolved in fewer rounds compared with TA (P < 0.001; standard permutation test).

Conclusions

Remote, tool-based adjudication presents a flexible and reliable alternative to in-person adjudication for DR diagnosis. Feature-based rubrics can help accelerate consensus for tool-based adjudication of DR without compromising label quality.

Translational relevance

This approach can generate reference standards to validate automated methods, and resolve ambiguous diagnoses by integrating into existing telemedical workflows.

SUBMITTER: Schaekermann M

PROVIDER: S-EPMC6922270 | biostudies-literature | 2019 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Remote Tool-Based Adjudication for Grading Diabetic Retinopathy.

Schaekermann Mike M Hammel Naama N Terry Michael M Ali Tayyeba K TK Liu Yun Y Basham Brian B Campana Bilson B Chen William W Ji Xiang X Krause Jonathan J Corrado Greg S GS Peng Lily L Webster Dale R DR Law Edith E Sayres Rory R

Translational vision science & technology 20191101 6

<h4>Purpose</h4>To present and evaluate a remote, tool-based system and structured grading rubric for adjudicating image-based diabetic retinopathy (DR) grades.<h4>Methods</h4>We compared three different procedures for adjudicating DR severity assessments among retina specialist panels, including (1) in-person adjudication based on a previously described procedure (Baseline), (2) remote, tool-based adjudication for assessing DR severity alone (TA), and (3) remote, tool-based adjudication using a ...[more]

PMID: 31867141

Similar Datasets

Project description:Summary We described a challenge named “Diabetic Retinopathy (DR)—Grading and Image Quality Estimation Challenge” in conjunction with ISBI 2020 to hold three sub-challenges and develop deep learning models for DR image assessment and grading. The scientific community responded positively to the challenge, with 34 submissions from 574 registrations. In the challenge, we provided the DeepDRiD dataset containing 2,000 regular DR images (500 patients) and 256 ultra-widefield images (128 patients), both having DR quality and grading annotations. We discussed details of the top 3 algorithms in each sub-challenges. The weighted kappa for DR grading ranged from 0.93 to 0.82, and the accuracy for image quality evaluation ranged from 0.70 to 0.65. The results showed that image quality assessment can be used as a further target for exploration. We also have released the DeepDRiD dataset on GitHub to help develop automatic systems and improve human judgment in DR screening and diagnosis. Graphical abstract Highlights • Provides the DeepDRiD dataset, performance evaluation, top methods and results• Presents deep learning approaches in DR image quality assessment and grading• Discusses the future work of DR automatic screening The bigger picture Diabetic retinopathy (DR) is the most common disease caused by diabetes. Challenges are held to address real-world issues encountered in the design of DR automated screening systems to advance the technology in this area. Thus, we described a challenge named "Diabetic Retinopathy (DR)—Grading and Image Quality Estimation Challenge" in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI 2020) for fundus image assessment and DR grading. The scientific community responded positively to the challenge. In the challenge, we provided a deep DR image dataset (DeepDRiD) containing regular DR images and ultra-widefield (UWF) DR images, both having image quality and DR grading diagnosis. We discussed details of the three best algorithms in each sub-challenges. The results by the top algorithms showed that image quality assessment can be used as a target for further exploration. In DeepDRiD challenge, organizers hold a real-world exploration in diabetic retinopathy (DR) auto-screening systems using regular fundus images from 500 participants and ultra-widefield fundus images from 128 participants. Among the 34 participating teams, we summarized the top 3 teams in the three sub-challenges involved in DR grading and image quality assessment. In addition to providing new insights into image quality assessment strategy, these models can enhance the judgment of healthcare workers in DR screening and bring precise screening results.

Project description:ImportanceTelemedicine in retinopathy of prematurity (ROP) has the potential for delivering timely care to premature infants at risk for serious ROP.ObjectiveTo describe the characteristics of eyes at risk for ROP to provide insights into what types of ROP are most easily detected early by image grading.Design, setting, and participantsSecondary analysis of eyes with referral-warranted (RW) ROP (stage 3 ROP, zone I ROP, plus disease) on diagnostic examination from the Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity (e-ROP) study was conducted from May 1, 2011, to October 31, 2013, in 1257 premature infants with birth weights less than 1251 g in 13 neonatal units in North America. Data analysis was performed between February 1, 2016, and June 5, 2017.InterventionsSerial imaging sessions with concurrent diagnostic examinations for ROP.Main outcomes and measuresTime of detecting RW-ROP on image evaluation compared with clinical examination.ResultsIn the e-ROP study, 246 infants (492 eyes) were included in the analysis; 138 (56.1%) were male. A total of 447 eyes had RW-ROP on diagnostic examination. Image grading in 123 infants (mean [SD] gestational age, 24.8 [1.4] weeks) detected RW-ROP earlier than diagnostic examination (early) in 191 (42.7%) eyes by about 15 days and detected RW-ROP in 123 infants (mean [SD] gestational age, 24.6 [1.5] weeks) at the same time (same) in 200 (44.7%) eyes. Most of the early eyes (153 [80.1%]) interpreted as being RW-ROP positive on imaging evaluation agreed with examination findings when the examination subsequently documented RW-ROP. At the sessions in which RW-ROP was first found by examination, stage 3 or more in 123 infants (mean [SD] gestational age, 24.8 [1.4] weeks) ROP was noted earlier on image evaluation in 151 of 191 early eyes (79.1%) and in 172 of 200 of same eyes (86.0%) (P = .08); the presence of zone I ROP was detected in 57 of 191 (29.8%) early eyes vs 64 of 200 (32.0%) same eyes (P = .90); and plus disease was noted in 30 of 191 (15.7%) early eyes and 45 of 200 (22.5%) same eyes (P = .08).Conclusions and relevanceIn both early and same eyes, zone I and/or stage 3 ROP determined a significant proportion of RW-ROP; plus disease played a relatively minor role. In most early RW-ROP eyes, the findings were consistent with clinical examination and/or image grading at the next session. Because ROP telemedicine is used more widely, development of standard approaches and protocols is essential.

Project description:BackgroundTo evaluate the accuracy and reliability of fundus retinal photos graded by local graders in Bangladesh with those graded by an expert at the Centre for Eye Research Australia (CERA) in the context of mass scale diabetic retinopathy (DR) screening in Bangladesh.MethodsA population-based cross-sectional study of 3,104 adults identified 213 (7.2%) eligible patients with diabetes of age ? 40 years in 2012-2013. Retinal photographs were collected using a non-mydriatic digital fundus retinal camera and a two-field imaging protocol. The photos were graded by two remote graders (G1 and G2) who were trained by a retinal specialist (RS) in Bangladesh, by the RS himself, and by a Centre for Eye Research Australia (CERA) grader. The local graders up skilled their grading ability by comparing 30% of the photos graded by the CERA grader with their own grades. Learning from that exercise was applied to the remaining 70% of photos, which were re-graded. Reliability and accuracy of grading amongst the graders were reported using cross tabulation, inter- and intra-grader reliability, and with sensitivity and specificity.ResultsOf 122 eyes from 61 patients, the mild (R1) DR was estimated to be 14 to 25%, pre-proliferative (R2) DR 4-8%, and proliferative (R3) DR 0.8 to 1.6%, whereas 25%, 8%, 18%, and 15% were found to be ungradable by CERA, RS, G1, and G2, respectively. Of 8 (6.6%) eyes identified as R2 by the CERA grader, 5 (63%), 3 (38%) and 3 (38%) were correctly classified as R2, whereas the rest were classified either as R1 or R3 but none were classified as no DR (R0) or ungradable by the RS, G1 and G2, respectively. After getting experience reviewing the 30% test set graded by the CERA grader, the local graders graded moderate and severe DR with 100% accuracy. After excluding ungradable photos, the sensitivity (specificity) relative to the CERA grader was 82% (88%) before and 80% (93%) after training for G1 and 56% (87%) before and 77% (90%) after training for G2. In case of maculopathy, the CERA grader reported 11.2% eyes with maculopathy, which included 100% of the 4.9% by RS, 6.6% by G1, and 7.4% by G2.ConclusionsLocal graders in Bangladesh are able to grade retinal photos with high accuracy if the DR is at least of a moderate level. With appropriate training and experience, local graders have the ability to contribute significantly to the grading of millions of retinal photos, which required grading in resource- poor countries.

Project description:ImportanceMore than 60 million people in India have diabetes and are at risk for diabetic retinopathy (DR), a vision-threatening disease. Automated interpretation of retinal fundus photographs can help support and scale a robust screening program to detect DR.ObjectiveTo prospectively validate the performance of an automated DR system across 2 sites in India.Design, setting, and participantsThis prospective observational study was conducted at 2 eye care centers in India (Aravind Eye Hospital and Sankara Nethralaya) and included 3049 patients with diabetes. Data collection and patient enrollment took place between April 2016 and July 2016 at Aravind and May 2016 and April 2017 at Sankara Nethralaya. The model was trained and fixed in March 2016.InterventionsAutomated DR grading system compared with manual grading by 1 trained grader and 1 retina specialist from each site. Adjudication by a panel of 3 retinal specialists served as the reference standard in the cases of disagreement.Main outcomes and measuresSensitivity and specificity for moderate or worse DR or referable diabetic macula edema.ResultsOf 3049 patients, 1091 (35.8%) were women and the mean (SD) age for patients at Aravind and Sankara Nethralaya was 56.6 (9.0) years and 56.0 (10.0) years, respectively. For moderate or worse DR, the sensitivity and specificity for manual grading by individual nonadjudicator graders ranged from 73.4% to 89.8% and from 83.5% to 98.7%, respectively. The automated DR system's performance was equal to or exceeded manual grading, with an 88.9% sensitivity (95% CI, 85.8-91.5), 92.2% specificity (95% CI, 90.3-93.8), and an area under the curve of 0.963 on the data set from Aravind Eye Hospital and 92.1% sensitivity (95% CI, 90.1-93.8), 95.2% specificity (95% CI, 94.2-96.1), and an area under the curve of 0.980 on the data set from Sankara Nethralaya.Conclusions and relevanceThis study shows that the automated DR system generalizes to this population of Indian patients in a prospective setting and demonstrates the feasibility of using an automated DR grading system to expand screening programs.

Dataset Information

Remote Tool-Based Adjudication for Grading Diabetic Retinopathy.

Purpose

Methods

Results

Conclusions

Translational relevance

Publications

Remote Tool-Based Adjudication for Grading Diabetic Retinopathy.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets