Dataset Information

Validation of automated screening for referable diabetic retinopathy with the IDx-DR device in the Hoorn Diabetes Care System.

ABSTRACT: To increase the efficiency of retinal image grading, algorithms for automated grading have been developed, such as the IDx-DR 2.0 device. We aimed to determine the ability of this device, incorporated in clinical work flow, to detect retinopathy in persons with type 2 diabetes.Retinal images of persons treated by the Hoorn Diabetes Care System (DCS) were graded by the IDx-DR device and independently by three retinal specialists using the International Clinical Diabetic Retinopathy severity scale (ICDR) and EURODIAB criteria. Agreement between specialists was calculated. Results of the IDx-DR device and experts were compared using sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV), distinguishing between referable diabetic retinopathy (RDR) and vision-threatening retinopathy (VTDR). Area under the receiver operating characteristic curve (AUC) was calculated.Of the included 1415 persons, 898 (63.5%) had images of sufficient quality according to the experts and the IDx-DR device. Referable diabetic retinopathy (RDR) was diagnosed in 22 persons (2.4%) using EURODIAB and 73 persons (8.1%) using ICDR classification. Specific intergrader agreement ranged from 40% to 61%. Sensitivity, specificity, PPV and NPV of IDx-DR to detect RDR were 91% (95% CI: 0.69-0.98), 84% (95% CI: 0.81-0.86), 12% (95% CI: 0.08-0.18) and 100% (95% CI: 0.99-1.00; EURODIAB) and 68% (95% CI: 0.56-0.79), 86% (95% CI: 0.84-0.88), 30% (95% CI: 0.24-0.38) and 97% (95% CI: 0.95-0.98; ICDR). The AUC was 0.94 (95% CI: 0.88-1.00; EURODIAB) and 0.87 (95% CI: 0.83-0.92; ICDR). For detection of VTDR, sensitivity was lower and specificity was higher compared to RDR. AUC's were comparable.Automated grading using the IDx-DR device for RDR detection is a valid method and can be used in primary care, decreasing the demand on ophthalmologists.

SUBMITTER: van der Heijden AA

PROVIDER: S-EPMC5814834 | biostudies-other | 2018 Feb

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

Validation of automated screening for referable diabetic retinopathy with the IDx-DR device in the Hoorn Diabetes Care System.

van der Heijden Amber A AA Abramoff Michael D MD Verbraak Frank F van Hecke Manon V MV Liem Albert A Nijpels Giel G

Acta ophthalmologica 20171127 1

<h4>Purpose</h4>To increase the efficiency of retinal image grading, algorithms for automated grading have been developed, such as the IDx-DR 2.0 device. We aimed to determine the ability of this device, incorporated in clinical work flow, to detect retinopathy in persons with type 2 diabetes.<h4>Methods</h4>Retinal images of persons treated by the Hoorn Diabetes Care System (DCS) were graded by the IDx-DR device and independently by three retinal specialists using the International Clinical Dia ...[more]

PMID: 29178249

Similar Datasets

Project description:ImportanceScreening for diabetic retinopathy is recommended for children with type 1 diabetes (T1D) and type 2 diabetes (T2D), yet screening rates remain low. Point-of-care diabetic retinopathy screening using autonomous artificial intelligence (AI) has become available, providing immediate results in the clinic setting, but the cost-effectiveness of this strategy compared with standard examination is unknown.ObjectiveTo assess the cost-effectiveness of detecting and treating diabetic retinopathy and its sequelae among children with T1D and T2D using AI diabetic retinopathy screening vs standard screening by an eye care professional (ECP).Design, setting, and participantsIn this economic evaluation, parameter estimates were obtained from the literature from 1994 to 2019 and assessed from March 2019 to January 2020. Parameters included out-of-pocket cost for autonomous AI screening, ophthalmology visits, and treating diabetic retinopathy; probability of undergoing standard retinal examination; relative odds of undergoing screening; and sensitivity, specificity, and diagnosability of the ECP screening examination and autonomous AI screening.Main outcomes and measuresCosts or savings to the patient based on mean patient payment for diabetic retinopathy screening examination and cost-effectiveness based on costs or savings associated with the number of true-positive results identified by diabetic retinopathy screening.ResultsIn this study, the expected true-positive proportions for standard ophthalmologic screening by an ECP were 0.006 for T1D and 0.01 for T2D, and the expected true-positive proportions for autonomous AI were 0.03 for T1D and 0.04 for T2D. The base case scenario of 20% adherence estimated that use of autonomous AI would result in a higher mean patient payment ($8.52 for T1D and $10.85 for T2D) than conventional ECP screening ($7.91 for T1D and $8.20 for T2D). However, autonomous AI screening was the preferred strategy when at least 23% of patients adhered to diabetic retinopathy screening.Conclusions and relevanceThese results suggest that point-of-care diabetic retinopathy screening using autonomous AI systems is effective and cost saving for children with diabetes and their caregivers at recommended adherence rates.

Project description:Purpose: To assess the accuracy and efficacy of a semi-automated deep learning algorithm (DLA) assisted approach to detect vision-threatening diabetic retinopathy (DR). Methods: We developed a two-step semi-automated DLA-assisted approach to grade fundus photographs for vision-threatening referable DR. Study images were obtained from the Lingtou Cohort Study, and captured at participant enrollment in 2009-2010 ("baseline images") and annual follow-up between 2011 and 2017. To begin, a validated DLA automatically graded baseline images for referable DR and classified them as positive, negative, or ungradable. Following, each positive image, all other available images from patients who had a positive image, and a 5% random sample of all negative images were selected and regraded by trained human graders. A reference standard diagnosis was assigned once all graders achieved consistent grading outcomes or with a senior ophthalmologist's final diagnosis. The semi-automated DLA assisted approach combined initial DLA screening and subsequent human grading for images identified as high-risk. This approach was further validated within the follow-up image datasets and its time and economic costs evaluated against fully human grading. Results: For evaluation of baseline images, a total of 33,115 images were included and automatically graded by the DLA. 2,604 images (480 positive results, 624 available other images from participants with a positive result, and 1500 random negative samples) were selected and regraded by graders. The DLA achieved an area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and accuracy of 0.953, 0.970, 0.879, and 88.6%, respectively. In further validation within the follow-up image datasets, a total of 88,363 images were graded using this semi-automated approach and human grading was performed on 8975 selected images. The DLA achieved an AUC, sensitivity, and specificity of 0.914, 0.852, 0.853, respectively. Compared against fully human grading, the semi-automated DLA-assisted approach achieved an estimated 75.6% time and 90.1% economic cost saving. Conclusions: The DLA described in this study was able to achieve high accuracy, sensitivity, and specificity in grading fundus images for referable DR. Validated against long-term follow-up datasets, a semi-automated DLA-assisted approach was able to accurately identify suspect cases, and minimize misdiagnosis whilst balancing safety, time, and economic cost.

Project description:BackgroundDespite recent incentives through Medicare (Australia's universal health insurance scheme) to increase retinal screening rates in primary care, comprehensive diabetic retinopathy (DR) screening has not been reached in Australia. The current study aimed to identify key factors affecting the delivery of diabetic retinopathy (DR) screening in Australian general practices.MethodsA descriptive qualitative study involving in-depth interviews was carried out from November 2019 to March 2020. Using purposive snowballing sampling, 15 general practitioners (GPs) were recruited from urban and rural general practices in New South Wales and Western Australia. A semi-structured interview guide was used to collect data from participants. All interviews were conducted over the phone by one facilitator, and each interview lasted up to 45 min. The Socio-Ecological Model was used to inform the content of the interview topic guides and subsequent data analysis. Recorded data were transcribed verbatim, and thematic analysis was conducted to identify and classify recurrent themes.ResultsOf 15 GPs interviewed, 13 were male doctors, and the mean age was 54.7 ± 15.5 years. Seven participants were practising in urban areas, while eight were practising in regional or remote areas. All participants had access to a direct ophthalmoscope, but none owned retinal cameras. None of the participants reported performing DR screening. Only three participants were aware of the Medicare Benefits Schedule (MBS) items 12,325 and 12,326 that allow GPs to bill for retinal screening. Seven themes, a combination of facilitators and barriers, emerged from interviews with the GPs. Despite the strong belief in their role in managing chronic diseases, barriers such as costs of retinal cameras, time constraints, lack of skills to make DR diagnosis, and unawareness of Medicare incentives for non-mydriatic retinal photography made it difficult to conduct DR screening in general practice. However, several enabling strategies to deliver DR screening within primary care include increasing GPs' access to continuing professional development, subsidising the cost of retinal cameras, and the need for a champion ace to take the responsibility of retinal photography.ConclusionThis study identified essential areas at the system level that require addressing to promote the broader implementation of DR screening, in particular, a nationwide awareness campaign to maximise the use of MBS items, improve GPs' competency, and subsidise costs of the retinal cameras for small and rural general practices.

Project description:BackgroundThe automated screening of patients at risk of developing diabetic retinopathy represents an opportunity to improve their midterm outcome and lower the public expenditure associated with direct and indirect costs of common sight-threatening complications of diabetes.ObjectiveThis study aimed to develop and evaluate the performance of an automated deep learning-based system to classify retinal fundus images as referable and nonreferable diabetic retinopathy cases, from international and Mexican patients. In particular, we aimed to evaluate the performance of the automated retina image analysis (ARIA) system under an independent scheme (ie, only ARIA screening) and 2 assistive schemes (ie, hybrid ARIA plus ophthalmologist screening), using a web-based platform for remote image analysis to determine and compare the sensibility and specificity of the 3 schemes.MethodsA randomized controlled experiment was performed where 17 ophthalmologists were asked to classify a series of retinal fundus images under 3 different conditions. The conditions were to (1) screen the fundus image by themselves (solo); (2) screen the fundus image after exposure to the retina image classification of the ARIA system (ARIA answer); and (3) screen the fundus image after exposure to the classification of the ARIA system, as well as its level of confidence and an attention map highlighting the most important areas of interest in the image according to the ARIA system (ARIA explanation). The ophthalmologists' classification in each condition and the result from the ARIA system were compared against a gold standard generated by consulting and aggregating the opinion of 3 retina specialists for each fundus image.ResultsThe ARIA system was able to classify referable vs nonreferable cases with an area under the receiver operating characteristic curve of 98%, a sensitivity of 95.1%, and a specificity of 91.5% for international patient cases. There was an area under the receiver operating characteristic curve of 98.3%, a sensitivity of 95.2%, and a specificity of 90% for Mexican patient cases. The ARIA system performance was more successful than the average performance of the 17 ophthalmologists enrolled in the study. Additionally, the results suggest that the ARIA system can be useful as an assistive tool, as sensitivity was significantly higher in the experimental condition where ophthalmologists were exposed to the ARIA system's answer prior to their own classification (93.3%), compared with the sensitivity of the condition where participants assessed the images independently (87.3%; P=.05).ConclusionsThese results demonstrate that both independent and assistive use cases of the ARIA system present, for Latin American countries such as Mexico, a substantial opportunity toward expanding the monitoring capacity for the early detection of diabetes-related blindness.

Project description:ImportancePrimary care professionals (PCPs) have a central role in screening for diabetic retinopathy (DR), especially in settings where access to specialty eye care is limited. Data on current DR screening practice patterns in primary care are needed to inform screening strategies.ObjectivesTo assess the practice patterns of fundoscopic examination for DR screening in a large primary care network and to evaluate the sensitivity and accuracy of PCP fundoscopy for detecting DR.Design, setting, and participantsA retrospective cohort study was performed using random sampling and manual review of electronic health records of PCP fundoscopic examination documentation compared with documentation of an examination performed by an eye care professional (ophthalmologist or optometrist) within 2 years before or after primary care encounters. From a single-institution primary care network of 28 clinics, 7449 adult patients with diabetes seen at least once in the primary care network in 2019 were eligible for this study. Data from 2001 encounters were abstracted from the electronic health record for a random sample of 767 patients. Data analysis was performed from January 2021 to May 2022.ExposureFundoscopic examination by PCPs.Main outcomes and measuresThe frequency of PCPs performing fundoscopy at least once in the calendar year for patients with diabetes. Univariate and multivariable logistic regression analyses were performed to identify patient, clinician, and clinic factors associated with PCPs performing fundoscopy at least once in the calendar year. The PCP examination results were compared with diagnoses made by eye care professionals to assess the sensitivity and accuracy of the findings from PCP examinations.ResultsAmong the 767 adult patients with diabetes included in the analysis, 387 (50.5%) were female, and the median age was 64 years (IQR, 54-71 years). Primary care professionals documented a fundoscopic examination for 93 patients (12.1%); all results were documented as normal. When eye care professional examination results were used as the reference standard, the accuracy of PCP fundoscopic examination was 62.7% (95% CI, 50.0%-73.9%) and sensitivity for detecting disease was 0.0% (95% CI, 0.0%-14.9%). No patient demographic or clinical characteristics were associated with PCPs performing fundoscopy. In multivariable logistic regression, the number of PCP years in practice was associated with greater odds of patients receiving fundoscopy at least once in the year (adjusted odds ratio per 10 years in practice, 1.26; 95% CI, 1.01-1.59; P = .04); having nurse practitioner credentials was associated with lower odds of receiving fundoscopy (adjusted odds ratio, 0.23; 95% CI, 0.04-0.79; P = .049; compared with having physician credentials); after adjusting for rural clinic location, clinic location in a primary care shortage area, and documentation of an up-to-date eye care professional examination by a PCP in the study year.Conclusions and relevanceIn this cohort study, fundoscopic examination was rarely performed and was not sensitive for detecting DR in primary care practice. Because the rate of DR screening by eye care professionals remains low, research to explore and break down barriers to the implementation of effective primary care-based DR screening strategies, such as teleretinal imaging, is needed to prevent vision loss from undiagnosed DR.

Project description:BackgroundDiabetic retinopathy (DR) is a leading cause of blindness. Our objective was to evaluate the performance of an artificial intelligence (AI) system integrated into a handheld smartphone-based retinal camera for DR screening using a single retinal image per eye.MethodsImages were obtained from individuals with diabetes during a mass screening program for DR in Blumenau, Southern Brazil, conducted by trained operators. Automatic analysis was conducted using an AI system (EyerMaps™, Phelcom Technologies LLC, Boston, USA) with one macula-centered, 45-degree field of view retinal image per eye. The results were compared to the assessment by a retinal specialist, considered as the ground truth, using two images per eye. Patients with ungradable images were excluded from the analysis.ResultsA total of 686 individuals (average age 59.2 ± 13.3 years, 56.7% women, diabetes duration 12.1 ± 9.4 years) were included in the analysis. The rates of insulin use, daily glycemic monitoring, and systemic hypertension treatment were 68.4%, 70.2%, and 70.2%, respectively. Although 97.3% of patients were aware of the risk of blindness associated with diabetes, more than half of them underwent their first retinal examination during the event. The majority (82.5%) relied exclusively on the public health system. Approximately 43.4% of individuals were either illiterate or had not completed elementary school. DR classification based on the ground truth was as follows: absent or nonproliferative mild DR 86.9%, more than mild (mtm) DR 13.1%. The AI system achieved sensitivity, specificity, positive predictive value, and negative predictive value percentages (95% CI) for mtmDR as follows: 93.6% (87.8-97.2), 71.7% (67.8-75.4), 42.7% (39.3-46.2), and 98.0% (96.2-98.9), respectively. The area under the ROC curve was 86.4%.ConclusionThe portable retinal camera combined with AI demonstrated high sensitivity for DR screening using only one image per eye, offering a simpler protocol compared to the traditional approach of two images per eye. Simplifying the DR screening process could enhance adherence rates and overall program coverage.

Dataset Information

Validation of automated screening for referable diabetic retinopathy with the IDx-DR device in the Hoorn Diabetes Care System.

Publications

Validation of automated screening for referable diabetic retinopathy with the IDx-DR device in the Hoorn Diabetes Care System.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets