Dataset Information

Performance and Limitation of Machine Learning Algorithms for Diabetic Retinopathy Screening: Meta-analysis.

ABSTRACT:

Background

Diabetic retinopathy (DR), whose standard diagnosis is performed by human experts, has high prevalence and requires a more efficient screening method. Although machine learning (ML)-based automated DR diagnosis has gained attention due to recent approval of IDx-DR, performance of this tool has not been examined systematically, and the best ML technique for use in a real-world setting has not been discussed.

Objective

The aim of this study was to systematically examine the overall diagnostic accuracy of ML in diagnosing DR of different categories based on color fundus photographs and to determine the state-of-the-art ML approach.

Methods

Published studies in PubMed and EMBASE were searched from inception to June 2020. Studies were screened for relevant outcomes, publication types, and data sufficiency, and a total of 60 out of 2128 (2.82%) studies were retrieved after study selection. Extraction of data was performed by 2 authors according to PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses), and the quality assessment was performed according to the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2). Meta-analysis of diagnostic accuracy was pooled using a bivariate random effects model. The main outcomes included diagnostic accuracy, sensitivity, and specificity of ML in diagnosing DR based on color fundus photographs, as well as the performances of different major types of ML algorithms.

Results

The primary meta-analysis included 60 color fundus photograph studies (445,175 interpretations). Overall, ML demonstrated high accuracy in diagnosing DR of various categories, with a pooled area under the receiver operating characteristic (AUROC) ranging from 0.97 (95% CI 0.96-0.99) to 0.99 (95% CI 0.98-1.00). The performance of ML in detecting more-than-mild DR was robust (sensitivity 0.95; AUROC 0.97), and by subgroup analyses, we observed that robust performance of ML was not limited to benchmark data sets (sensitivity 0.92; AUROC 0.96) but could be generalized to images collected in clinical practice (sensitivity 0.97; AUROC 0.97). Neural network was the most widely used method, and the subgroup analysis revealed a pooled AUROC of 0.98 (95% CI 0.96-0.99) for studies that used neural networks to diagnose more-than-mild DR.

Conclusions

This meta-analysis demonstrated high diagnostic accuracy of ML algorithms in detecting DR on color fundus photographs, suggesting that state-of-the-art, ML-based DR screening algorithms are likely ready for clinical applications. However, a significant portion of the earlier published studies had methodology flaws, such as the lack of external validation and presence of spectrum bias. The results of these studies should be interpreted with caution.

SUBMITTER: Wu JH

PROVIDER: S-EPMC8406115 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:ObjectivesMachine learning algorithms are being increasingly used for predicting hospital readmissions. This meta-analysis evaluated the performance of logistic regression (LR) and machine learning (ML) models for the prediction of 30-day hospital readmission among patients in the US.MethodsElectronic databases (i.e., Medline, PubMed, and Embase) were searched from January 2015 to December 2019. Only studies in the English language were included. Two reviewers performed studies screening, quality appraisal, and data collection. The quality of the studies was assessed using the Quality in Prognosis Studies (QUIPS) tool. Model performance was evaluated using the Area Under the Curve (AUC). A random-effects meta-analysis was performed using STATA 16.ResultsNine studies were included based on the selection criteria. The most common ML techniques were tree-based methods such as boosting and random forest. Most of the studies had a low risk of bias (8/9). The AUC was greater with ML to predict 30-day all-cause hospital readmission compared with LR [Mean Difference (MD): 0.03; 95% Confidence Interval (CI) 0.01-0.05]. Subgroup analyses found that deep-learning methods had a better performance compared with LR (MD 0.06; 95% CI, 0.04-0.09), followed by neural networks (MD: 0.03; 95% CI, 0.03-0.03), while the AUCs of the tree-based (MD: 0.02; 95% CI -0.00-0.04) and kernel-based (MD: 0.02; 95% CI 0.02 (-0.13-0.16) methods were no different compared to LR. More than half of the studies evaluated heart failure-related rehospitalization (N = 5). For the readmission prediction among heart failure patients, ML performed better compared with LR, with a mean difference in AUC of 0.04 (95% CI, 0.01-0.07). The leave-one-out sensitivity analysis confirmed the robustness of the findings.ConclusionMultiple ML methods were used to predict 30-day all-cause hospital readmission. Performance varied across the ML methods, with deep-learning methods showing the best performance over the LR.

Project description:ObjectivesDifferent machine learning algorithms (MLAs) for automated segmentation of gliomas have been reported in the literature. Automated segmentation of different tumor characteristics can be of added value for the diagnostic work-up and treatment planning. The purpose of this study was to provide an overview and meta-analysis of different MLA methods.MethodsA systematic literature review and meta-analysis was performed on the eligible studies describing the segmentation of gliomas. Meta-analysis of the performance was conducted on the reported dice similarity coefficient (DSC) score of both the aggregated results as two subgroups (i.e., high-grade and low-grade gliomas). This study was registered in PROSPERO prior to initiation (CRD42020191033).ResultsAfter the literature search (n = 734), 42 studies were included in the systematic literature review. Ten studies were eligible for inclusion in the meta-analysis. Overall, the MLAs from the included studies showed an overall DSC score of 0.84 (95% CI: 0.82-0.86). In addition, a DSC score of 0.83 (95% CI: 0.80-0.87) and 0.82 (95% CI: 0.78-0.87) was observed for the automated glioma segmentation of the high-grade and low-grade gliomas, respectively. However, heterogeneity was considerably high between included studies, and publication bias was observed.ConclusionMLAs facilitating automated segmentation of gliomas show good accuracy, which is promising for future implementation in neuroradiology. However, before actual implementation, a few hurdles are yet to be overcome. It is crucial that quality guidelines are followed when reporting on MLAs, which includes validation on an external test set.Key points• MLAs from the included studies showed an overall DSC score of 0.84 (95% CI: 0.82-0.86), indicating a good performance. • MLA performance was comparable when comparing the segmentation results of the high-grade gliomas and the low-grade gliomas. • For future studies using MLAs, it is crucial that quality guidelines are followed when reporting on MLAs, which includes validation on an external test set.

Project description:Background:Epilepsy is a disorder that can manifest as abnormalities in neurological or physical function. Stress cardiomyopathy is closely associated with neurological stimulation. However, the mechanisms underlying the interrelationship between epilepsy and stress cardiomyopathy are unclear. This paper aims to explore the genetic features and potential molecular mechanisms shared in epilepsy and stress cardiomyopathy. Methods:By analyzing the epilepsy dataset and stress cardiomyopathy dataset separately, the intersection of the two disease co-expressed differential genes is obtained, the co-expressed differential genes reveal the biological functions, the network is constructed, and the core modules are identified to reveal the interaction mechanism, the co-expressed genes with diagnostic validity are screened by machine learning algorithms, and the co-expressed genes are validated in parallel on the epilepsy single-cell data and the stress cardiomyopathy rat model. Results: Epilepsy causes stress cardiomyopathy, and its key pathways are Complement and coagulation cascades, HIF-1 signaling pathway, its key co-expressed genes include SPOCK2, CTSZ, HLA-DMB, ALDOA, SFRP1, ERBB3.The key immune cell subpopulations localized by single-cell data are the T_cells subgroup, Microglia subgroup, Macrophage subgroup, Astrocyte subgroup, and Oligodendrocytes subgroup. Conclusion: We believe epilepsy causing stress cardiomyopathy results from a multi-gene, multi-pathway combination. We identified the core co-expressed genes (SPOCK2, CTSZ, HLA-DMB, ALDOA, SFRP1, ERBB3) and the pathways that function in them (Complement and coagulation cascades, HIF-1 signaling pathway,JAK-STAT signaling pathway), and finally localized their key cellular subgroups(T_cells subgroup, Microglia subgroup, Macrophage subgroup, Astrocyte subgroup,and Oligodendrocytes subgroup). Also, combining cell subpopulations with hypercoagulability as well as sympathetic excitation further narrowed the cell subpopulations of related functions.