Dataset Information

Development and Comparison of Machine Learning Algorithms to Determine Visual Field Progression.

ABSTRACT:

Purpose

To develop and test machine learning classifiers (MLCs) for determining visual field progression.

Methods

In total, 90,713 visual fields from 13,156 eyes were included. Six different progression algorithms (linear regression of mean deviation, linear regression of the visual field index, Advanced Glaucoma Intervention Study algorithm, Collaborative Initial Glaucoma Treatment Study algorithm, pointwise linear regression [PLR], and permutation of PLR) were applied to classify each eye as progressing or stable. Six MLCs were applied (logistic regression, random forest, extreme gradient boosting, support vector classifier, convolutional neural network, fully connected neural network) using a training and testing set. For MLC input, visual fields for a given eye were divided into the first and second half and each location averaged over time within each half. Each algorithm was tested for accuracy, sensitivity, positive predictive value, and class bias with a subset of visual fields labeled by a panel of three experts from 161 eyes.

Results

MLCs had similar performance metrics as some of the conventional algorithms and ranged from 87% to 91% accurate with sensitivity ranging from 0.83 to 0.88 and specificity from 0.92 to 0.96. All conventional algorithms showed significant class bias, meaning each individual algorithm was more likely to grade uncertain cases as either progressing or stable (P ≤ 0.01). Conversely, all MLCs were balanced, meaning they were equally likely to grade uncertain cases as either progressing or stable (P ≥ 0.08).

Conclusions

MLCs showed a moderate to high level of accuracy, sensitivity, and specificity and were more balanced than conventional algorithms.

Translational relevance

MLCs may help to determine visual field progression.

SUBMITTER: Saeedi O

PROVIDER: S-EPMC8237084 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:PURPOSE:To determine the agreement of 6 established visual field (VF) progression algorithms in a large dataset of VFs from multiple institutions and to determine predictors of discordance among these algorithms. DESIGN:Retrospective longitudinal cohort study. PARTICIPANTS:Visual fields from 5 major eye care institutions in the United States were analyzed, including a subset of eyes with at least 5 Swedish interactive threshold algorithm standard 24-2 VFs that met our reliability criteria. Of a total of 831?240 VFs, a subset of 90?713 VFs from 13?156 eyes of 8499 patients met the inclusion criteria. METHODS:Six commonly used VF progression algorithms (mean deviation [MD] slope, VF index slope, Advanced Glaucoma Intervention Study, Collaborative Initial Glaucoma Treatment Study, pointwise linear regression, and permutation of pointwise linear regression) were applied to this cohort, and each eye was determined to be stable or progressing using each measure. Agreement between individual algorithms was tested using Cohen's ? coefficient. Bivariate and multivariate analyses were used to determine predictors of discordance (3 algorithms progressing and 3 algorithms stable). MAIN OUTCOME MEASURES:Agreement and discordance between algorithms. RESULTS:Individual algorithms showed poor to moderate agreement with each other when compared directly (? range, 0.12-0.52). Based on at least 4 algorithms, 11.7% of eyes progressed. Major predictors of discordance or lack of agreement among algorithms were more depressed initial MD (P < 0.01) and older age at first available VF (P < 0.01). A greater number of VFs (P < 0.01), more years of follow-up (P < 0.01), and eye care institution (P = 0.03) also were associated with discordance. CONCLUSIONS:This extremely large comparative series demonstrated that existing algorithms have limited agreement and that agreement varies with clinical parameters, including institution. These issues underscore the challenges to the clinical use and application of progression algorithms and of applying big-data results to individual practices.

Project description:BackgroundDespite the promise of machine learning (ML) to inform individualized medical care, the clinical utility of ML in medicine has been limited by the minimal interpretability and black box nature of these algorithms.ObjectiveThe study aimed to demonstrate a general and simple framework for generating clinically relevant and interpretable visualizations of black box predictions to aid in the clinical translation of ML.MethodsTo obtain improved transparency of ML, simplified models and visual displays can be generated using common methods from clinical practice such as decision trees and effect plots. We illustrated the approach based on postprocessing of ML predictions, in this case random forest predictions, and applied the method to data from the Left Ventricular (LV) Structural Predictors of Sudden Cardiac Death (SCD) Registry for individualized risk prediction of SCD, a leading cause of death.ResultsWith the LV Structural Predictors of SCD Registry data, SCD risk predictions are obtained from a random forest algorithm that identifies the most important predictors, nonlinearities, and interactions among a large number of variables while naturally accounting for missing data. The black box predictions are postprocessed using classification and regression trees into a clinically relevant and interpretable visualization. The method also quantifies the relative importance of an individual or a combination of predictors. Several risk factors (heart failure hospitalization, cardiac magnetic resonance imaging indices, and serum concentration of systemic inflammation) can be clearly visualized as branch points of a decision tree to discriminate between low-, intermediate-, and high-risk patients.ConclusionsThrough a clinically important example, we illustrate a general and simple approach to increase the clinical translation of ML through clinician-tailored visual displays of results from black box algorithms. We illustrate this general model-agnostic framework by applying it to SCD risk prediction. Although we illustrate the methods using SCD prediction with random forest, the methods presented are applicable more broadly to improving the clinical translation of ML, regardless of the specific ML algorithm or clinical application. As any trained predictive model can be summarized in this manner to a prespecified level of precision, we encourage the use of simplified visual displays as an adjunct to the complex predictive model. Overall, this framework can allow clinicians to peek inside the black box and develop a deeper understanding of the most important features from a model to gain trust in the predictions and confidence in applying them to clinical care.

Project description:BackgroundUrinary protein quantification is critical for assessing the severity of chronic kidney disease (CKD). However, the current procedure for determining the severity of CKD is completed through evaluating 24-h urinary protein, which is inconvenient during follow-up.ObjectiveTo quickly predict the severity of CKD using more easily available demographic and blood biochemical features during follow-up, we developed and compared several predictive models using statistical, machine learning and neural network approaches.MethodsThe clinical and blood biochemical results from 551 patients with proteinuria were collected. Thirteen blood-derived tests and 5 demographic features were used as non-urinary clinical variables to predict the 24-h urinary protein outcome response. Nine predictive models were established and compared, including logistic regression, Elastic Net, lasso regression, ridge regression, support vector machine, random forest, XGBoost, neural network and k-nearest neighbor. The AU-ROC, sensitivity (recall), specificity, accuracy, log-loss and precision of each of the models were evaluated. The effect sizes of each variable were analysed and ranked.ResultsThe linear models including Elastic Net, lasso regression, ridge regression and logistic regression showed the highest overall predictive power, with an average AUC and a precision above 0.87 and 0.8, respectively. Logistic regression ranked first, reaching an AUC of 0.873, with a sensitivity and specificity of 0.83 and 0.82, respectively. The model with the highest sensitivity was Elastic Net (0.85), while XGBoost showed the highest specificity (0.83). In the effect size analyses, we identified that ALB, Scr, TG, LDL and EGFR had important impacts on the predictability of the models, while other predictors such as CRP, HDL and SNA were less important.ConclusionsBlood-derived tests could be applied as non-urinary predictors during outpatient follow-up. Features in routine blood tests, including ALB, Scr, TG, LDL and EGFR levels, showed predictive ability for CKD severity. The developed online tool can facilitate the prediction of proteinuria progress during follow-up in clinical practice.

Project description:OBJECTIVE:The Centers for Disease Control and Prevention (CDC) coordinates a labor-intensive process to measure the prevalence of autism spectrum disorder (ASD) among children in the United States. Random forests methods have shown promise in speeding up this process, but they lag behind human classification accuracy by about 5%. We explore whether more recently available document classification algorithms can close this gap. MATERIALS AND METHODS:Using data gathered from a single surveillance site, we applied 8 supervised learning algorithms to predict whether children meet the case definition for ASD based solely on the words in their evaluations. We compared the algorithms' performance across 10 random train-test splits of the data, using classification accuracy, F1 score, and number of positive calls to evaluate their potential use for surveillance. RESULTS:Across the 10 train-test cycles, the random forest and support vector machine with Naive Bayes features (NB-SVM) each achieved slightly more than 87% mean accuracy. The NB-SVM produced significantly more false negatives than false positives (P = 0.027), but the random forest did not, making its prevalence estimates very close to the true prevalence in the data. The best-performing neural network performed similarly to the random forest on both measures. DISCUSSION:The random forest performed as well as more recently available models like the NB-SVM and the neural network, and it also produced good prevalence estimates. NB-SVM may not be a good candidate for use in a fully-automated surveillance workflow due to increased false negatives. More sophisticated algorithms, like hierarchical convolutional neural networks, may not be feasible to train due to characteristics of the data. Current algorithms might perform better if the data are abstracted and processed differently and if they take into account information about the children in addition to their evaluations. CONCLUSION:Deep learning models performed similarly to traditional machine learning methods at predicting the clinician-assigned case status for CDC's autism surveillance system. While deep learning methods had limited benefit in this task, they may have applications in other surveillance systems.