Dataset Information

Large-scale assessment of consistency in sleep stage scoring rules among multiple sleep centers using an interpretable machine learning algorithm.

ABSTRACT:

Study objectives

Polysomnography is the gold standard in identifying sleep stages; however, there are discrepancies in how technicians use the standards. Because organizing meetings to evaluate this discrepancy and/or reach a consensus among multiple sleep centers is time-consuming, we developed an artificial intelligence system to efficiently evaluate the reliability and consistency of sleep scoring and hence the sleep center quality.

Methods

An interpretable machine learning algorithm was used to evaluate the interrater reliability (IRR) of sleep stage annotation among sleep centers. The artificial intelligence system was trained to learn raters from 1 hospital and was applied to patients from the same or other hospitals. The results were compared with the experts' annotation to determine IRR. Intracenter and intercenter assessments were conducted on 679 patients without sleep apnea from 6 sleep centers in Taiwan. Centers with potential quality issues were identified by the estimated IRR.

Results

In the intracenter assessment, the median accuracy ranged from 80.3%-83.3%, with the exception of 1 hospital, which had an accuracy of 72.3%. In the intercenter assessment, the median accuracy ranged from 75.7%-83.3% when the 1 hospital was excluded from testing and training. The performance of the proposed method was higher for the N2, awake, and REM sleep stages than for the N1 and N3 stages. The significant IRR discrepancy of the 1 hospital suggested a quality issue. This quality issue was confirmed by the physicians in charge of the 1 hospital.

Conclusions

The proposed artificial intelligence system proved effective in assessing IRR and hence the sleep center quality.

SUBMITTER: Liu GR

PROVIDER: S-EPMC7853209 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundEnd stage renal disease (ESRD) describes the most severe stage of chronic kidney disease (CKD), when patients need dialysis or renal transplant. There is often a delay in recognizing, diagnosing, and treating the various etiologies of CKD. The objective of the present study was to employ machine learning algorithms to develop a prediction model for progression to ESRD based on a large-scale multidimensional database.MethodsThis study analyzed 10,000,000 medical insurance claims from 550,000 patient records using a commercial health insurance database. Inclusion criteria were patients over the age of 18 diagnosed with CKD Stages 1-4. We compiled 240 predictor candidates, divided into six feature groups: demographics, chronic conditions, diagnosis and procedure features, medication features, medical costs, and episode counts. We used a feature embedding method based on implementation of the Word2Vec algorithm to further capture temporal information for the three main components of the data: diagnosis, procedures, and medications. For the analysis, we used the gradient boosting tree algorithm (XGBoost implementation).ResultsThe C-statistic for the model was 0.93 [(0.916-0.943) 95% confidence interval], with a sensitivity of 0.715 and specificity of 0.958. Positive Predictive Value (PPV) was 0.517, and Negative Predictive Value (NPV) was 0.981. For the top 1 percentile of patients identified by our model, the PPV was 1.0. In addition, for the top 5 percentile of patients identified by our model, the PPV was 0.71. All the results above were tested on the test data only, and the threshold used to obtain these results was 0.1. Notable features contributing to the model were chronic heart and ischemic heart disease as a comorbidity, patient age, and number of hypertensive crisis events.ConclusionsWhen a patient is approaching the threshold of ESRD risk, a warning message can be sent electronically to the physician, who will initiate a referral for a nephrology consultation to ensure an investigation to hasten the establishment of a diagnosis and initiate management and therapy when appropriate.