Dataset Information

Large-scale assessment of consistency in sleep stage scoring rules among multiple sleep centers using an interpretable machine learning algorithm.

ABSTRACT:

Study objectives

Polysomnography is the gold standard in identifying sleep stages; however, there are discrepancies in how technicians use the standards. Because organizing meetings to evaluate this discrepancy and/or reach a consensus among multiple sleep centers is time-consuming, we developed an artificial intelligence system to efficiently evaluate the reliability and consistency of sleep scoring and hence the sleep center quality.

Methods

An interpretable machine learning algorithm was used to evaluate the interrater reliability (IRR) of sleep stage annotation among sleep centers. The artificial intelligence system was trained to learn raters from 1 hospital and was applied to patients from the same or other hospitals. The results were compared with the experts' annotation to determine IRR. Intracenter and intercenter assessments were conducted on 679 patients without sleep apnea from 6 sleep centers in Taiwan. Centers with potential quality issues were identified by the estimated IRR.

Results

In the intracenter assessment, the median accuracy ranged from 80.3%-83.3%, with the exception of 1 hospital, which had an accuracy of 72.3%. In the intercenter assessment, the median accuracy ranged from 75.7%-83.3% when the 1 hospital was excluded from testing and training. The performance of the proposed method was higher for the N2, awake, and REM sleep stages than for the N1 and N3 stages. The significant IRR discrepancy of the 1 hospital suggested a quality issue. This quality issue was confirmed by the physicians in charge of the 1 hospital.

Conclusions

The proposed artificial intelligence system proved effective in assessing IRR and hence the sleep center quality.

SUBMITTER: Liu GR

PROVIDER: S-EPMC7853209 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:Machine learning has been applied in recent years to categorize sleep stages (NREM, REM, and wake) using electroencephalogram (EEG) recordings; however, a well-validated sleep scoring automatic pipeline in rodent research is still not publicly available. Here, we present IntelliSleepScorer, a software package with a graphic user interface to score sleep stages automatically in mice. IntelliSleepScorer uses the light gradient boosting machine (LightGBM) to score sleep stages for each epoch of recordings. We developed LightGBM models using a large cohort of data, which consisted of 5776 h of sleep EEG and electromyogram (EMG) signals across 519 unique recordings from 124 mice. The LightGBM model achieved an overall accuracy of 95.2% and a Cohen's kappa of 0.91, which outperforms the baseline models such as the logistic regression model (accuracy = 93.3%, kappa = 0.88) and the random forest model (accuracy = 94.3%, kappa = 0.89). The overall performance of the LightGBM model as well as the performance across different sleep stages are on par with that of the human experts. Most importantly, we validated the generalizability of the LightGBM models: (1) The LightGBM model performed well on two publicly available, independent datasets (kappa > = 0.80), which have different sampling frequency and epoch lengths; (2) The LightGBM model performed well on data recorded at a lower sampling frequency (kappa = 0.90); (3) The performance of the LightGBM model is not affected by the light/dark cycle; and (4) A modified LightGBM model performed well on data containing only one EEG and one EMG electrode (kappa > = 0.89). Taken together, the LightGBM models offer state-of-the-art performance for automatic sleep stage scoring in mice. Last, we implemented the IntelliSleepScorer software package based on the validated model to provide an out-of-box solution to sleep researchers (available for download at https://sites.broadinstitute.org/pan-lab/resources ).

Project description:BackgroundEmergency readmission poses an additional burden on both patients and healthcare systems. Risk stratification is the first step of transitional care interventions targeted at reducing readmission. To accurately predict the short- and intermediate-term risks of readmission and provide information for further temporal risk stratification, we developed and validated an interpretable machine learning risk scoring system.MethodsIn this retrospective study, all emergency admission episodes from January 1st 2009 to December 31st 2016 at a tertiary hospital in Singapore were assessed. The primary outcome was time to emergency readmission within 90 days post discharge. The Score for Emergency ReAdmission Prediction (SERAP) tool was derived via an interpretable machine learning-based system for time-to-event outcomes. SERAP is six-variable survival score, and takes the number of emergency admissions last year, age, history of malignancy, history of renal diseases, serum creatinine level, and serum albumin level during index admission into consideration.FindingsA total of 293,589 ED admission episodes were finally included in the whole cohort. Among them, 203,748 episodes were included in the training cohort, 50,937 episodes in the validation cohort, and 38,904 in the testing cohort. Readmission within 90 days was documented in 80,213 (27.3%) episodes, with a median time to emergency readmission of 22 days (Interquartile range: 8-47). For different time points, the readmission rates observed in the whole cohort were 6.7% at 7 days, 10.6% at 14 days, 13.6% at 21 days, 16.4% at 30 days, and 23.0% at 60 days. In the testing cohort, the SERAP achieved an integrated area under the curve of 0.737 (95% confidence interval: 0.730-0.743). For a specific 30-day readmission prediction, SERAP outperformed the LACE index (Length of stay, Acuity of admission, Charlson comorbidity index, and Emergency department visits in past six months) and the HOSPITAL score (Hemoglobin at discharge, discharge from an Oncology service, Sodium level at discharge, Procedure during the index admission, Index Type of admission, number of Admissions during the last 12 months, and Length of stay). Besides 30-day readmission, SERAP can predict readmission rates at any time point during the 90-day period.InterpretationBetter performance in risk prediction was achieved by the SERAP than other existing scores, and accurate information about time to emergency readmission was generated for further temporal risk stratification and clinical decision-making. In the future, external validation studies are needed to evaluate the SERAP at different settings and assess their real-world performance.FundingThis study was supported by the Singapore National Medical Research Council under the PULSES Center Grant, and Duke-NUS Medical School.