Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods.
Ontology highlight
ABSTRACT:
SUBMITTER: Warby SC
PROVIDER: S-EPMC3972193 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Accurate and efficient detection of sleep spindles is a methodological challenge. The present study describes a method of using non-experts for manual detection of sleep spindles. We recruited five experts and 168 non-experts to manually identify spindles in stage N2 and stage N3 sleep data using a MATLAB interface. Scorers classified each spindle into definite and indefinite spindle (with weights of 1 and 0.5, respectively). First, a method of optimizing the thresholds of the expert/non-expert group consensus according to the results of experts and non-experts themselves is described. Using this method, we established expert and non-expert group standards from expert and non-expert scorers, respectively, and evaluated the performance of the non-expert group standards by compared with the expert group standard (termed EGS). The results indicated that the highest performance was the non-expert group standard when definite spindles were only considered (termed nEGS-1; F1 score = 0.78 for N2; 0.68 for N3). Second, four automatic spindle detection methods were compared with the EGS. We found that the performance of nEGS-1 versus EGS was higher than that of the four automated methods. Our results also showed positive correlation between the mean F1 score of individual expert in EGS and the F1 score of nEGS-1 versus EGS across 30 segments of stage N2 data (r = 0.61, P < 0.001). Further, we found that six and nine non-experts were needed to manually identify spindles in stages N2 and N3, respectively, while maintaining acceptable performance of nEGS-1 versus EGS (F1 score = 0.79 for N2; 0.64 for N3). In conclusion, this study establishes a detailed process for detection of sleep spindles by non-experts in a crowdsourcing scheme.
| S-EPMC5426701 | biostudies-literature
Project description:Sleep spindles are brief bursts of brain activity in the sigma frequency range (11-16 Hz) measured by electroencephalography (EEG) mostly during non-rapid eye movement (NREM) stage 2 sleep. These oscillations are of great biological and clinical interests because they potentially play an important role in identifying and characterizing the processes of various neurological disorders. Conventionally, sleep spindles are identified by expert sleep clinicians via visual inspection of EEG signals. The process is laborious and the results are inconsistent among different experts. To resolve the problem, numerous computerized methods have been developed to automate the process of sleep spindle identification. Still, the performance of these automated sleep spindle detection methods varies inconsistently from study to study. There are two reasons: (1) the lack of common benchmark databases, and (2) the lack of commonly accepted evaluation metrics. In this study, we focus on tackling the second problem by proposing to evaluate the performance of a spindle detector in a multi-objective optimization context and hypothesize that using the resultant Pareto fronts for deriving evaluation metrics will improve automatic sleep spindle detection. We use a popular multi-objective evolutionary algorithm (MOEA), the Strength Pareto Evolutionary Algorithm (SPEA2), to optimize six existing frequency-based sleep spindle detection algorithms. They include three Fourier, one continuous wavelet transform (CWT), and two Hilbert-Huang transform (HHT) based algorithms. We also explore three hybrid approaches. Trained and tested on open-access DREAMS and MASS databases, two new hybrid methods of combining Fourier with HHT algorithms show significant performance improvement with F1-scores of 0.726-0.737.
| S-EPMC5435763 | biostudies-literature
Project description:Spindle event detection is a key component in analyzing human sleep. However, detection of these oscillatory patterns by experts is time consuming and costly. Automated detection algorithms are cost efficient and reproducible but require robust datasets to be trained and validated. Using the MODA (Massive Online Data Annotation) platform, we used crowdsourcing to produce a large open-source dataset of high quality, human-scored sleep spindles (5342 spindles, from 180 subjects). We evaluated the performance of three subtype scorers: "experts, researchers and non-experts", as well as 7 previously published spindle detection algorithms. Our findings show that only two algorithms had performance scores similar to human experts. Furthermore, the human scorers agreed on the average spindle characteristics (density, duration and amplitude), but there were significant age and sex differences (also observed in the set of detected spindles). This study demonstrates how the MODA platform can be used to generate a highly valid open source standardized dataset for researchers to train, validate and compare automated detectors of biological signals such as the EEG.
| S-EPMC7305234 | biostudies-literature
Project description:PurposeTo compare sensitivity in the detection of lung nodules between the deep learning (DL) model and radiologists using various patient population and scanning parameters and to assess whether the radiologists' detection performance could be enhanced when using the DL model for assistance.Materials and methodsA total of 12 754 thin-section chest CT scans from January 2012 to June 2017 were retrospectively collected for DL model training, validation, and testing. Pulmonary nodules from these scans were categorized into four types: solid, subsolid, calcified, and pleural. The testing dataset was divided into three cohorts based on radiation dose, patient age, and CT manufacturer. Detection performance of the DL model was analyzed by using a free-response receiver operating characteristic curve. Sensitivities of the DL model and radiologists were compared by using exploratory data analysis. False-positive detection rates of the DL model were compared within each cohort. Detection performance of the same radiologist with and without the DL model were compared by using nodule-level sensitivity and patient-level localization receiver operating characteristic curves.ResultsThe DL model showed elevated overall sensitivity compared with manual review of pulmonary nodules. No significant dependence regarding radiation dose, patient age range, or CT manufacturer was observed. The sensitivity of the junior radiologist was significantly dependent on patient age. When radiologists used the DL model for assistance, their performance improved and reading time was reduced.ConclusionDL shows promise to enhance the identification of pulmonary nodules and benefit nodule management.© RSNA, 2019Supplemental material is available for this article.
| S-EPMC8017422 | biostudies-literature
Project description:BackgroundSleep spindles are a marker of stage 2 NREM sleep that are linked to learning & memory and are altered by many neurological diseases. Although visual inspection of the EEG is considered the gold standard for spindle detection, it is time-consuming, costly and can introduce inter/ra-scorer bias.New methodOur goal was to develop a simple and efficient sleep-spindle detector (algorithm #7, or 'A7') that emulates human scoring. 'A7' runs on a single EEG channel and relies on four parameters: the absolute sigma power, relative sigma power, and correlation/covariance of the sigma band-passed signal to the original EEG signal. To test the performance of the detector, we compared it against a gold standard spindle dataset derived from the consensus of a group of human experts.ResultsThe by-event performance of the 'A7' spindle detector was 74% precision, 68% recall (sensitivity), and an F1-score of 0.70. This performance was equivalent to an individual human expert (average F1-score = 0.67).Comparison with existing method(s)The F1-score of 'A7' was 0.17 points higher than other spindle detectors tested. Existing detectors have a tendency to find large numbers of false positives compared to human scorers. On a by-subject basis, the spindle density estimates produced by A7 were well correlated with human experts (r2 = 0.82) compared to the existing detectors (average r2 = 0.27).ConclusionsThe 'A7' detector is a sensitive and precise tool designed to emulate human spindle scoring by minimizing the number of 'hidden spindles' detected. We provide an open-source implementation of this detector for further use and testing.
| S-EPMC6415669 | biostudies-literature
Project description:OBJECTIVE Central line-associated bloodstream infection (CLABSI) is associated with significant morbidity and mortality. Despite a nationwide decline in CLABSI rates, individual hospital success in preventing CLABSI is variable. Difficulty in interpreting and applying complex CLABSI metrics may explain this problem. Therefore, we assessed expert interpretation of CLABSI quality data. DESIGN:Cross-sectional survey PARTICIPANTS. Members of the Society for Healthcare Epidemiology of America (SHEA) Research Network (SRN) METHODS. We administered a 10-item test of CLABSI data comprehension. The primary outcome was percent correct of attempted questions pertaining to the CLABSI data. We also assessed expert perceptions of CLABSI reporting. RESULTS The response rate was 51% (n=67).Among experts, the average proportion of correct responses was 73% (95% confidence interval [CI], 69%-77%). Expert performance on unadjusted data was significantly better than risk-adjusted data (86% [95% CI, 81%-90%] vs 65% [95% CI, 60%-70%]; P<.001). Using a scale of 1 to 100 (0, never reliable; 100, always reliable), experts rated the reliability of CLABSI data as 61. Perceived reliability showed a significant inverse relationship with performance (r=-0.28; P=.03), and as interpretation of data improved, perceptions regarding reliability of those data decreased. Experts identified concerns regarding understanding and applying CLABSI definitions as barriers to care. CONCLUSIONS Significant variability in the interpretation of CLABSI data exists among experts. This finding is likely related to data complexity, particularly with respect to risk-adjusted data. Improvements appear necessary in data sharing and public policy efforts to account for this complexity. Infect Control Hosp Epidemiol 2018;39:71-76.
| S-EPMC7181973 | biostudies-literature
Project description:The staircase method has been widely used in measuring perceptual learning. Recently, Zhao, Lesmes, and Lu (2017, 2019) developed the quick Change Detection (qCD) method and applied it to measure the trial-by-trial time course of dark adaptation. In the current study, we conducted two simulations to evaluate the performance of the 3-down/1-up staircase and qCD methods in measuring perceptual learning in a two-alternative forced-choice task. In Study 1, three observers with different time constants (40, 80, and 160 trials) of an exponential learning curve were simulated. Each simulated observer completed staircases with six step sizes (1%, 5%, 10%, 20%, 30%, and 60%) and a qCD procedure, each starting at five levels (+50%, +25%, 0, -25%, and -50% different from the true threshold in the first trial). We found the following results: Staircases with 1% and 5% step sizes failed to generate more than five reversals half of the time; and the bias and standard deviations of thresholds estimated from the post hoc segment-by-segment qCD analysis were much smaller than those from the staircase method with the other four step sizes. In Study 2, we simulated thresholds in the transfer phases with the same time constants and 50% transfer for each observer in Study 1. We found that the estimated transfer indexes from qCD showed smaller biases and standard deviations than those from the staircase method. In addition, rescoring the simulated data from the staircase method using the Bayesian estimation component of the qCD method resulted in much-improved estimates. We conclude that the qCD method characterizes the time course of perceptual learning and transfer more accurately, precisely, and efficiently than the staircase method, even with the optimal 10% step size.
| S-EPMC6645707 | biostudies-literature
Project description:BackgroundNLP models like ChatGPT promise to revolutionize text-based content delivery, particularly in medicine. Yet, doubts remain about ChatGPT's ability to reliably support evaluations of cognitive performance, warranting further investigation into its accuracy and comprehensiveness in this area.MethodA cohort of 60 cognitively normal individuals and 30 stroke survivors underwent a comprehensive evaluation, covering memory, numerical processing, verbal fluency, and abstract thinking. Healthcare professionals and NLP models GPT-3.5 and GPT-4 conducted evaluations following established standards. Scores were compared, and efforts were made to refine scoring protocols and interaction methods to enhance ChatGPT's potential in these evaluations.ResultWithin the cohort of healthy participants, the utilization of GPT-3.5 revealed significant disparities in memory evaluation compared to both physician-led assessments and those conducted utilizing GPT-4 (P < 0.001). Furthermore, within the domain of memory evaluation, GPT-3.5 exhibited discrepancies in 8 out of 21 specific measures when compared to assessments conducted by physicians (P < 0.05). Additionally, GPT-3.5 demonstrated statistically significant deviations from physician assessments in speech evaluation (P = 0.009). Among participants with a history of stroke, GPT-3.5 exhibited differences solely in verbal assessment compared to physician-led evaluations (P = 0.002). Notably, through the implementation of optimized scoring methodologies and refinement of interaction protocols, partial mitigation of these disparities was achieved.ConclusionChatGPT can produce evaluation outcomes comparable to traditional methods. Despite differences from physician evaluations, refinement of scoring algorithms and interaction protocols has improved alignment. ChatGPT performs well even in populations with specific conditions like stroke, suggesting its versatility. GPT-4 yields results closer to physician ratings, indicating potential for further enhancement. These findings highlight ChatGPT's importance as a supplementary tool, offering new avenues for information gathering in medical fields and guiding its ongoing development and application.
| S-EPMC11329975 | biostudies-literature
Project description:AimsEchocardiographic measurement of left ventricular function using a user-friendly automated three-dimensional algorithm is highly attractive as it promises quick and accurate diagnosis, circumventing limitations associated with visual estimation or manual biplane measurements. We sought to assess the feasibility and correlation of such automated analysis with clinically established methods.MethodsA total of 198 patients undergoing transthoracic echocardiography (TTE) with assessment of left ventricular parameters by automated software algorithm (Philips 3D-Heartmodel; 3D-HM) which additionally had either left ventricular angiography (LVA) or cardiac magnetic resonance (CMR) within 24 h of the TTE examination were analyzed. Left ventricular parameters (left ventricular end-diastolic volume, LVEDV, left ventricular end-systolic volume, LVESV as well as left ventricular ejection fraction, LVEF) were compared between 3D-HM, CMR and LVA.ResultsCorrelation of left ventricular measurements was overall good to excellent and stronger for CMR (EF r = 0.824) than for LVA (EF r = 0.746). Unexperienced and expert clinicians yielded comparable good results. For CMR, highest correlation was detected in patients with BMI < 25 and excellent image quality. High agreement was seen between 3D-HM and CMR or LVA when stratifying patients according to heart failure categories.ConclusionsEchocardiographic quantification of left ventricular parameters using a software-based algorithm correlated well with established invasive and non-invasive modalities in the clinical setting, even for unexperienced clinicians. Such automated approaches are promising as they allow a reliable, more observer-independent as well as reproducible assessment of left ventricular function.
| S-EPMC10195710 | biostudies-literature
Project description:The clinical and societal measurement of human sleep has increased exponentially in recent years. However, unlike other fields of medical analysis that have become highly automated, basic and clinical sleep research still relies on human visual scoring. Such human-based evaluations are time-consuming, tedious, and can be prone to subjective bias. Here, we describe a novel algorithm trained and validated on +30,000 hr of polysomnographic sleep recordings across heterogeneous populations around the world. This tool offers high sleep-staging accuracy that matches human scoring accuracy and interscorer agreement no matter the population kind. The software is designed to be especially easy to use, computationally low-demanding, open source, and free. Our hope is that this software facilitates the broad adoption of an industry-standard automated sleep staging software package.
| S-EPMC8516415 | biostudies-literature