Performance of seven consumer sleep-tracking devices compared with polysomnography.
Ontology highlight
ABSTRACT: Study objectives
Consumer sleep-tracking devices are widely used and becoming more technologically advanced, creating strong interest from researchers and clinicians for their possible use as alternatives to standard actigraphy. We, therefore, tested the performance of many of the latest consumer sleep-tracking devices, alongside actigraphy, versus the gold-standard sleep assessment technique, polysomnography (PSG).Methods
In total, 34 healthy young adults (22 women; 28.1 ± 3.9 years, mean ± SD) were tested on three consecutive nights (including a disrupted sleep condition) in a sleep laboratory with PSG, along with actigraphy (Philips Respironics Actiwatch 2) and a subset of consumer sleep-tracking devices. Altogether, four wearable (Fatigue Science Readiband, Fitbit Alta HR, Garmin Fenix 5S, Garmin Vivosmart 3) and three nonwearable (EarlySense Live, ResMed S+, SleepScore Max) devices were tested. Sleep/wake summary and epoch-by-epoch agreement measures were compared with PSG.Results
Most devices (Fatigue Science Readiband, Fitbit Alta HR, EarlySense Live, ResMed S+, SleepScore Max) performed as well as or better than actigraphy on sleep/wake performance measures, while the Garmin devices performed worse. Overall, epoch-by-epoch sensitivity was high (all ≥0.93), specificity was low-to-medium (0.18-0.54), sleep stage comparisons were mixed, and devices tended to perform worse on nights with poorer/disrupted sleep.Conclusions
Consumer sleep-tracking devices exhibited high performance in detecting sleep, and most performed equivalent to (or better than) actigraphy in detecting wake. Device sleep stage assessments were inconsistent. Findings indicate that many newer sleep-tracking devices demonstrate promising performance for tracking sleep and wake. Devices should be tested in different populations and settings to further examine their wider validity and utility.
SUBMITTER: Chinoy ED
PROVIDER: S-EPMC8120339 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:ObjectivesConsumer-grade smart devices are now commonly used by the public to measure waking activity and sleep. However, the ability of these devices to accurately measure sleep in clinical populations warrants more examination. The aim of the present study was to assess the accuracy of three consumer-grade sleep monitors compared with gold standard polysomnography (PSG).DesignA prospective cohort study was performed.SettingAdults undergoing PSG for investigation of a suspected sleep disorder.Participants54 sleep-clinic patients were assessed using three consumer-grade sleep monitors (Jawbone UP3, ResMed S+ and Beddit) in addition to PSG.OutcomesJawbone UP3, ResMed S+ and Beddit were compared with gold standard in-laboratory PSG on four major sleep parameters-total sleep time (TST), sleep onset latency (SOL), wake after sleep onset (WASO) and sleep efficiency (SE).ResultsThe accelerometer Jawbone UP3 was found to overestimate TST by 28 min (limits of agreement, LOA=-100.23 to 157.37), with reasonable agreement compared with gold standard for TST, WASO and SE. The doppler radar ResMed S+ device underestimated TST by 34 min (LOA=-257.06 to 188.34) and had poor absolute agreement compared with PSG for TST, SOL and SE. The mattress device, Beddit underestimated TST by 53 min (LOA=-238.79 to 132) on average and poor reliability compared with PSG for all measures except TST. High device synchronisation failure occurred, with 20% of recordings incomplete due to Bluetooth drop out and recording loss.ConclusionPoor to moderate agreement was found between PSG and each of the tested devices, however, Jawbone UP3 had relatively better absolute agreement than other devices in sleep measurements compared with PSG. Consumer grade devices assessed do not have strong enough agreement with gold standard measurement to replace clinical evaluation and PSG sleep testing. The models tested here have been superseded and newer models may have increase accuracy and thus potentially powerful patient engagement tools for long-term sleep measurement.
| S-EPMC8578969 | biostudies-literature
Project description:Comparisons of actigraphy findings between studies are challenging given differences between brand-specific algorithms. This issue may be minimized by using open-source algorithms. However, the accuracy of actigraphy-derived sleep parameters processed in open-source software needs to be assessed against polysomnography (PSG). Middle-aged adults from the Raine Study (n = 835; F 58%; Age 56.7 ± 5.6 years) completed one night of in-laboratory PSG and concurrent actigraphy (GT3X+ ActiGraph). Actigraphic measures of total sleep time (TST) were analyzed and processed using the open-source R-package GENEActiv and GENEA data in R (GGIR) with and without a sleep diary and additionally processed using proprietary software, ActiLife, for comparison. Bias and agreement (intraclass correlation coefficient) between actigraphy and PSG were examined. Common PSG and sleep health variables associated with the discrepancy between actigraphy, and PSG TST were examined using linear regression. Actigraphy, assessed in GGIR, with and without a sleep diary overestimated PSG TST by (mean ± SD) 31.0 ± 50.0 and 26.4 ± 69.0 minutes, respectively. This overestimation was greater (46.8 ± 50.4 minutes) when actigraphy was analyzed in ActiLife. Agreement between actigraphy and PSG TST was poor (ICC = 0.27-0.44) across all three methods of actigraphy analysis. Longer sleep onset latency and longer wakefulness after sleep onset were associated with overestimation of PSG TST. Open-source processing of actigraphy in a middle-aged community population, agreed poorly with PSG and, on average, overestimated TST. TST overestimation increased with increasing wakefulness overnight. Processing of actigraphy without a diary in GGIR was comparable to when a sleep diary was used and comparable to actigraphy processed with proprietary algorithms in ActiLife.
| S-EPMC10362889 | biostudies-literature
Project description:Study objectivesTo compare the quality and consistency in sleep measurement of a consumer wearable device and a research-grade actigraph with polysomnography (PSG) in adolescents.MethodsFifty-eight healthy adolescents (aged 15-19 years; 30 males) underwent overnight PSG while wearing both a Fitbit Alta HR and a Philips Respironics Actiwatch 2 (AW2) for 5 nights, with either 5 hours or 6.5 hours time in bed (TIB) and for 4 nights with 9 hours TIB. AW2 data were evaluated using two different wake and immobility thresholds. Discrepancies in estimated total sleep time (TST) and wake after sleep onset (WASO) between devices and PSG, as well as epoch-by-epoch agreements in sleep/wake classification, were assessed. Fitbit-generated sleep staging was compared to PSG.ResultsFitbit and AW2 under default settings similarly underestimated TST and overestimated WASO (TST: medium setting (M10) ≤ 38 minutes, Fitbit ≤ 47 minutes; WASO: M10 ≤ 38 minutes; Fitbit ≤ 42 minutes). AW2 at the high motion threshold setting provided readings closest to PSG (TST: ≤ 12 minutes; WASO: ≤ 18 minutes). Sensitivity for detecting sleep was ≥ 90% for both wearable devices and further improved to 95% by using the high threshold (H5) setting for the AW2 (0.95). Wake detection specificity was highest in Fitbit (≥ 0.88), followed by the AW2 at M10 (≥ 0.80) and H5 thresholds (≤ 0.73). In addition, Fitbit inconsistently estimated stage N1 + N2 sleep depending on TIB, underestimated stage N3 sleep (21-46 min), but was comparable to PSG for rapid eye movement sleep. Fitbit sensitivity values for the detection of N1 + N2, N3 and rapid eye movement sleep were ≥ 0.68, ≥ 0.50, and ≥ 0.72, respectively.ConclusionsA consumer-grade wearable device can measure sleep duration as well as a research actigraph. However, sleep staging would benefit from further refinement before these methods can be reliably used for adolescents.Clinical trial registrationRegistry: ClinicalTrials.gov; Title: The Cognitive and Metabolic Effects of Sleep Restriction in Adolescents; Identifier: NCT03333512; URL: https://clinicaltrials.gov/ct2/show/NCT03333512.CitationLee XK, Chee NIYN, Ong JL, Teo TB, van Rijn E, Lo JC, Chee MWL. Validation of a consumer sleep wearable device with actigraphy and polysomnography in adolescents across sleep opportunity manipulations. J Clin Sleep Med. 2019;15(9):1337-1346.
| S-EPMC6760396 | biostudies-literature
Project description:We validated actigraphy for detecting sleep and wakefulness versus polysomnography (PSG).Actigraphy and polysomnography were simultaneously collected during sleep laboratory admissions. All studies involved 8.5 h time in bed, except for sleep restriction studies. Epochs (30-sec; n = 232,849) were characterized for sensitivity (actigraphy = sleep when PSG = sleep), specificity (actigraphy = wake when PSG = wake), and accuracy (total proportion correct); the amount of wakefulness after sleep onset (WASO) was also assessed. A generalized estimating equation (GEE) model included age, gender, insomnia diagnosis, and daytime/nighttime sleep timing factors.Controlled sleep laboratory conditions.Young and older adults, healthy or chronic primary insomniac (PI) patients, and daytime sleep of 23 night-workers (n = 77, age 35.0 ± 12.5, 30F, mean nights = 3.2).N/A.Overall, sensitivity (0.965) and accuracy (0.863) were high, whereas specificity (0.329) was low; each was only slightly modified by gender, insomnia, day/night sleep timing (magnitude of change < 0.04). Increasing age slightly reduced specificity. Mean WASO/night was 49.1 min by PSG compared to 36.8 min/night by actigraphy (? = 0.81; CI = 0.42, 1.21), unbiased when WASO < 30 min/night, and overestimated when WASO > 30 min/night.This validation quantifies strengths and weaknesses of actigraphy as a tool measuring sleep in clinical and population studies. Overall, the participant-specific accuracy is relatively high, and for most participants, above 80%. We validate this finding across multiple nights and a variety of adults across much of the young to midlife years, in both men and women, in those with and without insomnia, and in 77 participants. We conclude that actigraphy is overall a useful and valid means for estimating total sleep time and wakefulness after sleep onset in field and workplace studies, with some limitations in specificity.
| S-EPMC3792393 | biostudies-literature
Project description:The development of ambulatory technologies capable of monitoring brain activity during sleep longitudinally is critical for advancing sleep science. The aim of this study was to assess the signal acquisition and the performance of the automatic sleep staging algorithms of a reduced-montage dry-electroencephalographic (EEG) device (Dreem headband, DH) compared to the gold-standard polysomnography (PSG) scored by five sleep experts. A total of 25 subjects who completed an overnight sleep study at a sleep center while wearing both a PSG and the DH simultaneously have been included in the analysis. We assessed (1) similarity of measured EEG brain waves between the DH and the PSG; (2) the heart rate, breathing frequency, and respiration rate variability (RRV) agreement between the DH and the PSG; and (3) the performance of the DH's automatic sleep staging according to American Academy of Sleep Medicine guidelines versus PSG sleep experts manual scoring. The mean percentage error between the EEG signals acquired by the DH and those from the PSG for the monitoring of α was 15 ± 3.5%, 16 ± 4.3% for β, 16 ± 6.1% for λ, and 10 ± 1.4% for θ frequencies during sleep. The mean absolute error for heart rate, breathing frequency, and RRV was 1.2 ± 0.5 bpm, 0.3 ± 0.2 cpm, and 3.2 ± 0.6%, respectively. Automatic sleep staging reached an overall accuracy of 83.5 ± 6.4% (F1 score: 83.8 ± 6.3) for the DH to be compared with an average of 86.4 ± 8.0% (F1 score: 86.3 ± 7.4) for the 5 sleep experts. These results demonstrate the capacity of the DH to both monitor sleep-related physiological signals and process them accurately into sleep stages. This device paves the way for, large-scale, longitudinal sleep studies. NCT03725943.
| S-EPMC7751170 | biostudies-literature
Project description:Actigraphy (ACT) is a non-invasive objective assessment tool for the study of sleep-wake rhythms. It is of particular interest in children with autism spectrum disorder (ASD), as sleep disorders are highly prevalent and have a significant impact on both cognitive and behavioral functions. As polysomnography (PSG), the gold standard for the assessment of sleep, is difficult to perform in children with ASD, ACT has become a tool of choice but has not yet been validated against PSG using state-of-the-art methodology. The main objective of this study was to assess, for the first time, the validity of ACT compared to PSG for the measurement of sleep in children with ASD. During the same night of hospitalization, PSG and ACT were conducted in 26 children (6 girls and 20 boys; mean age 5.4 years ± 1.6) diagnosed with ASD according to DSM-5 criteria and standardized diagnostic scales. Sleep parameters were total sleep time (TST), sleep latency (SL), wake after sleep onset (WASO), and sleep efficiency (SE). To compare PSG and ACT, we conducted sleep parameter agreement analyses including: intraclass correlation coefficient (ICC), Bland-Altman plots, and equivalence tests. The comparison also included an epoch-by-epoch (EBE) agreement analysis to determine sensitivity (ability to detect sleep) and specificity (ability to detect wake). According to equivalence tests, the difference between ACT and PSG measures was clinically acceptable for TST (<30 min, p < 0.01), SL (<15 min, p < 0.001), and SE (10%, p < 0.01), but not for WASO (<15 min, p = 0.13). There was a good agreement between methods for SL (ICC = 0.79) and TST (ICC = 0.85) and a moderate agreement for WASO (ICC = 0.73) and SE (ICC = 0.68). The EBE agreement analysis revealed a high sensitivity (0.94 ± 0.06) and moderate specificity (0.5 ± 0.2). Since sleep disorders are one of the most common comorbidities within the ASD population and are highly prevalent, it is essential to validate objective tools of assessment. To our knowledge, our study is the first to validate ACT compared to PSG, using a state-of-the-art methodology, in children with ASD. The results suggest ACT to be a valid method to evaluate sleep within this population, with a good reliability for most sleep parameters.
| S-EPMC6688709 | biostudies-literature
Project description:Commercial sleep devices and mobile-phone applications for scoring sleep are gaining ground. In order to provide reliable information about the quantity and/or quality of sleep, their performance needs to be assessed against the current gold standard, i.e., polysomnography (PSG; measuring brain, eye, and muscle activity). Here, we assessed some commercially available sleep trackers, namely an activity tracker; Mi band (Xiaomi, Beijing, China), a scientific actigraph: Motionwatch 8 (CamNTech, Cambridge, UK), and a much-used mobile phone application: Sleep Cycle (Northcube, Gothenburg, Sweden). We recorded 27 nights in healthy sleepers using PSG and these devices and compared the results. Surprisingly, all devices had poor agreement with the PSG gold standard. Sleep parameter comparisons revealed that, specifically, the Mi band and the Sleep Cycle application had difficulties in detecting wake periods which negatively affected their total sleep time and sleep-efficiency estimations. However, all 3 devices were good in detecting the most basic parameter, the actual time in bed. In summary, our results suggest that, to date, the available sleep trackers do not provide meaningful sleep analysis but may be interesting for simply tracking time in bed. A much closer interaction with the scientific field seems necessary if reliable information shall be derived from such devices in the future.
| S-EPMC6806072 | biostudies-literature
Project description:There has been a proliferation of inexpensive consumer-grade devices for monitoring air pollutants, including PM2.5 and certain gasses. This study compared the performance of four consumer-grade devices-the Air Quality Egg 2 (AQE2), BlueAir Aware, Foobot, and Speck-that utilize optical sensors to measure the PM2.5 concentration. The devices were collocated and operated for 7 days in each of three residences, and the PM2.5 mass concentrations were compared with those measured by established optical sensing devices, viz., the personal DataRAM and DustTrak DRX, as well as the filter-based Personal Modular Impactor (PMI). Overall, the Foobot and BlueAir displayed the strongest correlations with the direct-reading reference instruments for both the hourly and daily PM2.5 mass concentrations. Comparing the 1-hour averages obtained with the DustTrak DRX for all of the residences with those obtained with the Foobot, BlueAir, AQE2, and Speck, the Pearson's correlation coefficients (R's) were 0.80, 0.88, -0.028, and 0.60, respectively. Overall, the strength of the correlation depended on the specific residence, likely due to the differences in aerosol composition. The correlations with the PMI measurements were moderate, with R values of 0.44 and 0.56 for the BlueAir and Foobot, respectively. The correlation coefficients for the daily values obtained with the AQE2 and Speck were -0.59 and 0.70 compared to the PMI. According to a paired t-test, the average 24-h PM2.5 concentration data obtained using the consumer-grade monitors were statistically different (p > 0.05) from the mass values measured by the gravimetric filters. Overall, this study demonstrates the ability of consumer grade air pollution monitors to report PM2.5 trends accurately; however, for accurate mass concentration measurements, these monitors must be calibrated for a particular location and application. Further testing is needed to determine their suitability for long-term indoor field studies.
| S-EPMC7654960 | biostudies-literature
Project description:We evaluated the performance of Fitbit Charge 3™ (FC3), a multi-sensor commercial sleep-tracker, for measuring sleep in adolescents against gold-standard laboratory polysomnography (PSG). Single-night PSG and FC3 sleep outcomes were compared in thirty-nine adolescents (22 girls; 16-19 years), 12 of whom presented with clinical/subclinical DSM-5 insomnia symptoms (7 girls). Discrepancy analysis, Bland-Altman plots, and epoch-by-epoch analyses were used to evaluate FC3 performance. The influence of several factors potentially affecting FC3 performance (e.g., sex, age, body mass index, firmware version, and magnitude of heart rate changes between consecutive PSG epochs) was also tested. In the sample of healthy adolescents, FC3 systematically underestimated PSG total sleep time by about 11 min and sleep efficiency by 2.5%, and overestimated wake after sleep onset by 9 min. Proportional biases were detected for "light" and "deep" sleep duration, resulting in significant underestimation of these parameters for those participants having longer PSG N1+ N2 and N3 durations, respectively. No significant systematic bias was detected for sleep efficiency and sleep onset latency. Epoch-by-epoch analysis showed sleep-stage sensitivity (average proportion of PSG epochs correctly classified by the device for a given sleep stage) of 68% for wake, 78% for "light" sleep, 59% for "deep" sleep, and 69% for rapid eye movement (REM) sleep in healthy sleepers. Similar results were found in the sample of adolescents with insomnia symptoms. Body mass index was positively associated with FC3-PSG discrepancies in wake after sleep onset (R2 = .16, p = .048). The magnitude of the heart rate acceleration/deceleration between consecutive PSG epochs was an important factor affecting FC3 classifications of sleep stages. Our results are in line with a general trend in the literature, suggesting better performance for the recently introduced multi-sensor devices compared to motion-only devices, although further developments are needed to improve accuracy in sleep stage classification and wake detection. Further insight is needed to determine factors potentially affecting device performance, such as accuracy and reliability (consistency of performance over time), in different samples and conditions.
| S-EPMC8255273 | biostudies-literature
Project description:Electroencephalography (EEG) recordings represent a vital component of the assessment of sleep physiology, but the methodology presently used is costly, intrusive to participants, and laborious in application. There is a recognized need to develop more easily applicable yet reliable EEG systems that allow unobtrusive long-term recording of sleep-wake EEG ideally away from the laboratory setting. cEEGrid is a recently developed flex-printed around-the-ear electrode array, which holds great potential for sleep-wake monitoring research. It is comfortable to wear, simple to apply, and minimally intrusive during sleep. Moreover, it can be combined with a smartphone-controlled miniaturized amplifier and is fully portable. Evaluation of cEEGrid as a motion-tolerant device is ongoing, but initial findings clearly indicate that it is very well suited for cognitive research. The present study aimed to explore the suitability of cEEGrid for sleep research, by testing whether cEEGrid data affords the signal quality and characteristics necessary for sleep stage scoring. In an accredited sleep laboratory, sleep data from cEEGrid and a standard PSG system were acquired simultaneously. Twenty participants were recorded for one extended nocturnal sleep opportunity. Fifteen data sets were scored manually. Sleep parameters relating to sleep maintenance and sleep architecture were then extracted and statistically assessed for signal quality and concordance. The findings suggest that the cEEGrid system is a viable and robust recording tool to capture sleep and wake EEG. Further research is needed to fully determine the suitability of cEEGrid for basic and applied research as well as sleep medicine.
| S-EPMC6276915 | biostudies-literature