Reproducibility and replicability of high-frequency, in-home digital biomarkers in reducing sample sizes for clinical trials.
Ontology highlight
ABSTRACT: Introduction
Reproducibility and replicability of results are rarely achieved for digital biomarkers analyses. We reproduced and replicated previously reported sample size estimates based on digital biomarker and neuropsychological test outcomes in a hypothetical 4-year early-phase Alzheimer's disease trial.Methods
Original data and newly collected data (using a different motion sensor) came from the Oregon Center for Aging & Technology (ORCATECH). Given trajectories of those with incident mild cognitive impairment and normal cognition would represent trajectories of the control and experimental groups in a hypothetical trial, sample sizes to provide 80% power to detect effect sizes ranging from 20% to 50% were calculated.Results
For the reproducibility, identical P-values and slope estimates were found with both digital biomarkers and neuropsychological test measures between the previous and current studies. As for the replicability, a greater correlation was found between original and replicated sample size estimates for digital biomarkers (r = 0.87, P < .001) than neuropsychological test outcomes (r = 0.75, P < .001).Discussion
Reproducibility and replicability of digital biomarker analyses are feasible and encouraged to establish the reliability of findings.
SUBMITTER: Wu CY
PROVIDER: S-EPMC8719347 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BackgroundTrials in Alzheimer's disease are increasingly focusing on prevention in asymptomatic individuals. This poses a challenge in examining treatment effects since currently available approaches are often unable to detect cognitive and functional changes among asymptomatic individuals. Resultant small effect sizes require large sample sizes using biomarkers or secondary measures for randomized controlled trials (RCTs). Better assessment approaches and outcomes capable of capturing subtle changes during asymptomatic disease stages are needed.ObjectiveWe aimed to develop a new approach to track changes in functional outcomes by using individual-specific distributions (as opposed to group-norms) of unobtrusive continuously monitored in-home data. Our objective was to compare sample sizes required to achieve sufficient power to detect prevention trial effects in trajectories of outcomes in two scenarios: (1) annually assessed neuropsychological test scores (a conventional approach), and (2) the likelihood of having subject-specific low performance thresholds, both modeled as a function of time.MethodsOne hundred nineteen cognitively intact subjects were enrolled and followed over 3 years in the Intelligent Systems for Assessing Aging Change (ISAAC) study. Using the difference in empirically identified time slopes between those who remained cognitively intact during follow-up (normal control, NC) and those who transitioned to mild cognitive impairment (MCI), we estimated comparative sample sizes required to achieve up to 80% statistical power over a range of effect sizes for detecting reductions in the difference in time slopes between NC and MCI incidence before transition.ResultsSample size estimates indicated approximately 2000 subjects with a follow-up duration of 4 years would be needed to achieve a 30% effect size when the outcome is an annually assessed memory test score. When the outcome is likelihood of low walking speed defined using the individual-specific distributions of walking speed collected at baseline, 262 subjects are required. Similarly for computer use, 26 subjects are required.ConclusionsIndividual-specific thresholds of low functional performance based on high-frequency in-home monitoring data distinguish trajectories of MCI from NC and could substantially reduce sample sizes needed in dementia prevention RCTs.
| S-EPMC4574479 | biostudies-literature
Project description:Despite a growing body of research suggesting that task-based functional magnetic resonance imaging (fMRI) studies often suffer from a lack of statistical power due to too-small samples, the proliferation of such underpowered studies continues unabated. Using large independent samples across eleven tasks, we demonstrate the impact of sample size on replicability, assessed at different levels of analysis relevant to fMRI researchers. We find that the degree of replicability for typical sample sizes is modest and that sample sizes much larger than typical (e.g., N = 100) produce results that fall well short of perfectly replicable. Thus, our results join the existing line of work advocating for larger sample sizes. Moreover, because we test sample sizes over a fairly large range and use intuitive metrics of replicability, our hope is that our results are more understandable and convincing to researchers who may have found previous results advocating for larger samples inaccessible.
| S-EPMC6123695 | biostudies-other
Project description:OBJECTIVE:The identification of sensitive biomarkers is essential to validate therapeutics for Huntington disease (HD). We directly compare structural imaging markers across the largest collective imaging HD dataset to identify a set of imaging markers robust to multicenter variation and to derive upper estimates on sample sizes for clinical trials in HD. METHODS:We used 1 postprocessing pipeline to retrospectively analyze T1-weighted magnetic resonance imaging (MRI) scans from 624 participants at 3 time points, from the PREDICT-HD, TRACK-HD, and IMAGE-HD studies. We used mixed effects models to adjust regional brain volumes for covariates, calculate effect sizes, and simulate possible treatment effects in disease-affected anatomical regions. We used our model to estimate the statistical power of possible treatment effects for anatomical regions and clinical markers. RESULTS:We identified a set of common anatomical regions that have similarly large standardized effect sizes (>0.5) between healthy control and premanifest HD (PreHD) groups. These included subcortical, white matter, and cortical regions and nonventricular cerebrospinal fluid (CSF). We also observed a consistent spatial distribution of effect size by region across the whole brain. We found that multicenter studies were necessary to capture treatment effect variance; for a 20% treatment effect, power of >80% was achieved for the caudate (n?=?661), pallidum (n?=?687), and nonventricular CSF (n?=?939), and, crucially, these imaging markers provided greater power than standard clinical markers. INTERPRETATION:Our findings provide the first cross-study validation of structural imaging markers in HD, supporting the use of these measurements as endpoints for both observational studies and clinical trials. ANN NEUROL 2020;87:751-762.
| S-EPMC7187160 | biostudies-literature
Project description:This paper espouses a view of reproducibility in the computational sciences as a process and not just a point-in-time “achievement”. As a concrete case study, we revisit the Open-Source IR Reproducibility Challenge from 2015 and attempt to replicate those experiments: four years later, are those computational artifacts still functional? Perhaps not surprisingly, we are not able to replicate most of the retrieval runs encapsulated by those artifacts in a modern computational environment. We outline the various idiosyncratic reasons why, distilled into a series of “lessons learned” to help form an emerging set of best practices for the long-term sustainability of reproducibility efforts.
| S-EPMC7148033 | biostudies-literature
Project description:The scientific community is increasingly concerned with the proportion of published "discoveries" that are not replicated in subsequent studies. The field of rodent behavioral phenotyping was one of the first to raise this concern, and to relate it to other methodological issues: the complex interaction between genotype and environment; the definitions of behavioral constructs; and the use of laboratory mice and rats as model species for investigating human health and disease mechanisms. In January 2015, researchers from various disciplines gathered at Tel Aviv University to discuss these issues. The general consensus was that the issue is prevalent and of concern, and should be addressed at the statistical, methodological and policy levels, but is not so severe as to call into question the validity and the usefulness of model organisms as a whole. Well-organized community efforts, coupled with improved data and metadata sharing, have a key role in identifying specific problems and promoting effective solutions. Replicability is closely related to validity, may affect generalizability and translation of findings, and has important ethical implications.
| S-EPMC6071910 | biostudies-other
Project description:To identify robust and reproducible methods of cerebellar morphometry that can be used in future large-scale structural MRI studies, we investigated the replicability, repeatability, and long-term reproducibility of three fully automated software tools: FreeSurfer, CEREbellum Segmentation (CERES), and automatic cerebellum anatomical parcellation using U-Net with locally constrained optimization (ACAPULCO). Replicability was defined as computational replicability, determined by comparing two analyses of the same high-resolution MRI data set performed with identical analysis software and computer hardware. Repeatability was determined by comparing the analyses of two MRI scans of the same participant taken during two independent MRI sessions on the same day for the Kirby-21 study. Long-term reproducibility was assessed by analyzing two MRI scans of the same participant in the longitudinal OASIS-2 study. We determined percent difference, the image intraclass correlation coefficient, the coefficient of variation, and the intraclass correlation coefficient between two analyses. Our results show that CERES and ACAPULCO use stochastic algorithms that result in surprisingly high differences between identical analyses for ACAPULCO and small differences for CERES. Changes between two consecutive scans from the Kirby-21 study were less than ± 5% in most cases for FreeSurfer and CERES (i.e., demonstrating high repeatability). As expected, long-term reproducibility was lower than repeatability for all software tools. In summary, CERES is an accurate, as demonstrated before, and reproducible tool for fully automated segmentation and parcellation of the cerebellum. We conclude with recommendations for the assessment of replicability, repeatability, and long-term reproducibility in future studies on cerebellar structure.
| S-EPMC8213608 | biostudies-literature
Project description:Cerebral atrophy rate is increasingly used as an outcome measure for Alzheimer's disease (AD) trials. We used the Alzheimer's disease Neuroimaging initiative (ADNI) dataset to assess if adjusting for baseline characteristics can reduce sample sizes. Controls (n = 199), patients with mild cognitive impairment (MCI) (n = 334) and AD (n = 144) had two MRI scans, 1-year apart; approximately 55% had baseline CSF tau, p-tau, and Abeta1-42. Whole brain (KN-BSI) and hippocampal (HMAPS-HBSI) atrophy rate, and ventricular expansion (VBSI) were calculated for each group; numbers required to power a placebo-controlled trial were estimated. Sample sizes per arm (80% power, 25% absolute rate reduction) for AD were (95% CI): brain atrophy = 81 (64,109), hippocampal atrophy = 88 (68,119), ventricular expansion = 118 (92,157); and for MCI: brain atrophy = 149 (122,188), hippocampal atrophy = 201 (160,262), ventricular expansion = 234 (191,295). To detect a 25% reduction relative to normal aging required increased sample sizes approximately 3-fold (AD), and approximately 5-fold (MCI). Disease severity and Abeta1-42 contributed significantly to atrophy rate variability. Adjusting for 11 predefined covariates reduced sample sizes by up to 30%. Treatment trials in AD should consider the effects of normal aging; adjusting for baseline characteristics can significantly reduce required sample sizes.
| S-EPMC2947486 | biostudies-literature
Project description:BackgroundProgressive brain atrophy in multiple sclerosis (MS) may reflect neuroaxonal and myelin loss and MRI measures of brain tissue loss are used as outcome measures in MS treatment trials. This study investigated sample sizes required to demonstrate reduction of brain atrophy using three outcome measures in a parallel group, placebo-controlled trial for secondary progressive MS (SPMS).MethodsData were taken from a cohort of 43 patients with SPMS who had been followed up with 6-monthly T1-weighted MRI for up to 3 years within the placebo arm of a therapeutic trial. Central cerebral volumes (CCVs) were measured using a semiautomated segmentation approach, and brain volume normalized for skull size (NBV) was measured using automated segmentation (SIENAX). Change in CCV and NBV was measured by subtraction of baseline from serial CCV and SIENAX images; in addition, percentage brain volume change relative to baseline was measured directly using a registration-based method (SIENA). Sample sizes for given treatment effects and power were calculated for standard analyses using parameters estimated from the sample.ResultsFor a 2-year trial duration, minimum sample sizes per arm required to detect a 50% treatment effect at 80% power were 32 for SIENA, 69 for CCV, and 273 for SIENAX. Two-year minimum sample sizes were smaller than 1-year by 71% for SIENAX, 55% for CCV, and 44% for SIENA.ConclusionSIENA and central cerebral volume are feasible outcome measures for inclusion in placebo-controlled trials in secondary progressive multiple sclerosis.
| S-EPMC2818185 | biostudies-literature
Project description:Concerns regarding reproducibility of resting-state functional magnetic resonance imaging (R-fMRI) findings have been raised. Little is known about how to operationally define R-fMRI reproducibility and to what extent it is affected by multiple comparison correction strategies and sample size. We comprehensively assessed two aspects of reproducibility, test-retest reliability and replicability, on widely used R-fMRI metrics in both between-subject contrasts of sex differences and within-subject comparisons of eyes-open and eyes-closed (EOEC) conditions. We noted permutation test with Threshold-Free Cluster Enhancement (TFCE), a strict multiple comparison correction strategy, reached the best balance between family-wise error rate (under 5%) and test-retest reliability/replicability (e.g., 0.68 for test-retest reliability and 0.25 for replicability of amplitude of low-frequency fluctuations (ALFF) for between-subject sex differences, 0.49 for replicability of ALFF for within-subject EOEC differences). Although R-fMRI indices attained moderate reliabilities, they replicated poorly in distinct datasets (replicability?<?0.3 for between-subject sex differences,?<?0.5 for within-subject EOEC differences). By randomly drawing different sample sizes from a single site, we found reliability, sensitivity and positive predictive value (PPV) rose as sample size increased. Small sample sizes (e.g.,?<?80 [40 per group]) not only minimized power (sensitivity?<?2%), but also decreased the likelihood that significant results reflect "true" effects (PPV?<?0.26) in sex differences. Our findings have implications for how to select multiple comparison correction strategies and highlight the importance of sufficiently large sample sizes in R-fMRI studies to enhance reproducibility. Hum Brain Mapp 39:300-318, 2018. © 2017 Wiley Periodicals, Inc.
| S-EPMC6866539 | biostudies-literature
Project description:This study modeled predementia Alzheimer's disease clinical trials. Longitudinal data from cognitively normal (CN) and mild cognitive impairment (MCI) participants in the Alzheimer's Disease Neuroimaging Initiative were used to calculate sample size requirements for trials using outcome measures, including the Clinical Dementia Rating scale sum of boxes, Mini-Mental State Examination, Alzheimer's Disease Assessment Scale-cognitive subscale with and without delayed recall, and the Rey Auditory Verbal Learning Task. We examined the impact on sample sizes of enrichment for genetic and biomarker criteria, including cerebrospinal fluid protein and neuroimaging analyses. We observed little cognitive decline in the CN population at 36 months, regardless of the enrichment strategy. Nonetheless, in CN subjects, using Rey Auditory Verbal Learning Task total as an outcome at 36 months required the fewest subjects across enrichment strategies, with apolipoprotein E genotype ?4 carrier status requiring the fewest (n = 499 per arm to demonstrate a 25% reduction in disease progression). In MCI, enrichment reduced the required sample sizes for trials, relative to estimates based on all subjects. For MCI, the Clinical Dementia Rating scale sum of boxes consistently required the smallest sample sizes. We conclude that predementia clinical trial conduct in Alzheimer's disease is enhanced by the use of biomarker inclusion criteria.
| S-EPMC3412892 | biostudies-literature