Enhancing reproducibility in developmental EEG research: BIDS, cluster-based permutation tests, and effect sizes
Ontology highlight
ABSTRACT:
SUBMITTER: Meyer M
PROVIDER: S-EPMC8607163 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Permutation tests are useful in stepped-wedge trials to provide robust statistical tests of intervention-effect estimates. However, the Stata command permute does not produce valid tests in this setting because individual observations are not exchangeable. We introduce the swpermute command that permutes clusters to sequences to maintain exchangeability. The command provides additional functionality to aid users in performing analyses of stepped-wedge trials. In particular, we include the option "withinperiod" that performs the specified analysis separately in each period of the study with the resulting period-specific intervention-effect estimates combined as a weighted average. We also include functionality to test non-zero null hypotheses to aid the construction of confidence intervals. Examples of the application of swpermute are given using data from a trial testing the impact of a new tuberculosis diagnostic test on bacterial confirmation of a tuberculosis diagnosis.
| S-EPMC7305031 | biostudies-literature
Project description:MotivationPermutation tests offer a straightforward framework to assess the significance of differences in sample statistics. A significant advantage of permutation tests are the relatively few assumptions about the distribution of the test statistic are needed, as they rely on the assumption of exchangeability of the group labels. They have great value, as they allow a sensitivity analysis to determine the extent to which the assumed broad sample distribution of the test statistic applies. However, in this situation, permutation tests are rarely applied because the running time of naïve implementations is too slow and grows exponentially with the sample size. Nevertheless, continued development in the 1980s introduced dynamic programming algorithms that compute exact permutation tests in polynomial time. Albeit this significant running time reduction, the exact test has not yet become one of the predominant statistical tests for medium sample size. Here, we propose a computational parallelization of one such dynamic programming-based permutation test, the Green algorithm, which makes the permutation test more attractive.ResultsParallelization of the Green algorithm was found possible by non-trivial rearrangement of the structure of the algorithm. A speed-up-by orders of magnitude-is achievable by executing the parallelized algorithm on a GPU. We demonstrate that the execution time essentially becomes a non-issue for sample sizes, even as high as hundreds of samples. This improvement makes our method an attractive alternative to, e.g. the widely used asymptotic Mann-Whitney U-test.Availabilityand implementationIn Python 3 code from the GitHub repository https://github.com/statisticalbiotechnology/parallelPermutationTest under an Apache 2.0 license.Supplementary informationSupplementary data are available at Bioinformatics online.
| S-EPMC8016463 | biostudies-literature
Project description:Truncated survival data arise when the event time is observed only if it falls within a subject-specific region, known as the truncation set. Left-truncated data arise when there is delayed entry into a study, such that subjects are included only if their event time exceeds some other time. Quasi-independence of truncation and failure refers to factorization of their joint density in the observable region. Under quasi-independence, standard methods for survival data such as the Kaplan-Meier estimator and Cox regression can be applied after simple adjustments to the risk sets. Unlike the requisite assumption of independent censoring, quasi-independence can be tested, e.g., using a conditional Kendall's tau test. Current methods for testing for quasi-independence are powerful for monotone alternatives. Nonetheless, it is essential to detect any kind of deviation from quasi-independence so as not to report a biased Kaplan-Meier estimator or regression effect, which would arise from applying the simple risk set adjustment when dependence holds. Nonparametric, minimum p-value tests that are powerful against non-monotone alternatives are developed to offer protection against erroneous assumptions of quasi-independence. The use of conditional and unconditional methods of permutation for evaluation of the proposed tests are investigated in simulation studies. The proposed tests are applied to a study on the cognitive and functional decline in aging.
| S-EPMC6317381 | biostudies-literature
Project description:BackgroundTime series analysis is critical for understanding brain signals and their relationship to behavior and cognition. Cluster-based permutation tests (CBPT) are commonly used to analyze a variety of electrophysiological signals including EEG, MEG, ECoG, and sEEG data without a priori assumptions about specific temporal effects. However, two major limitations of CBPT include the inability to directly analyze experiments with multiple fixed effects and the inability to account for random effects (e.g. variability across subjects). Here, we propose a flexible multi-step hypothesis testing strategy using CBPT with Linear Mixed Effects Models (LMEs) and Generalized Linear Mixed Effects Models (GLMEs) that can be applied to a wide range of experimental designs and data types.MethodsWe first evaluate the statistical robustness of LMEs and GLMEs using simulated data distributions. Second, we apply a multi-step hypothesis testing strategy to analyze ERPs and broadband power signals extracted from human ECoG recordings collected during a simple image viewing experiment with image category and novelty as fixed effects. Third, we assess the statistical power differences between analyzing signals with CBPT using LMEs compared to CBPT using separate t-tests run on each fixed effect through simulations that emulate broadband power signals. Finally, we apply CBPT using GLMEs to high-gamma burst data to demonstrate the extension of the proposed method to the analysis of nonlinear data.ResultsFirst, we found that LMEs and GLMEs are robust statistical models. In simple simulations LMEs produced highly congruent results with other appropriately applied linear statistical models, but LMEs outperformed many linear statistical models in the analysis of "suboptimal" data and maintained power better than analyzing individual fixed effects with separate t-tests. GLMEs also performed similarly to other nonlinear statistical models. Second, in real world human ECoG data, LMEs performed at least as well as separate t-tests when applied to predefined time windows or when used in conjunction with CBPT. Additionally, fixed effects time courses extracted with CBPT using LMEs from group-level models of pseudo-populations replicated latency effects found in individual category-selective channels. Third, analysis of simulated broadband power signals demonstrated that CBPT using LMEs was superior to CBPT using separate t-tests in identifying time windows with significant fixed effects especially for small effect sizes. Lastly, the analysis of high-gamma burst data using CBPT with GLMEs produced results consistent with CBPT using LMEs applied to broadband power data.ConclusionsWe propose a general approach for statistical analysis of electrophysiological data using CBPT in conjunction with LMEs and GLMEs. We demonstrate that this method is robust for experiments with multiple fixed effects and applicable to the analysis of linear and nonlinear data. Our methodology maximizes the statistical power available in a dataset across multiple experimental variables while accounting for hierarchical random effects and controlling FWER across fixed effects. This approach substantially improves power and accuracy leading to better reproducibility. Additionally, CBPT using LMEs and GLMEs can be used to analyze individual channels or pseudo-population data for the comparison of functional or anatomical groups of data.
| S-EPMC10081325 | biostudies-literature
Project description:BackgroundGenomics and proteomics analyses regularly involve the simultaneous test of hundreds of hypotheses, either on numerical or categorical data. To correct for the occurrence of false positives, validation tests based on multiple testing correction, such as Bonferroni and Benjamini and Hochberg, and re-sampling, such as permutation tests, are frequently used. Despite the known power of permutation-based tests, most available tools offer such tests for either t-test or ANOVA only. Less attention has been given to tests for categorical data, such as the Chi-square. This project takes a first step by developing an open-source software tool, Ptest, that addresses the need to offer public software tools incorporating these and other statistical tests with options for correcting for multiple hypotheses.ResultsThis study developed a public-domain, user-friendly software whose purpose was twofold: first, to estimate test statistics for categorical and numerical data; and second, to validate the significance of the test statistics via Bonferroni, Benjamini and Hochberg, and a permutation test of numerical and categorical data. The tool allows the calculation of Chi-square test for categorical data, and ANOVA test, Bartlett's test and t-test for paired and unpaired data. Once a test statistic is calculated, Bonferroni, Benjamini and Hochberg, and a permutation tests are implemented, independently, to control for Type I errors. An evaluation of the software using different public data sets is reported, which illustrates the power of permutation tests for multiple hypotheses assessment and for controlling the rate of Type I errors.ConclusionThe analytical options offered by the software can be applied to support a significant spectrum of hypothesis testing tasks in functional genomics, using both numerical and categorical data.
| S-EPMC2611984 | biostudies-literature
Project description:The fields of applied and translational microRNA research have exploded in recent years as microRNAs have been implicated across a spectrum of diseases. MicroRNA biomarkers, microRNA therapeutics, microRNA regulation of cellular physiology and even xenomiRs have stimulated great interest, which have brought many researchers into the field. Despite many successes in determining general mechanisms of microRNA generation and function, the application of microRNAs in translational areas has not had as much success. It has been a challenge to localize microRNAs to a given cell type within tissues and assay them reliably. At supraphysiologic levels, microRNAs may regulate hosts of genes that are not the physiologic biochemical targets. Thus the applied and translational microRNA literature is filled with pitfalls and claims that are neither scientifically rigorous nor reproducible. This review is focused on increasing awareness of the challenges of working with microRNAs in translational research and recommends better practices in this area of discovery.
| S-EPMC5100345 | biostudies-literature
Project description:BackgroundMediation analysis can be used to evaluate the effect of an exposure on an outcome acting through an intermediate variable or mediator. For studies with small sample sizes, permutation testing may be useful in evaluating the indirect effect (i.e., the effect of exposure on the outcome through the mediator) while maintaining the appropriate type I error rate. For mediation analysis in studies with small sample sizes, existing permutation testing methods permute the residuals under the full or alternative model, but have not been evaluated under situations where covariates are included. In this article, we consider and evaluate two additional permutation approaches for testing the indirect effect in mediation analysis based on permutating the residuals under the reduced or null model which allows for the inclusion of covariates.MethodsSimulation studies were used to empirically evaluate the behavior of these two additional approaches: (1) the permutation test of the Indirect Effect under Reduced Models (IERM) and (2) the Permutation Supremum test under Reduced Models (PSRM). The performance of these methods was compared to the standard permutation approach for mediation analysis, the permutation test of the Indirect Effect under Full Models (IEFM). We evaluated the type 1 error rates and power of these methods in the presence of covariates since mediation analysis assumes no unmeasured confounders of the exposure-mediator-outcome relationships.ResultsThe proposed PSRM approach maintained type I error rates below nominal levels under all conditions, while the proposed IERM approach exhibited grossly inflated type I rates in many conditions and the standard IEFM exhibited inflated type I error rates under a small number of conditions. Power did not differ substantially between the proposed PSRM approach and the standard IEFM approach.ConclusionsThe proposed PSRM approach is recommended over the existing IEFM approach for mediation analysis in studies with small sample sizes.
| S-EPMC6982415 | biostudies-literature
Project description:In this work, we show how permutation methods can be applied to combination analyses such as those that include multiple imaging modalities, multiple data acquisitions of the same modality, or simply multiple hypotheses on the same data. Using the well-known definition of union-intersection tests and closed testing procedures, we use synchronized permutations to correct for such multiplicity of tests, allowing flexibility to integrate imaging data with different spatial resolutions, surface and/or volume-based representations of the brain, including non-imaging data. For the problem of joint inference, we propose and evaluate a modification of the recently introduced non-parametric combination (NPC) methodology, such that instead of a two-phase algorithm and large data storage requirements, the inference can be performed in a single phase, with reasonable computational demands. The method compares favorably to classical multivariate tests (such as MANCOVA), even when the latter is assessed using permutations. We also evaluate, in the context of permutation tests, various combining methods that have been proposed in the past decades, and identify those that provide the best control over error rate and power across a range of situations. We show that one of these, the method of Tippett, provides a link between correction for the multiplicity of tests and their combination. Finally, we discuss how the correction can solve certain problems of multiple comparisons in one-way ANOVA designs, and how the combination is distinguished from conjunctions, even though both can be assessed using permutation tests. We also provide a common algorithm that accommodates combination and correction.
| S-EPMC4783210 | biostudies-literature
Project description: Not available
| S-EPMC11297028 | biostudies-literature
Project description:Cell fate decisions during multicellular development are precisely coordinated, leading to highly reproducible macroscopic structural outcomes [1-3]. The origins of this reproducibility are found at the molecular level during the earliest stages of development when patterns of morphogen molecules emerge reproducibly [4, 5]. However, although the initial conditions for these early stages are determined by the female during oogenesis, it is unknown whether reproducibility is perpetuated from oogenesis or reacquired by the zygote. To address this issue in the early Drosophila embryo, we sought to count individual maternally deposited bicoid mRNA molecules and compare variability between embryos with previously observed fluctuations in the Bicoid protein gradient [6, 7]. Here, we develop independent methods to quantify total amounts of mRNA in individual embryos and show that mRNA counts are highly reproducible between embryos to within ~9%, matching the reproducibility of the protein gradient. Reproducibility emerges from perfectly linear feedforward processes: changing the genetic dosage in the female leads to proportional changes in both mRNA and protein numbers in the embryo. Our results indicate that the reproducibility of the morphological structures of embryos originates during oogenesis, which is when the expression of maternally provided patterning factors is precisely controlled.
| S-EPMC4061740 | biostudies-literature