A robust permutation test for the concordance correlation coefficient.
Ontology highlight
ABSTRACT:
SUBMITTER: Hutson AD
PROVIDER: S-EPMC8359348 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Statistical inference in neuroimaging research often involves testing the significance of regression coefficients in a general linear model. In many applications, the researcher assumes a model of the form Y=?+X?+Z?+?, where Y is the observed brain signal, and X and Z contain explanatory variables that are thought to be related to the brain signal. The goal is to test the null hypothesis H0:?=0 with the nuisance parameters ? included in the model. Several nonparametric (permutation) methods have been proposed for this problem, and each method uses some variant of the F ratio as the test statistic. However, recent research suggests that the F ratio can produce invalid permutation tests of H0:?=0 when the ? terms are heteroscedastic (i.e., have non-constant variance), which can occur for a variety of reasons. This study compares the classic F test statistic to the robust W (Wald) test statistic using eight different permutation methods. The results reveal that permutation tests using the F ratio can produce accurate results when the errors are homoscedastic, but high false positive rates when the errors are heteroscedastic. In contrast, permutation tests using the W test statistic produced valid results when the errors were homoscedastic, and asymptotically valid results when the errors were heteroscedastic. In the situation with homoscedastic errors, permutation tests using the W statistic showed slightly reduced power compared to the F statistic, but the difference disappeared as the sample size n increased. Consequently, the W test statistic is recommended for robust nonparametric hypothesis tests of regression coefficients in neuroimaging research.
| S-EPMC6765412 | biostudies-literature
Project description:The human microbiota composition is associated with a number of diseases including obesity, inflammatory bowel disease, and bacterial vaginosis. Thus, microbiome research has the potential to reshape clinical and therapeutic approaches. However, raw microbiome count data require careful pre-processing steps that take into account both the sparsity of counts and the large number of taxa that are being measured. Filtering is defined as removing taxa that are present in a small number of samples and have small counts in the samples where they are observed. Despite progress in the number and quality of filtering approaches, there is no consensus on filtering standards and quality assessment. This can adversely affect downstream analyses and reproducibility of results across platforms and software. We introduce PERFect, a novel permutation filtering approach designed to address two unsolved problems in microbiome data processing: (i) define and quantify loss due to filtering by implementing thresholds and (ii) introduce and evaluate a permutation test for filtering loss to provide a measure of excessive filtering. Methods are assessed on three "mock experiment" data sets, where the true taxa compositions are known, and are applied to two publicly available real microbiome data sets. The method correctly removes contaminant taxa in "mock" data sets, quantifies and visualizes the corresponding filtering loss, providing a uniform data-driven filtering criteria for real microbiome data sets. In real data analyses PERFect tends to remove more taxa than existing approaches; this likely happens because the method is based on an explicit loss function, uses statistically principled testing, and takes into account correlation between taxa. The PERFect software is freely available at https://github.com/katiasmirn/PERFect.
| S-EPMC6797060 | biostudies-literature
Project description:Case-control association studies often suffer from population stratification bias. A previous triple combination strategy of stratum matching, genomic controlling, and multiple DNA pooling can correct the bias and save genotyping cost. However the method requires researchers to prepare a multitude of DNA pools-more than 30 case-control pooling sets in total (polyset). In this paper, the authors propose a permutation test for oligoset DNA pooling studies. Monte-Carlo simulations show that the proposed test has a type I error rate under control and a power comparable to that of individual genotyping. For a researcher on a tight budget, oligoset DNA pooling is a viable option.
| S-EPMC4357378 | biostudies-literature
Project description:Many key findings in neuroimaging studies involve similarities between brain maps, but statistical methods used to measure these findings have varied. Current state-of-the-art methods involve comparing observed group-level brain maps (after averaging intensities at each image location across multiple subjects) against spatial null models of these group-level maps. However, these methods typically make strong and potentially unrealistic statistical assumptions, such as covariance stationarity. To address these issues, in this article we propose using subject-level data and a classical permutation testing framework to test and assess similarities between brain maps. Our method is comparable to traditional permutation tests in that it involves randomly permuting subjects to generate a null distribution of intermodal correspondence statistics, which we compare to an observed statistic to estimate a p-value. We apply and compare our method in simulated and real neuroimaging data from the Philadelphia Neurodevelopmental Cohort. We show that our method performs well for detecting relationships between modalities known to be strongly related (cortical thickness and sulcal depth), and it is conservative when an association would not be expected (cortical thickness and activation on the n-back working memory task). Notably, our method is the most flexible and reliable for localizing intermodal relationships within subregions of the brain and allows for generalizable statistical inference.
| S-EPMC8519855 | biostudies-literature
Project description:In response to the importance of individual-level effects, the purpose of this paper is to describe the new randomization permutation (RP) test for a mediation mechanism for a single subject. We extend seminal work on permutation tests for individual-level data by proposing a test for mediation for one person. The method requires random assignment to the levels of the treatment variable at each measurement occasion, and repeated measures of the mediator and outcome from one subject. If several assumptions are met, the process by which a treatment changes an outcome can be statistically evaluated for a single subject, using the permutation mediation test method and the permutation confidence interval method for residuals. A simulation study evaluated the statistical properties of the new method suggesting that at least eight repeated measures are needed to control Type I error rates and larger sample sizes are needed for power approaching .8 even for large effects. The RP mediation test is a promising method for elucidating intraindividual processes of change that may inform personalized medicine and tailoring of process-based treatments for one subject.
| S-EPMC8995141 | biostudies-literature
Project description:Testing for random mating of a population is important in population genetics, because deviations from randomness of mating may indicate inbreeding, population stratification, natural selection, or sampling bias. However, current methods use only observed numbers of genotypes and alleles, and do not take advantage of the fact that the advent of sequencing technology provides an opportunity to investigate this topic in unprecedented detail. To address this opportunity, a novel statistical test for random mating is required in population genomics studies for which large sequencing datasets are generally available. Here, we propose a Monte-Carlo-based-permutation test (MCP) as an approach to detect random mating. Computer simulations used to evaluate the performance of the permutation test indicate that its type I error is well controlled and that its statistical power is greater than that of the commonly used chi-square test (CHI). Our simulation study shows the power of our test is greater for datasets characterized by lower levels of migration between subpopulations. In addition, test power increases with increasing recombination rate, sample size, and divergence time of subpopulations. For populations exhibiting limited migration and having average levels of population divergence, the statistical power approaches 1 for sequences longer than 1 Mbp and for samples of 400 individuals or more. Taken together, our results suggest that our permutation test is a valuable tool to detect random mating of populations, especially in population genomics studies.
| S-EPMC3734302 | biostudies-literature
Project description:When a statistical test is repeatedly applied to rows of a data matrix, correlations among data rows will give rise to correlations among corresponding test statistics. We investigate the relationship between test-statistic correlation and data-row correlation and discuss its implications.
| S-EPMC7723344 | biostudies-literature
Project description:BackgroundRandom forests (RF) have been increasingly used in applications such as genome-wide association and microarray studies where predictor correlation is frequently observed. Recent works on permutation-based variable importance measures (VIMs) used in RF have come to apparently contradictory conclusions. We present an extended simulation study to synthesize results.ResultsIn the case when both predictor correlation was present and predictors were associated with the outcome (HA), the unconditional RF VIM attributed a higher share of importance to correlated predictors, while under the null hypothesis that no predictors are associated with the outcome (H0) the unconditional RF VIM was unbiased. Conditional VIMs showed a decrease in VIM values for correlated predictors versus the unconditional VIMs under HA and was unbiased under H0. Scaled VIMs were clearly biased under HA and H0.ConclusionsUnconditional unscaled VIMs are a computationally tractable choice for large datasets and are unbiased under the null hypothesis. Whether the observed increased VIMs for correlated predictors may be considered a "bias" - because they do not directly reflect the coefficients in the generating model - or if it is a beneficial attribute of these VIMs is dependent on the application. For example, in genetic association studies, where correlation between markers may help to localize the functionally relevant variant, the increased importance of correlated predictors may be an advantage. On the other hand, we show examples where this increased importance may result in spurious signals.
| S-EPMC2848005 | biostudies-literature
Project description:SummaryDue to the sparsity and high dimensionality, microbiome data are routinely summarized into pairwise distances capturing the compositional differences. Many biological insights can be gained by analyzing the distance matrix in relation to some covariates. A microbiome sampling method that characterizes the inter-sample relationship more reproducibly is expected to yield higher statistical power. Traditionally, the intraclass correlation coefficient (ICC) has been used to quantify the degree of reproducibility for a univariate measurement using technical replicates. In this work, we extend the traditional ICC to distance measures and propose a distance-based ICC (dICC). We derive the asymptotic distribution of the sample-based dICC to facilitate statistical inference. We illustrate dICC using a real dataset from a metagenomic reproducibility study.Availability and implementationdICC is implemented in the R CRAN package GUniFrac.Supplementary informationSupplementary data are available at Bioinformatics online.
| S-EPMC9801959 | biostudies-literature
Project description:Data imbalance is frequently encountered in biomedical applications. Resampling techniques can be used in binary classification to tackle this issue. However such solutions are not desired when the number of samples in the small class is limited. Moreover the use of inadequate performance metrics, such as accuracy, lead to poor generalization results because the classifiers tend to predict the largest size class. One of the good approaches to deal with this issue is to optimize performance metrics that are designed to handle data imbalance. Matthews Correlation Coefficient (MCC) is widely used in Bioinformatics as a performance metric. We are interested in developing a new classifier based on the MCC metric to handle imbalanced data. We derive an optimal Bayes classifier for the MCC metric using an approach based on Frechet derivative. We show that the proposed algorithm has the nice theoretical property of consistency. Using simulated data, we verify the correctness of our optimality result by searching in the space of all possible binary classifiers. The proposed classifier is evaluated on 64 datasets from a wide range data imbalance. We compare both classification performance and CPU efficiency for three classifiers: 1) the proposed algorithm (MCC-classifier), the Bayes classifier with a default threshold (MCC-base) and imbalanced SVM (SVM-imba). The experimental evaluation shows that MCC-classifier has a close performance to SVM-imba while being simpler and more efficient.
| S-EPMC5456046 | biostudies-literature