Small sample sizes reduce the replicability of task-based fMRI studies.
Ontology highlight
ABSTRACT:
SUBMITTER: Turner BO
PROVIDER: S-EPMC6123695 | biostudies-other | 2018
REPOSITORIES: biostudies-other
Publications
Small sample sizes reduce the replicability of task-based fMRI studies.
Turner Benjamin O BO Paul Erick J EJ Miller Michael B MB Barbey Aron K AK
Communications biology 20180607
Despite a growing body of research suggesting that task-based functional magnetic resonance imaging (fMRI) studies often suffer from a lack of statistical power due to too-small samples, the proliferation of such underpowered studies continues unabated. Using large independent samples across eleven tasks, we demonstrate the impact of sample size on replicability, assessed at different levels of analysis relevant to fMRI researchers. We find that the degree of replicability for typical sample siz ...[more]
Similar Datasets
Project description:BackgroundMediation analysis can be used to evaluate the effect of an exposure on an outcome acting through an intermediate variable or mediator. For studies with small sample sizes, permutation testing may be useful in evaluating the indirect effect (i.e., the effect of exposure on the outcome through the mediator) while maintaining the appropriate type I error rate. For mediation analysis in studies with small sample sizes, existing permutation testing methods permute the residuals under the full or alternative model, but have not been evaluated under situations where covariates are included. In this article, we consider and evaluate two additional permutation approaches for testing the indirect effect in mediation analysis based on permutating the residuals under the reduced or null model which allows for the inclusion of covariates.MethodsSimulation studies were used to empirically evaluate the behavior of these two additional approaches: (1) the permutation test of the Indirect Effect under Reduced Models (IERM) and (2) the Permutation Supremum test under Reduced Models (PSRM). The performance of these methods was compared to the standard permutation approach for mediation analysis, the permutation test of the Indirect Effect under Full Models (IEFM). We evaluated the type 1 error rates and power of these methods in the presence of covariates since mediation analysis assumes no unmeasured confounders of the exposure-mediator-outcome relationships.ResultsThe proposed PSRM approach maintained type I error rates below nominal levels under all conditions, while the proposed IERM approach exhibited grossly inflated type I rates in many conditions and the standard IEFM exhibited inflated type I error rates under a small number of conditions. Power did not differ substantially between the proposed PSRM approach and the standard IEFM approach.ConclusionsThe proposed PSRM approach is recommended over the existing IEFM approach for mediation analysis in studies with small sample sizes.
| S-EPMC6982415 | biostudies-literature
Project description:Recently, several deep learning methods have been applied to decoding in task-related fMRI, and their advantages have been exploited in a variety of ways. However, this paradigm is sometimes problematic, due to the difficulty of applying deep learning to high-dimensional data and small sample size conditions. The difficulties in gathering a large amount of data to develop predictive machine learning models with multiple layers from fMRI experiments with complicated designs and tasks are well-recognized. Group-level, multi-voxel pattern analysis with small sample sizes results in low statistical power and large accuracy evaluation errors; failure in such instances is ascribed to the individual variability that risks information leakage, a particular issue when dealing with a limited number of subjects. In this study, using a small-size fMRI dataset evaluating bilingual language switch in a property generation task, we evaluated the relative fit of different deep learning models, incorporating moderate split methods to control the amount of information leakage. Our results indicated that using the session shuffle split as the data folding method, along with the multichannel 2D convolutional neural network (M2DCNN) classifier, recorded the best authentic classification accuracy, which outperformed the efficiency of 3D convolutional neural network (3DCNN). In this manuscript, we discuss the tolerability of within-subject or within-session information leakage, of which the impact is generally considered small but complex and essentially unknown; this requires clarification in future studies.
| S-EPMC7928289 | biostudies-literature
Project description:Task-based functional magnetic resonance imaging is a powerful tool for studying brain function, but neuroimaging research produces ongoing concerns regarding small-sample studies and how to interpret them. Although it is well understood that larger samples are preferable, many situations require researchers to make judgments from small studies, including reviewing the existing literature, analyzing pilot data, or assessing subsamples. Quantitative guidance on how to make these judgments remains scarce. To address this, we leverage the Human Connectome Project's Young Adult dataset to survey various analyses-from regional activation maps to predictive models. We find that, for some classic analyses such as detecting regional activation or cluster peak location, studies with as few as 40 subjects are adequate, although this depends crucially on effect sizes. For predictive modeling, similar sizes can be adequate for detecting whether features are predictable, but at least an order of magnitude more (at least hundreds) may be required for developing consistent predictions. These results offer valuable insights for designing and interpreting fMRI studies, emphasizing the importance of considering effect size, sample size, and analysis approach when assessing the reliability of findings. We hope that this survey serves as a reference for identifying which kinds of research questions can be reliably answered with small-scale studies.
| S-EPMC11326202 | biostudies-literature
Project description:Limited sample sizes can lead to spurious modeling findings in biomedical research. The objective of this work is to present a new method to generate synthetic populations (SPs) from limited samples using matched case-control data (n = 180 pairs), considered as two separate limited samples. SPs were generated with multivariate kernel density estimations (KDEs) with unconstrained bandwidth matrices. We included four continuous variables and one categorical variable for each individual. Bandwidth matrices were determined with Differential Evolution (DE) optimization by covariance comparisons. Four synthetic samples (n = 180) were derived from their respective SPs. Similarity between observed samples with synthetic samples was compared assuming their empirical probability density functions (EPDFs) were similar. EPDFs were compared with the maximum mean discrepancy (MMD) test statistic based on the Kernel Two-Sample Test. To evaluate similarity within a modeling context, EPDFs derived from the Principal Component Analysis (PCA) scores and residuals were summarized with the distance to the model in X-space (DModX) as additional comparisons. Four SPs were generated from each sample. The probability of selecting a replicate when randomly constructing synthetic samples (n = 180) was infinitesimally small. MMD tests indicated that the observed sample EPDFs were similar to the respective synthetic EPDFs. For the samples, PCA scores and residuals did not deviate significantly when compared with their respective synthetic samples. The feasibility of this approach was demonstrated by producing synthetic data at the individual level, statistically similar to the observed samples. The methodology coupled KDE with DE optimization and deployed novel similarity metrics derived from PCA. This approach could be used to generate larger-sized synthetic samples. To develop this approach into a research tool for data exploration purposes, additional evaluation with increased dimensionality is required. Moreover, given a fully specified population, the degree to which individuals can be discarded while synthesizing the respective population accurately will be investigated. When these objectives are addressed, comparisons with other techniques such as bootstrapping will be required for a complete evaluation.
| S-EPMC7839232 | biostudies-literature
Project description:ObjectiveChildhood irritability, operationalized as disproportionate and frequent temper tantrums and low frustration tolerance relative to peers, is a transdiagnostic symptom across many pediatric disorders. Studies using task-dependent functional magnetic resonance imaging (fMRI) to probe neural dysfunction in irritability have increased. However, an integrated review summarizing the published methods and synthesized fMRI results remains lacking.MethodWe conducted a systematic search using irritability terms and task functional neuroimaging in key databases in March 2021, and identified 30 studies for our systematic review. Sample characteristics and fMRI methods were summarized. A subset of 28 studies met the criteria for extracting coordinate-based data for quantitative meta-analysis. Ten activation-likelihood estimations were performed to examine neural convergence across irritability measures and fMRI task domains.ResultsSystematic review revealed small sample sizes (median = 58, mean age range = 8-16 years) with heterogeneous sample characteristics, irritability measures, tasks, and analytical procedures. Meta-analyses found no evidence for neural activation convergence of irritability across neurocognitive functions related to emotional reactivity, cognitive control, and reward processing, or within each domain. Sensitivity analyses partialing out variances driven by heterogeneous tasks, irritability measures, stimulus types, and developmental ages all yielded null findings. Results were compared with a review on irritability-related structural anomalies from 11 studies.ConclusionThe lack of neural convergence suggests a need for common, standardized irritability assessments and more homogeneous fMRI tasks. Thoughtfully designed fMRI studies probing commonly defined neurocognitive functions may be more fruitful to elucidate the neural mechanisms of irritability. Open science practices, data mining in large neuroscience databases, and standardized analytical methods promote meaningful collaboration in irritability research.
| S-EPMC9892288 | biostudies-literature
Project description:Over the past two decades, functional neuroimaging has not only grown into a large field of research, but also substantially evolved. Here we provide a quantitative assessment of these presumed in sample composition and data analysis, using fMRI studies on food/taste research published between 1998 and 2019 as an exemplary case in which the scientific objectives themselves have remained largely stable. A systematic search for papers written in English was done using multiple databases and identified 426 original articles that were subsequently analyzed. The median sample size significantly increased from 11.5 to 35.5 while the ratio of male to female subjects remained stable. There were, however, more papers involving female subjects only, rather than male subjects only, since 2003. There was a decline in uncorrected results and statistical correction by false-discovery rate. Reflecting a trend toward more conservative thresholding, the number of foci reported per paper did not change significantly and sample size (power) did not correlate with the number of reported foci. The median journal impact factor and the normalized number of citations (citations per year) of the papers, in turn, showed a significantly decreasing trend. Number of citations negatively correlated to sample size, publication year but positively correlated to journal impact factor, and was also influenced by statistical correction method. There was a decreasing trend in studies recruiting both left-handed and right-handed subjects. In summary, the present paper quantifies several large-scale trends that have often been anecdotally discussed and reveals the changing nature of neuroimaging studies that may be considered when pursuing meta-analytic approaches.
| S-EPMC7267904 | biostudies-literature
Project description:Accurate quantification of forest carbon stocks is required for constraining the global carbon cycle and its impacts on climate. The accuracies of forest biomass maps are inherently dependent on the accuracy of the field biomass estimates used to calibrate models, which are generated with allometric equations. Here, we provide a quantitative assessment of the sensitivity of allometric parameters to sample size in temperate forests, focusing on the allometric relationship between tree height and crown radius. We use LiDAR remote sensing to isolate between 10,000 to more than 1,000,000 tree height and crown radius measurements per site in six U.S. forests. We find that fitted allometric parameters are highly sensitive to sample size, producing systematic overestimates of height. We extend our analysis to biomass through the application of empirical relationships from the literature, and show that given the small sample sizes used in common allometric equations for biomass, the average site-level biomass bias is ~+70% with a standard deviation of 71%, ranging from -4% to +193%. These findings underscore the importance of increasing the sample sizes used for allometric equation generation.
| S-EPMC4657147 | biostudies-literature
Project description:The conventional approach of choosing sample size to provide 80% or greater power ignores the cost implications of different sample size choices. Costs, however, are often impossible for investigators and funders to ignore in actual practice. Here, we propose and justify a new approach for choosing sample size based on cost efficiency, the ratio of a study's projected scientific and/or practical value to its total cost. By showing that a study's projected value exhibits diminishing marginal returns as a function of increasing sample size for a wide variety of definitions of study value, we are able to develop two simple choices that can be defended as more cost efficient than any larger sample size. The first is to choose the sample size that minimizes the average cost per subject. The second is to choose sample size to minimize total cost divided by the square root of sample size. This latter method is theoretically more justifiable for innovative studies, but also performs reasonably well and has some justification in other cases. For example, if projected study value is assumed to be proportional to power at a specific alternative and total cost is a linear function of sample size, then this approach is guaranteed either to produce more than 90% power or to be more cost efficient than any sample size that does. These methods are easy to implement, based on reliable inputs, and well justified, so they should be regarded as acceptable alternatives to current conventional approaches.
| S-EPMC2769573 | biostudies-literature
Project description:Multi-echo fMRI, particularly the multi-echo independent component analysis (ME-ICA) algorithm, has previously proven useful for increasing the sensitivity and reducing false positives for functional MRI (fMRI) based resting state connectivity studies. Less is known about its efficacy for task-based fMRI, especially at the single subject level. This work, which focuses exclusively on individual subject results, compares ME-ICA to single-echo fMRI and a voxel-wise T2(⁎) weighted combination of multi-echo data for task-based fMRI under the following scenarios: cardiac-gated block designs, constant repetition time (TR) block designs, and constant TR rapid event-related designs. Performance is evaluated primarily in terms of sensitivity (i.e., activation extent, activation magnitude, percent detected trials and effect size estimates) using five different tasks expected to evoke neuronal activity in a distributed set of regions. The ME-ICA algorithm significantly outperformed all other evaluated processing alternatives in all scenarios. Largest improvements were observed for the cardiac-gated dataset, where ME-ICA was able to reliably detect and remove non-neural T1 signal fluctuations caused by non-constant repetition times. Although ME-ICA also outperformed the other options in terms of percent detection of individual trials for rapid event-related experiments, only 46% of all events were detected after ME-ICA; suggesting additional improvements in sensitivity are required to reliably detect individual short event occurrences. We conclude the manuscript with a detailed evaluation of ME-ICA outcomes and a discussion of how the ME-ICA algorithm could be further improved. Overall, our results suggest that ME-ICA constitutes a versatile, powerful approach for advanced denoising of task-based fMRI, not just resting-state data.
| S-EPMC5026969 | biostudies-literature
Project description:Quantitative morphometric analyses, particularly ontogenetic allometry, are common methods used in quantifying shape, and changes therein, in both extinct and extant organisms. Due to incompleteness and the potential for restricted sample sizes in the fossil record, palaeobiological analyses of allometry may encounter higher rates of error. Differences in sample size between fossil and extant studies and any resulting effects on allometric analyses have not been thoroughly investigated, and a logical lower threshold to sample size is not clear. Here we show that studies based on fossil datasets have smaller sample sizes than those based on extant taxa. A similar pattern between vertebrates and invertebrates indicates this is not a problem unique to either group, but common to both. We investigate the relationship between sample size, ontogenetic allometric relationship and statistical power using an empirical dataset of skull measurements of modern Alligator mississippiensis. Across a variety of subsampling techniques, used to simulate different taphonomic and/or sampling effects, smaller sample sizes gave less reliable and more variable results, often with the result that allometric relationships will go undetected due to Type II error (failure to reject the null hypothesis). This may result in a false impression of fewer instances of positive/negative allometric growth in fossils compared to living organisms. These limitations are not restricted to fossil data and are equally applicable to allometric analyses of rare extant taxa. No mathematically derived minimum sample size for ontogenetic allometric studies is found; rather results of isometry (but not necessarily allometry) should not be viewed with confidence at small sample sizes.
| S-EPMC4358694 | biostudies-literature