Dataset Information

Accumulation Bias in meta-analysis: the need to consider time in error control.

ABSTRACT: Studies accumulate over time and meta-analyses are mainly retrospective. These two characteristics introduce dependencies between the analysis time, at which a series of studies is up for meta-analysis, and results within the series. Dependencies introduce bias - Accumulation Bias - and invalidate the sampling distribution assumed for p-value tests, thus inflating type-I errors. But dependencies are also inevitable, since for science to accumulate efficiently, new research needs to be informed by past results. Here, we investigate various ways in which time influences error control in meta-analysis testing. We introduce an Accumulation Bias Framework that allows us to model a wide variety of practically occurring dependencies including study series accumulation, meta-analysis timing, and approaches to multiple testing in living systematic reviews. The strength of this framework is that it shows how all dependencies affect p-value-based tests in a similar manner. This leads to two main conclusions. First, Accumulation Bias is inevitable, and even if it can be approximated and accounted for, no valid p-value tests can be constructed. Second, tests based on likelihood ratios withstand Accumulation Bias: they provide bounds on error probabilities that remain valid despite the bias. We leave the reader with a choice between two proposals to consider time in error control: either treat individual (primary) studies and meta-analyses as two separate worlds - each with their own timing - or integrate individual studies in the meta-analysis world. Taking up likelihood ratios in either approach allows for valid tests that relate well to the accumulating nature of scientific knowledge. Likelihood ratios can be interpreted as betting profits, earned in previous studies and invested in new ones, while the meta-analyst is allowed to cash out at any time and advice against future studies.

SUBMITTER: Ter Schure J

PROVIDER: S-EPMC6808047 | biostudies-literature | 2019

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Accumulation Bias in meta-analysis: the need to consider time in error control.

Ter Schure Judith J Grünwald Peter P

F1000Research 20190625

Studies accumulate over time and meta-analyses are mainly retrospective. These two characteristics introduce dependencies between the analysis time, at which a series of studies is up for meta-analysis, and results within the series. Dependencies introduce bias -Accumulation Bias- and invalidate the sampling distribution assumed for p-value tests, thus inflating type-I errors. But dependencies are also inevitable, since for science to accumulate efficiently, new resea ...[more]

PMID: 31737258

Similar Datasets

Project description:BackgroundThe rank correlation test introduced by Begg and Mazumdar is extensively used in meta-analysis to test for publication bias in clinical and epidemiological studies. It is based on correlating the standardized treatment effect with the variance of the treatment effect using Kendall's tau as the measure of association. To our knowledge, the operational characteristics regarding the significance level of the test have not, however, been fully assessed.MethodsWe propose an alternative rank correlation test to improve the error rates of the original Begg and Mazumdar test. This test is based on the simulated distribution of the estimated measure of association, conditional on sampling variances. Furthermore, Spearman's rho is suggested as an alternative rank correlation coefficient. The attained level and power of the tests are studied by simulations of meta-analyses assuming the fixed effects model.ResultsThe significance levels of the original Begg and Mazumdar test often deviate considerably from the nominal level, the null hypothesis being rejected too infrequently. It is proven mathematically that the assumptions for using the rank correlation test are not strictly satisfied. The pairs of variables fail to be independent, and there is a correlation between the standardized effect sizes and sampling variances under the null hypothesis of no publication bias. In the meta-analysis setting, the adverse consequences of a false negative test are more profound than the disadvantages of a false positive test. Our alternative test improves the error rates in fixed effects meta-analysis. Its significance level equals the nominal value, and the Type II error rate is reduced. In small data sets Spearman's rho should be preferred to Kendall's tau as the measure of association.ConclusionsAs the attained significance levels of the test introduced by Begg and Mazumdar often deviate greatly from the nominal level, modified rank correlation tests, improving the error rates, should be preferred when testing for publication bias assuming fixed effects meta-analysis.

Project description:BackgroundError processing and inhibitory control enable the adjustment of behaviors to meet task demands. Functional magnetic resonance imaging studies report brain activation abnormalities in patients with obsessive-compulsive disorder (OCD) during both processes. However, conclusions are limited by inconsistencies in the literature and small sample sizes. Therefore, the aim here was to perform a meta-analysis of the existing literature using unthresholded statistical maps from previous studies.MethodsA voxelwise seed-based d mapping meta-analysis was performed using t-maps from studies comparing patients with OCD and healthy control subjects (HCs) during error processing and inhibitory control. For the error processing analysis, 239 patients with OCD (120 male; 79 medicated) and 229 HCs (129 male) were included, while the inhibitory control analysis included 245 patients with OCD (120 male; 91 medicated) and 239 HCs (135 male).ResultsPatients with OCD, relative to HCs, showed longer inhibitory control reaction time (standardized mean difference = 0.20, p = .03, 95% confidence interval = 0.016, 0.393) and more inhibitory control errors (standardized mean difference = 0.22, p = .02, 95% confidence interval = 0.039, 0.399). In the brain, patients showed hyperactivation in the bilateral dorsal anterior cingulate cortex, supplementary motor area, and pre-supplementary motor area as well as right anterior insula/frontal operculum and anterior lateral prefrontal cortex during error processing but showed hypoactivation during inhibitory control in the rostral and ventral anterior cingulate cortices and bilateral thalamus/caudate, as well as the right anterior insula/frontal operculum, supramarginal gyrus, and medial orbitofrontal cortex (all seed-based d mapping z value >2, p < .001).ConclusionsA hyperactive error processing mechanism in conjunction with impairments in implementing inhibitory control may underlie deficits in stopping unwanted compulsive behaviors in the disorder.

Dataset Information

Accumulation Bias in meta-analysis: the need to consider time in error control.

Publications

Accumulation Bias in meta-analysis: the need to consider <i>time</i> in error control.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets