Dataset Information

Reanalyzing Head et al. (2015): investigating the robustness of widespread p-hacking.

ABSTRACT: Head et al. (2015) provided a large collection of p-values that, from their perspective, indicates widespread statistical significance seeking (i.e., p-hacking). This paper inspects this result for robustness. Theoretically, the p-value distribution should be a smooth, decreasing function, but the distribution of reported p-values shows systematically more reported p-values for .01, .02, .03, .04, and .05 than p-values reported to three decimal places, due to apparent tendencies to round p-values to two decimal places. Head et al. (2015) correctly argue that an aggregate p-value distribution could show a bump below .05 when left-skew p-hacking occurs frequently. Moreover, the elimination of p = .045 and p = .05, as done in the original paper, is debatable. Given that eliminating p = .045 is a result of the need for symmetric bins and systematically more p-values are reported to two decimal places than to three decimal places, I did not exclude p = .045 and p = .05. I conducted Fisher's method .04 < p < .05 and reanalyzed the data by adjusting the bin selection to .03875 < p ? .04 versus .04875 < p ? .05. Results of the reanalysis indicate that no evidence for left-skew p-hacking remains when we look at the entire range between .04 < p < .05 or when we inspect the second-decimal. Taking into account reporting tendencies when selecting the bins to compare is especially important because this dataset does not allow for the recalculation of the p-values. Moreover, inspecting the bins that include two-decimal reported p-values potentially increases sensitivity if strategic rounding down of p-values as a form of p-hacking is widespread. Given the far-reaching implications of supposed widespread p-hacking throughout the sciences Head et al. (2015), it is important that these findings are robust to data analysis choices if the conclusion is to be considered unequivocal. Although no evidence of widespread left-skew p-hacking is found in this reanalysis, this does not mean that there is no p-hacking at all. These results nuance the conclusion by Head et al. (2015), indicating that the results are not robust and that the evidence for widespread left-skew p-hacking is ambiguous at best.

SUBMITTER: Hartgerink CH

PROVIDER: S-EPMC5337083 | biostudies-literature | 2017

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Reanalyzing Head et al. (2015): investigating the robustness of widespread p-hacking.

Hartgerink Chris H J CH

PeerJ 20170302

Head et al. (2015) provided a large collection of p-values that, from their perspective, indicates widespread statistical significance seeking (i.e., p-hacking). This paper inspects this result for robustness. Theoretically, the p-value distribution should be a smooth, decreasing function, but the distribution of reported p-values shows systematically more reported p-values for .01, .02, .03, .04, and .05 than p-values reported to three decimal places, d ...[more]

PMID: 28265523

Similar Datasets

Project description:Aneuploidy is a hallmark of tumor cells and yet the precise relationship between aneuploidy and a cell’s proliferative ability, or cellular fitness, has remained elusive. In this study we have combined a detailed analysis of aneuploid clones isolated from laboratory-evolved populations of Saccharomyces cerevisiae with a systematic, genome-wide screen for the fitness effects of telomeric amplifications to address the relationship between aneuploidy and cellular fitness. We found that aneuploid clones rise to high population frequencies in nutrient-limited evolution experiments and show increased fitness relative to wild-type. Direct competition experiments confirmed that three out of four aneuploid events isolated from evolved populations were themselves sufficient to improve fitness. To expand the scope beyond this small number of exemplars, we created a genome-wide collection of >1,800 diploid yeast strains each containing a different telomeric amplicon (Tamp) ranging in size from 0.4 to 1,000kb. Using pooled competition experiments in nutrient-limited chemostats followed by high-throughput sequencing of strain-identifying barcodes, we determined the fitness effects of these >1,800 Tamps under three different conditions. Our data revealed that the fitness landscape explored by telomeric amplifications is much broader than that explored by single-gene amplifications. As also observed in the evolved clones, we found the fitness effects of most Tamps to be condition specific with a minority showing common effects in all three conditions. By integrating our data with previous work that examined the fitness effects of single-gene amplifications genome wide, we found that a small number of genes within each Tamp are centrally responsible for each Tamp’s fitness effects. Our genome-wide Tamp screen confirmed that telomeric amplifications identified in laboratory-evolved populations generally increased fitness. Our results show that Tamps are mutations that produce large, typically condition-dependent changes in fitness that are important drivers of increased fitness in asexually evolving populations. Each of these arrays is a Comparative Genomic Hybridization experiment to detect copy number differences between a reference strain and a strain of interest.

Dataset Information

Reanalyzing Head et al. (2015): investigating the robustness of widespread p-hacking.

Publications

Reanalyzing Head et al. (2015): investigating the robustness of widespread <i>p</i>-hacking.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets