Dataset Information

Sometimes More is Too Much: A Rejoinder to the Commentaries on Greiff et al. (2015).

ABSTRACT: In this rejoinder, we respond to two commentaries on the study by Greiff, S.; Stadler, M.; Sonnleitner, P.; Wolff, C.; Martin, R. Sometimes less is more: Comparing the validity of complex problem solving measures. Intelligence 2015, 50, 100-113. The study was the first to address the important comparison between a classical measure of complex problem solving (CPS) and the more recent multiple complex systems (MCS) approach regarding their validity. In the study, we investigated the relations between one classical microworld as the initially developed method (here, the Tailorshop) with three more recently developed multiple complex systems (MCS; here, MicroDYN, Genetics Lab, and MicroFIN) tests. We found that the MCS tests showed higher levels of convergent validity with each other than with the Tailorshop even after reasoning was controlled for, thus empirically distinguishing between the two approaches. The commentary by Kretzschmar and the commentary by Funke, Fischer, and Holt expressed several concerns with how our study was conducted, our data was analyzed, and our results were interpreted. Whereas we acknowledge and agree with some of the more general statements made in these commentaries, we respectfully disagree with others, or we consider them to be at least partially in contrast with the existing literature and the currently available empirical evidence.

SUBMITTER: Greiff S

PROVIDER: S-EPMC6526441 | biostudies-other | 2017 Jan

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

Sometimes More is Too Much: A Rejoinder to the Commentaries on Greiff et al. (2015).

Greiff Samuel S Stadler Matthias M Sonnleitner Philipp P Wolff Christian C Martin Romain R

Journal of Intelligence 20170105 1

In this rejoinder, we respond to two commentaries on the study by Greiff, S.; Stadler, M.; Sonnleitner, P.; Wolff, C.; Martin, R. Sometimes less is more: Comparing the validity of complex problem solving measures. Intelligence 2015, 50, 100-113. The study was the first to address the important comparison between a classical measure of complex problem solving (CPS) and the more recent multiple complex systems (MCS) approach regarding their validity. In the study, we investiga ...[more]

PMID: 31162396

Similar Datasets

Project description:Aneuploidy is a hallmark of tumor cells and yet the precise relationship between aneuploidy and a cell’s proliferative ability, or cellular fitness, has remained elusive. In this study we have combined a detailed analysis of aneuploid clones isolated from laboratory-evolved populations of Saccharomyces cerevisiae with a systematic, genome-wide screen for the fitness effects of telomeric amplifications to address the relationship between aneuploidy and cellular fitness. We found that aneuploid clones rise to high population frequencies in nutrient-limited evolution experiments and show increased fitness relative to wild-type. Direct competition experiments confirmed that three out of four aneuploid events isolated from evolved populations were themselves sufficient to improve fitness. To expand the scope beyond this small number of exemplars, we created a genome-wide collection of >1,800 diploid yeast strains each containing a different telomeric amplicon (Tamp) ranging in size from 0.4 to 1,000kb. Using pooled competition experiments in nutrient-limited chemostats followed by high-throughput sequencing of strain-identifying barcodes, we determined the fitness effects of these >1,800 Tamps under three different conditions. Our data revealed that the fitness landscape explored by telomeric amplifications is much broader than that explored by single-gene amplifications. As also observed in the evolved clones, we found the fitness effects of most Tamps to be condition specific with a minority showing common effects in all three conditions. By integrating our data with previous work that examined the fitness effects of single-gene amplifications genome wide, we found that a small number of genes within each Tamp are centrally responsible for each Tamp’s fitness effects. Our genome-wide Tamp screen confirmed that telomeric amplifications identified in laboratory-evolved populations generally increased fitness. Our results show that Tamps are mutations that produce large, typically condition-dependent changes in fitness that are important drivers of increased fitness in asexually evolving populations. Each of these arrays is a Comparative Genomic Hybridization experiment to detect copy number differences between a reference strain and a strain of interest.

Project description:Head et al. (2015) provided a large collection of p-values that, from their perspective, indicates widespread statistical significance seeking (i.e., p-hacking). This paper inspects this result for robustness. Theoretically, the p-value distribution should be a smooth, decreasing function, but the distribution of reported p-values shows systematically more reported p-values for .01, .02, .03, .04, and .05 than p-values reported to three decimal places, due to apparent tendencies to round p-values to two decimal places. Head et al. (2015) correctly argue that an aggregate p-value distribution could show a bump below .05 when left-skew p-hacking occurs frequently. Moreover, the elimination of p = .045 and p = .05, as done in the original paper, is debatable. Given that eliminating p = .045 is a result of the need for symmetric bins and systematically more p-values are reported to two decimal places than to three decimal places, I did not exclude p = .045 and p = .05. I conducted Fisher's method .04 < p < .05 and reanalyzed the data by adjusting the bin selection to .03875 < p ≤ .04 versus .04875 < p ≤ .05. Results of the reanalysis indicate that no evidence for left-skew p-hacking remains when we look at the entire range between .04 < p < .05 or when we inspect the second-decimal. Taking into account reporting tendencies when selecting the bins to compare is especially important because this dataset does not allow for the recalculation of the p-values. Moreover, inspecting the bins that include two-decimal reported p-values potentially increases sensitivity if strategic rounding down of p-values as a form of p-hacking is widespread. Given the far-reaching implications of supposed widespread p-hacking throughout the sciences Head et al. (2015), it is important that these findings are robust to data analysis choices if the conclusion is to be considered unequivocal. Although no evidence of widespread left-skew p-hacking is found in this reanalysis, this does not mean that there is no p-hacking at all. These results nuance the conclusion by Head et al. (2015), indicating that the results are not robust and that the evidence for widespread left-skew p-hacking is ambiguous at best.

Dataset Information

Sometimes More is Too Much: A Rejoinder to the Commentaries on Greiff et al. (2015).

Publications

Sometimes More is Too Much: A Rejoinder to the Commentaries on Greiff et al. (2015).

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets