Self-reports from behind the scenes: Questionable research practices and rates of replication in ego depletion research.
Ontology highlight
ABSTRACT:
SUBMITTER: Wolff W
PROVIDER: S-EPMC6016937 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:A survey in the United States revealed that an alarmingly large percentage of university psychologists admitted having used questionable research practices that can contaminate the research literature with false positive and biased findings. We conducted a replication of this study among Italian research psychologists to investigate whether these findings generalize to other countries. All the original materials were translated into Italian, and members of the Italian Association of Psychology were invited to participate via an online survey. The percentages of Italian psychologists who admitted to having used ten questionable research practices were similar to the results obtained in the United States although there were small but significant differences in self-admission rates for some QRPs. Nearly all researchers (88%) admitted using at least one of the practices, and researchers generally considered a practice possibly defensible if they admitted using it, but Italian researchers were much less likely than US researchers to consider a practice defensible. Participants' estimates of the percentage of researchers who have used these practices were greater than the self-admission rates, and participants estimated that researchers would be unlikely to admit it. In written responses, participants argued that some of these practices are not questionable and they have used some practices because reviewers and journals demand it. The similarity of results obtained in the United States, this study, and a related study conducted in Germany suggest that adoption of these practices is an international phenomenon and is likely due to systemic features of the international research and publication processes.
| S-EPMC5351839 | biostudies-literature
Project description:We surveyed 807 researchers (494 ecologists and 313 evolutionary biologists) about their use of Questionable Research Practices (QRPs), including cherry picking statistically significant results, p hacking, and hypothesising after the results are known (HARKing). We also asked them to estimate the proportion of their colleagues that use each of these QRPs. Several of the QRPs were prevalent within the ecology and evolution research community. Across the two groups, we found 64% of surveyed researchers reported they had at least once failed to report results because they were not statistically significant (cherry picking); 42% had collected more data after inspecting whether results were statistically significant (a form of p hacking) and 51% had reported an unexpected finding as though it had been hypothesised from the start (HARKing). Such practices have been directly implicated in the low rates of reproducible results uncovered by recent large scale replication studies in psychology and other disciplines. The rates of QRPs found in this study are comparable with the rates seen in psychology, indicating that the reproducibility problems discovered in psychology are also likely to be present in ecology and evolution.
| S-EPMC6047784 | biostudies-literature
Project description:This article examines why many studies fail to replicate statistically significant published results. We address this issue within a general statistical framework that also allows us to include various questionable research practices (QRPs) that are thought to reduce replicability. The analyses indicate that the base rate of true effects is the major factor that determines the replication rate of scientific results. Specifically, for purely statistical reasons, replicability is low in research domains where true effects are rare (e.g., search for effective drugs in pharmacology). This point is under-appreciated in current scientific and media discussions of replicability, which often attribute poor replicability mainly to QRPs.
| S-EPMC7561355 | biostudies-literature
Project description:There has been a surge of interest in research integrity over the last decade, with a wide range of studies investigating the prevalence of questionable research practices (QRPs). However, nearly all these studies focus on research design, data collection and analysis, and hardly any empirical research has been done on the occurrence of QRPs in the context of research funding. To fill this gap, we conducted a cross-sectional pre-registered survey of applicants, reviewers and panel members from the Research Foundation-Flanders (FWO), one of the main funding agencies in Belgium. We developed a bespoke survey and further refined it through feedback from experienced researchers and a pilot study. We asked how often respondents had engaged in a series of QRPs over the last ten years. A total of 1748 emails were sent, inviting recipients to participate in the survey, complemented by featuring the survey in the FWO newsletter. This resulted in 704 complete responses. Our results indicate that such QRPs are remarkably prevalent. Of the 496 participants who answered both the applicant and reviewer track, more than 60% responded that they engaged regularly in at least one of such practices, and around 40% indicated that they engaged at least occasionally in half of the QRPs queried. Only 12% reported not to have engaged in any of the QRPs. Contrary to our hypotheses, male respondents did not self-report to engage in the QRPs more often than female respondents, nor was there an association between the prevalence of QRPs and self-reported success rate in grant funding. Furthermore, half of the respondents indicated that they doubted the reliability of the grant peer review process more often than not. These results suggest that preventive action is needed, and provide new reasons to reconsider the practice of allocating research money through grant peer review.
| S-EPMC10621923 | biostudies-literature
Project description:BackgroundClinical prediction models are widely used in health and medical research. The area under the receiver operating characteristic curve (AUC) is a frequently used estimate to describe the discriminatory ability of a clinical prediction model. The AUC is often interpreted relative to thresholds, with "good" or "excellent" models defined at 0.7, 0.8 or 0.9. These thresholds may create targets that result in "hacking", where researchers are motivated to re-analyse their data until they achieve a "good" result.MethodsWe extracted AUC values from PubMed abstracts to look for evidence of hacking. We used histograms of the AUC values in bins of size 0.01 and compared the observed distribution to a smooth distribution from a spline.ResultsThe distribution of 306,888 AUC values showed clear excesses above the thresholds of 0.7, 0.8 and 0.9 and shortfalls below the thresholds.ConclusionsThe AUCs for some models are over-inflated, which risks exposing patients to sub-optimal clinical decision-making. Greater modelling transparency is needed, including published protocols, and data and code sharing.
| S-EPMC10478406 | biostudies-literature
Project description:IntroductionIn this study, we tested a simple, active "ethical consistency" intervention aimed at reducing researchers' endorsement of questionable research practices (QRPs).MethodsWe developed a simple, active ethical consistency intervention and tested it against a control using an established QRP survey instrument. Before responding to a survey that asked about attitudes towards each of fifteen QRPs, participants were randomly assigned to either a consistency or control 3-5-min writing task. A total of 201 participants completed the survey: 121 participants were recruited from a database of currently funded NSF/NIH scientists, and 80 participants were recruited from a pool of active researchers at a large university medical center in the southeastern US. Narrative responses to the writing prompts were coded and analyzed to assist post hoc interpretation of the quantitative data.ResultsWe hypothesized that participants in the consistency condition would find ethically ambiguous QRPs less defensible and would indicate less willingness to engage in them than participants in the control condition. The results showed that the consistency intervention had no significant effect on respondents' reactions regarding the defensibility of the QRPs or their willingness to engage in them. Exploratory analyses considering the narrative themes of participants' responses indicated that participants in the control condition expressed lower perceptions of QRP defensibility and willingness.ConclusionThe results did not support the main hypothesis, and the consistency intervention may have had the unwanted effect of inducing increased rationalization. These results may partially explain why RCR courses often seem to have little positive effect.
| S-EPMC6883712 | biostudies-literature
Project description:Implicit skill learning occurs incidentally and without conscious awareness of what is learned. However, the rate and effectiveness of learning may still be affected by decreased availability of central processing resources. Dual-task experiments have generally found impairments in implicit learning, however, these studies have also shown that certain characteristics of the secondary task (e.g., timing) can complicate the interpretation of these results. To avoid this problem, the current experiments used a novel method to impose resource constraints prior to engaging in skill learning. Ego depletion theory states that humans possess a limited store of cognitive resources that, when depleted, results in deficits in self-regulation and cognitive control. In a first experiment, we used a standard ego depletion manipulation prior to performance of the Serial Interception Sequence Learning (SISL) task. Depleted participants exhibited poorer test performance than did non-depleted controls, indicating that reducing available executive resources may adversely affect implicit sequence learning, expression of sequence knowledge, or both. In a second experiment, depletion was administered either prior to or after training. Participants who reported higher levels of depletion before or after training again showed less sequence-specific knowledge on the post-training assessment. However, the results did not allow for clear separation of ego depletion effects on learning versus subsequent sequence-specific performance. These results indicate that performance on an implicitly learned sequence can be impaired by a reduction in executive resources, in spite of learning taking place outside of awareness and without conscious intent.
| S-EPMC4183724 | biostudies-literature
Project description:INTRODUCTION:Engaging in scientific misconduct and questionable research practices (QRPs) is a noted problem across fields, including health professions education (HPE). To mitigate these practices, other disciplines have enacted strategies based on researcher characteristics and practice factors. Thus, to inform HPE, this study seeks to determine which researcher characteristics and practice factors, if any, might explain the frequency of irresponsible research practices. METHOD:In 2017, a cross-sectional survey of HPE researchers was conducted. The survey included 66 items adapted from three published surveys: two published QRP surveys and a publication pressure scale. The outcome variable was a self-reported misconduct score, which is a weighted mean score for each respondent on all misconduct and QRP items. Statistical analysis included descriptive statistics, reliability and correlation analysis, and multiple linear regression modelling. RESULTS AND DISCUSSION:In total, 590 researchers completed the survey. Results from the final regression model indicated that researcher age had a negative association with the misconduct score (b = -0.01, β = -0.22, t = -2.91, p <0.05), suggesting that older researchers tended to report less misconduct. On the other hand, those with more publications had higher misconduct scores (b = 0.001, β = 0.17, t = 3.27, p < 0.05) and, compared with researchers in the region of North America, researchers in Asia tended to have higher misconduct scores (b = 0.21, β = 0.12, t = 2.84, p < 0.01). In addition, compared with those who defined their work role as clinician, those who defined their role as researcher tended to have higher misconduct scores (b = 0.12, β = 0.13, t = 2.15, p < 0.05). Finally, publication pressure emerged as the strongest individual predictor of misconduct (b = 0.20, β = 0.34, t = 7.82, p < 0.01); the greater the publication pressure, the greater the reported misconduct. Overall, the explanatory variables accounted for 21% of the variance in the misconduct score, with publication pressure accounting for 10% of the variance in the outcome, above and beyond the other explanatory variables. Although correlational, these findings suggest several researcher characteristics and practice factors that could be targeted to address scientific misconduct and QRPs in HPE.
| S-EPMC6468038 | biostudies-literature
Project description:We describe a method of quantifying the effect of Questionable Research Practices (QRPs) on the results of meta-analyses. As an example we simulated a meta-analysis of a controversial telepathy protocol to assess the extent to which these experimental results could be explained by QRPs. Our simulations used the same numbers of studies and trials as the original meta-analysis and the frequencies with which various QRPs were applied in the simulated experiments were based on surveys of experimental psychologists. Results of both the meta-analysis and simulations were characterized by 4 metrics, two describing the trial and mean experiment hit rates (HR) of around 31%, where 25% is expected by chance, one the correlation between sample-size and hit-rate, and one the complete P-value distribution of the database. A genetic algorithm optimized the parameters describing the QRPs, and the fitness of the simulated meta-analysis was defined as the sum of the squares of Z-scores for the 4 metrics. Assuming no anomalous effect a good fit to the empirical meta-analysis was found only by using QRPs with unrealistic parameter-values. Restricting the parameter space to ranges observed in studies of QRP occurrence, under the untested assumption that parapsychologists use comparable QRPs, the fit to the published Ganzfeld meta-analysis with no anomalous effect was poor. We allowed for a real anomalous effect, be it unidentified QRPs or a paranormal effect, where the HR ranged from 25% (chance) to 31%. With an anomalous HR of 27% the fitness became F = 1.8 (p = 0.47 where F = 0 is a perfect fit). We conclude that the very significant probability cited by the Ganzfeld meta-analysis is likely inflated by QRPs, though results are still significant (p = 0.003) with QRPs. Our study demonstrates that quantitative simulations of QRPs can assess their impact. Since meta-analyses in general might be polluted by QRPs, this method has wide applicability outside the domain of experimental parapsychology.
| S-EPMC4856278 | biostudies-literature
Project description:The clinical and socioeconomic burden of asthma exacerbations (AEs) constitutes a major public health problem. In the last 4 years, there has been an increase in ethnic diversity in candidate-gene and genome-wide association studies of AEs, which in the latter case led to the identification of novel genes and underlying pathobiological processes. Pharmacogenomics, admixture mapping analyses, and the combination of multiple "omics" layers have helped to prioritize genomic regions of interest and/or facilitated our understanding of the functional consequences of genetic variation. Nevertheless, the field still lags behind the genomics of asthma, where a vast compendium of genetic approaches has been used (eg, gene-environment nteractions, next-generation sequencing, and polygenic risk scores). Furthermore, the roles of the DNA methylome and histone modifications in AEs have received little attention, and microRNA findings remain to be validated in independent studies. Likewise, the most recent transcriptomic studies highlight the importance of the host-airway microbiome interaction in the modulation of risk of AEs. Leveraging -omics and deep-phenotyping data from subtypes or homogenous subgroups of patients will be crucial if we are to overcome the inherent heterogeneity of AEs, boost the identification of potential therapeutic targets, and implement precision medicine approaches to AEs in clinical practice.
| S-EPMC10638677 | biostudies-literature