Opinion: Is science really facing a reproducibility crisis, and do we need it to?
Ontology highlight
ABSTRACT:
SUBMITTER: Fanelli D
PROVIDER: S-EPMC5856498 | biostudies-literature | 2018 Mar
REPOSITORIES: biostudies-literature
Publications
Opinion: Is science really facing a reproducibility crisis, and do we need it to?
Proceedings of the National Academy of Sciences of the United States of America 20180301 11
Efforts to improve the reproducibility and integrity of science are typically justified by a narrative of crisis, according to which most published results are unreliable due to growing problems with research and publication practices. This article provides an overview of recent evidence suggesting that this narrative is mistaken, and argues that a narrative of epochal changes and empowerment of scientists would be more accurate, inspiring, and compelling. ...[more]
Similar Datasets
Project description:The opioid epidemic in the United States has had a devastating impact on millions of people as well as on their families and communities. The increased prevalence of opioid misuse, use disorder and overdose in recent years has highlighted the need for improved public health approaches for reducing the tremendous harms of this illness. In this paper, we explain and call for the need for more systems science approaches, which can uncover the complexities of the opioid crisis, and help evaluate, analyse and forecast the effectiveness of ongoing and new policy interventions. Similar to how a stream of systems science research helped policy development in infectious diseases and obesity, more systems science research is needed in opioids.
| S-EPMC7414582 | biostudies-literature
Project description:A reproducibility crisis is a situation where many scientific studies cannot be reproduced. Inappropriate practices of science, such as HARKing, p-hacking, and selective reporting of positive results, have been suggested as causes of irreproducibility. In this editorial, I propose that a lack of raw data or data fabrication is another possible cause of irreproducibility.As an Editor-in-Chief of Molecular Brain, I have handled 180 manuscripts since early 2017 and have made 41 editorial decisions categorized as "Revise before review," requesting that the authors provide raw data. Surprisingly, among those 41 manuscripts, 21 were withdrawn without providing raw data, indicating that requiring raw data drove away more than half of the manuscripts. I rejected 19 out of the remaining 20 manuscripts because of insufficient raw data. Thus, more than 97% of the 41 manuscripts did not present the raw data supporting their results when requested by an editor, suggesting a possibility that the raw data did not exist from the beginning, at least in some portions of these cases.Considering that any scientific study should be based on raw data, and that data storage space should no longer be a challenge, journals, in principle, should try to have their authors publicize raw data in a public database or journal site upon the publication of the paper to increase reproducibility of the published results and to increase public trust in science.
| S-EPMC7033918 | biostudies-literature
Project description:Investigators from a large consortium of scientists recently performed a multi-year study in which they replicated 100 psychology experiments. Although statistically significant results were reported in 97% of the original studies, statistical significance was achieved in only 36% of the replicated studies. This article presents a reanalysis of these data based on a formal statistical model that accounts for publication bias by treating outcomes from unpublished studies as missing data, while simultaneously estimating the distribution of effect sizes for those studies that tested nonnull effects. The resulting model suggests that more than 90% of tests performed in eligible psychology experiments tested negligible effects, and that publication biases based on p-values caused the observed rates of nonreproducibility. The results of this reanalysis provide a compelling argument for both increasing the threshold required for declaring scientific discoveries and for adopting statistical summaries of evidence that account for the high proportion of tested hypotheses that are false. Supplementary materials for this article are available online.
| S-EPMC5976261 | biostudies-literature
Project description:Increasingly, science diaspora networks are managed by formal organizations such as embassies or non-profit organizations. Researchers have studied these networks to understand how they influence international collaborations and science diplomacy, and to determine which network activities foster those outcomes and which do not. In this perspective, we suggest that many of these network organizations confront an underappreciated conundrum for managing resources: organizations with few resources must learn how to obtain more resources despite lacking means to do so. To substantiate our suggestion, we do the following. We review exploratory results from a study of network organizations that indicate that these organizations generally lack resources, learn too little from each other, and struggle to overcome the resource conundrum. We also show that this conundrum is expected from organizational theory based on bounded rationality. To help organizations confront the issue, we do the following. First we provide a new database of operating science diaspora networks. We encourage managers of network organizations to use it as a resource to identify peers with whom to regularly exchange knowledge about securing resources. We also suggest that other scientific organizations should infuse network organizations with fresh resources. Ultimately, we urge all relevant stakeholders to recognize that the conundrum results not from the shortcomings of individual managers, but rather is a legitimate organizational phenomena that must be addressed by organizational design.
| S-EPMC9278311 | biostudies-literature
Project description:Electrical brain stimulation (EBS) is a trendy new technique used to change brain function and treat neurological, psychiatric and psychological disorders. We were curious whether the published literature, which is dominated by positive results, reflects the experience of researchers using EBS. Specifically, we wanted to know whether researchers are able to reproduce published EBS effects and whether they engage in, but fail to report, questionable research practices. We invited 976 researchers to complete an online survey. We also audited 100 randomly-selected published EBS papers. A total of 154 researchers completed the survey. Survey respondents had a median of 3 [1 to 6, IQR] published EBS papers (1180 total) and 2 [1 to 3] unpublished ones (380 total). With anodal and cathodal EBS, the two most widely used techniques, 45-50% of researchers reported being able to routinely reproduce published results. When asked about how study sample size was determined, 69% of respondents reported using the sample size of published studies, while 61% had used power calculations, and 32% had based their decision on pilot data. In contrast, our audit found only 6 papers where power calculations were used and a single paper in which pilot data was used. When asked about questionable research practices, survey respondents were aware of other researchers who selectively reported study outcomes (41%) and experimental conditions (36%), adjusted statistical analysis to optimise results (43%), and engaged in other shady practices (20%). Fewer respondents admitted to engaging in these practices themselves, although 25% admitted to adjusting statistical analysis to optimize results. There was strong agreement that such practices should be reported in research papers; however, our audit found only two such admissions. The present survey confirms that questionable research practices and poor reproducibility are present in EBS studies. The belief that EBS is effective needs to be replaced by a more rigorous approach so that reproducible brain stimulation methods can be devised and applied.
| S-EPMC5405934 | biostudies-literature
Project description:There is a clear demand for quantitative literacy in the life sciences, necessitating competent instructors in higher education. However, not all instructors are versed in data science skills or research-based teaching practices. We surveyed biological and environmental science instructors (n = 106) about the teaching of data science in higher education, identifying instructor needs and illuminating barriers to instruction. Our results indicate that instructors use, teach, and view data management, analysis, and visualization as important data science skills. Coding, modeling, and reproducibility were less valued by the instructors, although this differed according to institution type and career stage. The greatest barriers were instructor and student background and space in the curriculum. The instructors were most interested in training on how to teach coding and data analysis. Our study provides an important window into how data science is taught in higher education biology programs and how we can best move forward to empower instructors across disciplines.
| S-EPMC8634500 | biostudies-literature
Project description:Computers, the Internet and social media enable every individual to be a publisher, communicating true or false information instantly and globally. In the 'post-truth' era, deception is commonplace at all levels of contemporary life. Fakery affects science and social information and the two have become highly interactive globally, undermining trust in science and the capacity of individuals and society to make evidence-informed choices, including on life-or-death issues. Ironically, drivers of fake science are embedded in the current science publishing system intended to disseminate evidenced knowledge, in which the intersection of science advancement and reputational and financial rewards for scientists and publishers incentivize gaming and, in the extreme, creation and promotion of falsified results. In the battle for truth, individual scientists, professional associations, academic institutions and funding bodies must act to put their own house in order by promoting ethics and integrity and de-incentivizing the production and publishing of false data and results. They must speak out against false information and fake science in circulation and forcefully contradict public figures who promote it. They must contribute to research that helps understand and counter false information, to education that builds knowledge and skills in assessing information and to strengthening science literacy in society.
| S-EPMC6549953 | biostudies-literature
Project description:BACKGROUND:Reproducibility is the hallmark of good science. Maintaining a high degree of transparency in scientific reporting is essential not just for gaining trust and credibility within the scientific community but also for facilitating the development of new ideas. Sharing data and computer code associated with publications is becoming increasingly common, motivated partly in response to data deposition requirements from journals and mandates from funders. Despite this increase in transparency, it is still difficult to reproduce or build upon the findings of most scientific publications without access to a more complete workflow. FINDINGS:Version control systems (VCS), which have long been used to maintain code repositories in the software industry, are now finding new applications in science. One such open source VCS, Git, provides a lightweight yet robust framework that is ideal for managing the full suite of research outputs such as datasets, statistical code, figures, lab notes, and manuscripts. For individual researchers, Git provides a powerful way to track and compare versions, retrace errors, explore new approaches in a structured manner, while maintaining a full audit trail. For larger collaborative efforts, Git and Git hosting services make it possible for everyone to work asynchronously and merge their contributions at any time, all the while maintaining a complete authorship trail. In this paper I provide an overview of Git along with use-cases that highlight how this tool can be leveraged to make science more reproducible and transparent, foster new collaborations, and support novel uses.
| S-EPMC3639880 | biostudies-literature
Project description:PurposeTo evaluate whether an arterial phase scan improves the diagnostic performance of computed tomography to identify pelvic trauma patients who received angiographic intervention on demand of the trauma surgeon.MethodsThis retrospective single-center study was performed at an academic Scandinavian trauma center with approximately 2000 trauma admissions annually. Pelvic trauma patients with arterial and portal venous phase CT from 2009 to 2015 were included. The patients were identified from the institutional trauma registry. Images were interpreted by two radiologists with more than 10 years of trauma radiology experience. Positive findings for extravasation on portal venous phase alone or on both arterial and portal venous phase were compared, with angiographic intervention as clinical outcome.ResultsOne hundred fifty-seven patients (54 females, 103 males) with a median age of 45 years were enrolled. Sixteen patients received angiographic intervention. Positive CT findings on portal venous phase only had a sensitivity and specificity of 62% and 86%, vs. 56% and 93% for simultaneous findings on arterial and portal venous phase. Specificity was significantly higher for positive findings in both phases compared with portal venous phase only. Applying a threshold > 0.9 cm of extravasation diameter to portal venous phase only resulted in sensitivity and specificity identical to those of both phases.ConclusionArterial phase scan in addition to portal venous phase scan did not improve patient selection for angiography. Portal venous phase extravasation size alone may be used as an imaging-based biomarker of the need for angiographic intervention.
| S-EPMC7835176 | biostudies-literature
Project description:A striking contrast runs through the last 60 years of biopharmaceutical discovery, research, and development. Huge scientific and technological gains should have increased the quality of academic science and raised industrial R&D efficiency. However, academia faces a "reproducibility crisis"; inflation-adjusted industrial R&D costs per novel drug increased nearly 100 fold between 1950 and 2010; and drugs are more likely to fail in clinical development today than in the 1970s. The contrast is explicable only if powerful headwinds reversed the gains and/or if many "gains" have proved illusory. However, discussions of reproducibility and R&D productivity rarely address this point explicitly. The main objectives of the primary research in this paper are: (a) to provide quantitatively and historically plausible explanations of the contrast; and (b) identify factors to which R&D efficiency is sensitive. We present a quantitative decision-theoretic model of the R&D process. The model represents therapeutic candidates (e.g., putative drug targets, molecules in a screening library, etc.) within a "measurement space", with candidates' positions determined by their performance on a variety of assays (e.g., binding affinity, toxicity, in vivo efficacy, etc.) whose results correlate to a greater or lesser degree. We apply decision rules to segment the space, and assess the probability of correct R&D decisions. We find that when searching for rare positives (e.g., candidates that will successfully complete clinical development), changes in the predictive validity of screening and disease models that many people working in drug discovery would regard as small and/or unknowable (i.e., an 0.1 absolute change in correlation coefficient between model output and clinical outcomes in man) can offset large (e.g., 10 fold, even 100 fold) changes in models' brute-force efficiency. We also show how validity and reproducibility correlate across a population of simulated screening and disease models. We hypothesize that screening and disease models with high predictive validity are more likely to yield good answers and good treatments, so tend to render themselves and their diseases academically and commercially redundant. Perhaps there has also been too much enthusiasm for reductionist molecular models which have insufficient predictive validity. Thus we hypothesize that the average predictive validity of the stock of academically and industrially "interesting" screening and disease models has declined over time, with even small falls able to offset large gains in scientific knowledge and brute-force efficiency. The rate of creation of valid screening and disease models may be the major constraint on R&D productivity.
| S-EPMC4749240 | biostudies-literature