Dataset Information

Simple changes of individual studies can improve the reproducibility of the biomedical scientific process as a whole.

ABSTRACT: We developed a new probabilistic model to assess the impact of recommendations rectifying the reproducibility crisis (by publishing both positive and 'negative' results and increasing statistical power) on competing objectives, such as discovering causal relationships, avoiding publishing false positive results, and reducing resource consumption. In contrast to recent publications our model quantifies the impact of each single suggestion not only for an individual study but especially their relation and consequences for the overall scientific process. We can prove that higher-powered experiments can save resources in the overall research process without generating excess false positives. The better the quality of the pre-study information and its exploitation, the more likely this beneficial effect is to occur. Additionally, we quantify the adverse effects of both neglecting good practices in the design and conduct of hypotheses-based research, and the omission of the publication of 'negative' findings. Our contribution is a plea for adherence to or reinforcement of the good scientific practice and publication of 'negative' findings.

SUBMITTER: Steinfath M

PROVIDER: S-EPMC6135363 | biostudies-literature | 2018

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Simple changes of individual studies can improve the reproducibility of the biomedical scientific process as a whole.

Steinfath Matthias M Vogl Silvia S Violet Norman N Schwarz Franziska F Mielke Hans H Selhorst Thomas T Greiner Matthias M Schönfelder Gilbert G

PloS one 20180912 9

We developed a new probabilistic model to assess the impact of recommendations rectifying the reproducibility crisis (by publishing both positive and 'negative' results and increasing statistical power) on competing objectives, such as discovering causal relationships, avoiding publishing false positive results, and reducing resource consumption. In contrast to recent publications our model quantifies the impact of each single suggestion not only for an individual study but especially their rela ...[more]

PMID: 30208060

Similar Datasets

Project description:Simple interventions resolve the problem of missed lab appointments. It is essential that patients complete ordered laboratory studies. This maintains clinical quality and, potentially, keeps patients safe from harm. In our academic family medicine practice, baseline data demonstrated patients completed 94.7% of labs as ordered (26850/28348 patients per year) while 1498 (5.3%) did not. Our baseline patient reminder process, a mail or portal based generic letter, resulted in only 449 (30%) of patients ultimately completing them (1049 [70%] did not). Our baseline system was 96.3 % reliable. This process did not allow for provider review or input, and was not personalized for patients. We designed a quality improvement project involving three PDSA (Plan, Do, Study, and Act) cycles of about two months each. Desk staff created weekly reports of unresolved lab orders. A message in the electronic medical record (EMR) solicited provider input. Providers could elect to cancel studies (if already completed, reordered, or no longer clinically indicated) or have the patient receive a personalized reminder, including provider name and associated diagnoses. This reminder was sent by patient portal secure messaging (if an account existed) or with a mailed letter. These interventions resulted in 98.8 % process reliability. The frequency of unresolved lab orders decreased from 70% at baseline to 25%. In the second PDSA cycle, we contacted patients by the portal only if there was evidence of an active account. Otherwise, they were contacted by telephone. Patients without a portal account continued to receive a letter by mail. These modified processes resulted in an overall reliability rate of 99.2%. The frequency of unresolved lab orders decreased to 17%. A final PDSA cycle utilized only telephone contact with patients with unresolved lab orders. Schedulers offered patients a choice of appointment dates if they spoke personally. Otherwise, they were left messages with a future lab appointment date two weeks later.Overall process reliability now increased to 100%. The frequency of unresolved lab orders decreased to 0%. Our interventions resulted in increased system reliability.Provider input was not perceived as burdensome. Desk staff work effort was not increased.Telephone patient contact resulted in more frequent lab order completion than other methods.

Project description:We conducted an international cross-sectional survey of biomedical researchers' perspectives on the reproducibility of research. This study builds on a widely cited 2016 survey on reproducibility and provides a biomedical-specific and contemporary perspective on reproducibility. To sample the community, we randomly selected 400 journals indexed in MEDLINE, from which we extracted the author names and emails from all articles published between October 1, 2020 and October 1, 2021. We invited participants to complete an anonymous online survey which collected basic demographic information, perceptions about a reproducibility crisis, perceived causes of irreproducibility of research results, experience conducting reproducibility studies, and knowledge of funding and training for research on reproducibility. A total of 1,924 participants accessed our survey, of which 1,630 provided useable responses (response rate 7% of 23,234). Key findings include that 72% of participants agreed there was a reproducibility crisis in biomedicine, with 27% of participants indicating the crisis was "significant." The leading perceived cause of irreproducibility was a "pressure to publish" with 62% of participants indicating it "always" or "very often" contributes. About half of the participants (54%) had run a replication of their own previously published study while slightly more (57%) had run a replication of another researcher's study. Just 16% of participants indicated their institution had established procedures to enhance the reproducibility of biomedical research and 67% felt their institution valued new research over replication studies. Participants also reported few opportunities to obtain funding to attempt to reproduce a study and 83% perceived it would be harder to do so than to get funding to do a novel study. Our results may be used to guide training and interventions to improve research reproducibility and to monitor rates of reproducibility over time. The findings are also relevant to policy makers and academic leadership looking to create incentives and research cultures that support reproducibility and value research quality.

Project description:BackgroundJupyter notebooks facilitate the bundling of executable code with its documentation and output in one interactive environment, and they represent a popular mechanism to document and share computational workflows, including for research publications. The reproducibility of computational aspects of research is a key component of scientific reproducibility but has not yet been assessed at scale for Jupyter notebooks associated with biomedical publications.ApproachWe address computational reproducibility at 2 levels: (i) using fully automated workflows, we analyzed the computational reproducibility of Jupyter notebooks associated with publications indexed in the biomedical literature repository PubMed Central. We identified such notebooks by mining the article's full text, trying to locate them on GitHub, and attempting to rerun them in an environment as close to the original as possible. We documented reproduction success and exceptions and explored relationships between notebook reproducibility and variables related to the notebooks or publications. (ii) This study represents a reproducibility attempt in and of itself, using essentially the same methodology twice on PubMed Central over the course of 2 years, during which the corpus of Jupyter notebooks from articles indexed in PubMed Central has grown in a highly dynamic fashion.ResultsOut of 27,271 Jupyter notebooks from 2,660 GitHub repositories associated with 3,467 publications, 22,578 notebooks were written in Python, including 15,817 that had their dependencies declared in standard requirement files and that we attempted to rerun automatically. For 10,388 of these, all declared dependencies could be installed successfully, and we reran them to assess reproducibility. Of these, 1,203 notebooks ran through without any errors, including 879 that produced results identical to those reported in the original notebook and 324 for which our results differed from the originally reported ones. Running the other notebooks resulted in exceptions.ConclusionsWe zoom in on common problems and practices, highlight trends, and discuss potential improvements to Jupyter-related workflows associated with biomedical publications.

Dataset Information

Simple changes of individual studies can improve the reproducibility of the biomedical scientific process as a whole.

Publications

Simple changes of individual studies can improve the reproducibility of the biomedical scientific process as a whole.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets