Dataset Information

Multiplicity Eludes Peer Review: The Case of COVID-19 Research.

ABSTRACT: Multiplicity arises when data analysis involves multiple simultaneous inferences, increasing the chance of spurious findings. It is a widespread problem frequently ignored by researchers. In this paper, we perform an exploratory analysis of the Web of Science database for COVID-19 observational studies. We examined 100 top-cited COVID-19 peer-reviewed articles based on p-values, including up to 7100 simultaneous tests, with 50% including >34 tests, and 20% > 100 tests. We found that the larger the number of tests performed, the larger the number of significant results (r = 0.87, p < 10^-6). The number of p-values in the abstracts was not related to the number of p-values in the papers. However, the highly significant results (p < 0.001) in the abstracts were strongly correlated (r = 0.61, p < 10^-6) with the number of p < 0.001 significances in the papers. Furthermore, the abstracts included a higher proportion of significant results (0.91 vs. 0.50), and 80% reported only significant results. Only one reviewed paper addressed multiplicity-induced type I error inflation, pointing to potentially spurious results bypassing the peer-review process. We conclude the need to pay special attention to the increased chance of false discoveries in observational studies, including non-replicated striking discoveries with a potentially large social impact. We propose some easy-to-implement measures to assess and limit the effects of multiplicity.

SUBMITTER: Gutierrez-Hernandez O

PROVIDER: S-EPMC8430657 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundThe Patient-Centered Outcomes Research Institute (PCORI) is obligated to peer review and to post publicly "Final Research Reports" of all funded projects. PCORI peer review emphasizes adherence to PCORI's Methodology Standards and principles of ethical scientific communication. During the peer review process, reviewers and editors seek to ensure that results are presented objectively and interpreted appropriately, e.g., free of spin.MethodsTwo independent raters assessed PCORI peer review feedback sent to authors. We calculated the proportion of reports in which spin was identified during peer review, and the types of spin identified. We included reports submitted by April 2018 with at least one associated journal article. The same raters then assessed whether authors addressed reviewers' comments about spin. The raters also assessed whether spin identified during PCORI peer review was present in related journal articles.ResultsWe included 64 PCORI-funded projects. Peer reviewers or editors identified spin in 55/64 (86%) submitted research reports. Types of spin included reporting bias (46/55; 84%), inappropriate interpretation (40/55; 73%), inappropriate extrapolation of results (15/55; 27%), and inappropriate attribution of causality (5/55; 9%). Authors addressed comments about spin related to 47/55 (85%) of the reports. Of 110 associated journal articles, PCORI comments about spin were potentially applicable to 44/110 (40%) articles, of which 27/44 (61%) contained the same spin that was identified in the PCORI research report. The proportion of articles with spin was similar for articles accepted before and after PCORI peer review (63% vs 58%).DiscussionJust as spin is common in journal articles and press releases, we found that most reports submitted to PCORI included spin. While most spin was mitigated during the funder's peer review process, we found no evidence that review of PCORI reports influenced spin in journal articles. Funders could explore interventions aimed at reducing spin in published articles of studies they support.

Project description:Weather and climate play a significant role in infectious disease transmission, through changes to transmission dynamics, host susceptibility and virus survival in the environment. Exploring the association of weather variables and COVID-19 transmission is vital in understanding the potential for seasonality and future outbreaks and developing early warning systems. Previous research examined the effects of weather on COVID-19, but the findings appeared inconsistent. This review aims to summarize the currently available literature on the association between weather and COVID-19 incidence and provide possible suggestions for developing weather-based early warning system for COVID-19 transmission. Studies eligible for inclusion used ecological methods to evaluate associations between weather (i.e., temperature, humidity, wind speed and rainfall) and COVID-19 transmission. The review showed that temperature was reported as significant in the greatest number of studies, with COVID-19 incidence increasing as temperature decreased and the highest incidence reported in the temperature range of 0-17 °C. Humidity was also significantly associated with COVID-19 incidence, though the reported results were mixed, with studies reporting positive and negative correlation. A significant interaction between humidity and temperature was also reported. Wind speed and rainfall results were not consistent across studies. Weather variables including temperature and humidity can contribute to increased transmission of COVID-19, particularly in winter conditions through increased host susceptibility and viability of the virus. While there is less indication of an association with wind speed and rainfall, these may contribute to behavioral changes that decrease exposure and risk of infection. Understanding the implications of associations with weather variables and seasonal variations for monitoring and control of future outbreaks is essential for early warning systems.

Dataset Information

Multiplicity Eludes Peer Review: The Case of COVID-19 Research.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure