Dataset Information

Science is not a signal detection problem.

ABSTRACT: The perceived replication crisis and the reforms designed to address it are grounded in the notion that science is a binary signal detection problem. However, contrary to null hypothesis significance testing (NHST) logic, the magnitude of the underlying effect size for a given experiment is best conceptualized as a random draw from a continuous distribution, not as a random draw from a dichotomous distribution (null vs. alternative). Moreover, because continuously distributed effects selected using a P < 0.05 filter must be inflated, the fact that they are smaller when replicated (reflecting regression to the mean) is no reason to sound the alarm. Considered from this perspective, recent replication efforts suggest that most published P < 0.05 scientific findings are "true" (i.e., in the correct direction), with observed effect sizes that are inflated to varying degrees. We propose that original science is a screening process, one that adopts NHST logic as a useful fiction for selecting true effects that are potentially large enough to be of interest to other scientists. Unlike original science, replication science seeks to precisely measure the underlying effect size associated with an experimental protocol via large-N direct replication, without regard for statistical significance. Registered reports are well suited to (often resource-intensive) direct replications, which should focus on influential findings and be published regardless of outcome. Conceptual replications play an important but separate role in validating theories. However, because they are part of NHST-based original science, conceptual replications cannot serve as the field's self-correction mechanism. Only direct replications can do that.

SUBMITTER: Wilson BM

PROVIDER: S-EPMC7084063 | biostudies-literature | 2020 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Science is not a signal detection problem.

Wilson Brent M BM Harris Christine R CR Wixted John T JT

Proceedings of the National Academy of Sciences of the United States of America 20200303 11

The perceived replication crisis and the reforms designed to address it are grounded in the notion that science is a binary signal detection problem. However, contrary to null hypothesis significance testing (NHST) logic, the magnitude of the underlying effect size for a given experiment is best conceptualized as a random draw from a continuous distribution, not as a random draw from a dichotomous distribution (null vs. alternative). Moreover, because continuously distributed effects selected us ...[more]

PMID: 32127477

Similar Datasets

Project description:Smoke Sense is a citizen science project with investigative, educational, and action-oriented objectives at the intersection of wildland fire smoke and public health. Participants engage with a smartphone application to explore current and forecast visualizations of air quality, learn about how to protect health from wildfire smoke, and record their smoke experiences, health symptoms, and behaviors taken to reduce their exposures to smoke. Through participation in the project, individuals engage in observing changes in their environment and recording changes in their health, thus facilitating progression on awareness of health effects of air pollution and adoption of desired health-promoting behaviors. Participants can also view what others are reporting. Data from the pilot season (1 August 2017 to 7 January 2018; 5,598 downloads) suggest that there is a clear demand for personally relevant data during wildfire episodes motivated by recognition of environmental hazard and the personal concern for health. However, while participants shared clear perceptions of the environmental hazard and health risks in general, they did not consistently recognize their own personal health risk. The engagement in health protective behavior was driven in response to symptoms rather than as preventive courses of action. We also observed clear differences in the adoption likelihood of various health protective behaviors attributed to barriers and perceived benefits of these actions. As users experience a greater number and severity of symptoms, the perceived benefits of taking health protective actions exceeded the costs associated with the barriers and thus increased adoption of those actions. Based on pilot season data, we summarize key insights which may improve current health risk communications in nudging individuals toward health protective behavior; there is a need to increase personal awareness of risk and compelling evidence that health protective behaviors are beneficial.

Dataset Information

Science is not a signal detection problem.

Publications

Science is not a signal detection problem.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets