Dataset Information

Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions.

ABSTRACT: We can regard the wider incentive structures that operate across science, such as the priority given to novel findings, as an ecosystem within which scientists strive to maximise their fitness (i.e., publication record and career success). Here, we develop an optimality model that predicts the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study. We show that, for parameter values derived from the scientific literature, researchers acting to maximise their fitness should spend most of their effort seeking novel results and conduct small studies that have only 10%-40% statistical power. As a result, half of the studies they publish will report erroneous conclusions. Current incentive structures are in conflict with maximising the scientific value of research; we suggest ways that the scientific ecosystem could be improved.

SUBMITTER: Higginson AD

PROVIDER: S-EPMC5104444 | biostudies-literature | 2016 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions.

Higginson Andrew D AD Munafò Marcus R MR

PLoS biology 20161110 11

We can regard the wider incentive structures that operate across science, such as the priority given to novel findings, as an ecosystem within which scientists strive to maximise their fitness (i.e., publication record and career success). Here, we develop an optimality model that predicts the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study. W ...[more]

PMID: 27832072

Similar Datasets

Project description:ObjectivesTo provide a participant's account of the development of a paper commissioned by the tobacco industry examining the reliability of self reported smoking status; to redress the distorted report of this Japanese spousal smoking study which evaluated the reliability and validity of self reported smoking status, and estimated confounding by diet and lifestyle factors.DesignRepeated interviews on smoking status and its verification by environmental and biological markers for environmental tobacco smoke (ETS) exposure.SettingUrban wives in Osaka City and Sizuoka City, JapanParticipantsSemi-random sampling of 200 wives in each city. From the Osaka subjects, 100 non-smoking wives were selected for the validity study.Main outcome measuresKappa coefficient for reliability of self reported smoking status. Correlation coefficients between environmental nicotine concentration, cotinine in saliva and urine, and self reported smoking status.ResultsThe kappa coefficient for the repeated interview was high suggesting sufficient reliability of the response. The proportion of self reported current smokers misclassified as non-smokers was equivalent to the misclassified self reported non-smokers. Ambient concentration of nicotine and personal exposure to nicotine correlated with each other and also with salivary cotinine and self reported ETS exposure but not with urinary cotinine/creatinine ratio (CCR). There was no major difference in diet and lifestyle related to husband's smoking status.ConclusionSelf reported smoking status by Japanese wives shows high reliability. It also shows high validity when verified by both nicotine exposure and salivary cotinine, but not by CCR. A previous report questioning the credibility of self reported smoking status, based on questionable CCR, could thus be of dubious validity. In addition, possible dietary and lifestyle confounding factors associated with smoking husbands were not demonstrable, a finding not reported previously. Using all the data from this project changes the conclusion of the previous published report. In addition to the distortion of scientific findings by a tobacco industry affiliated researcher, anti-smoking campaigners made attempts to intimidate and suppress scientific activities. These distortions of science should be counteracted.

Project description:BackgroundMost meta-analyses include data from one or more small studies that, individually, do not have power to detect an intervention effect. The relative influence of adequately powered and underpowered studies in published meta-analyses has not previously been explored. We examine the distribution of power available in studies within meta-analyses published in Cochrane reviews, and investigate the impact of underpowered studies on meta-analysis results.Methods and findingsFor 14,886 meta-analyses of binary outcomes from 1,991 Cochrane reviews, we calculated power per study within each meta-analysis. We defined adequate power as ?50% power to detect a 30% relative risk reduction. In a subset of 1,107 meta-analyses including 5 or more studies with at least two adequately powered and at least one underpowered, results were compared with and without underpowered studies. In 10,492 (70%) of 14,886 meta-analyses, all included studies were underpowered; only 2,588 (17%) included at least two adequately powered studies. 34% of the meta-analyses themselves were adequately powered. The median of summary relative risks was 0.75 across all meta-analyses (inter-quartile range 0.55 to 0.89). In the subset examined, odds ratios in underpowered studies were 15% lower (95% CI 11% to 18%, P<0.0001) than in adequately powered studies, in meta-analyses of controlled pharmacological trials; and 12% lower (95% CI 7% to 17%, P<0.0001) in meta-analyses of controlled non-pharmacological trials. The standard error of the intervention effect increased by a median of 11% (inter-quartile range -1% to 35%) when underpowered studies were omitted; and between-study heterogeneity tended to decrease.ConclusionsWhen at least two adequately powered studies are available in meta-analyses reported by Cochrane reviews, underpowered studies often contribute little information, and could be left out if a rapid review of the evidence is required. However, underpowered studies made up the entirety of the evidence in most Cochrane reviews.

Project description:Wild animals are faced with a broad range of environmental stressors and research is needed to better understand their effect on populations. Hormone analysis based on enzyme immunoassays (EIAs) can provide valuable information on adrenocortical activity (stress), and assessment of cortisol in hair may allow the quantification of cortisol production. To validate hair hormone analysis, we compared two EIAs based on antibodies against cortisol-3-CMO-BSA and cortisol-21-HS-BSA for hair glucocorticoid (hGC) measurements in Egyptian mongoose, Iberian lynx, Alpine marmot, Asiatic black bear, spotted hyena and cheetah, with results obtained by liquid chromatography coupled with tandem mass spectrometry (LC-MS/MS) measurements. Both EIAs were also characterized by HPLC immunograms. Our results revealed that the cortisol-21-HS EIA measured 2.3- to 12-fold higher hGC concentrations than the cortisol-3-CMO assay. In dependence of the species, high-performance liquid chromatography (HPLC) immunograms showed that up to 70% of immunoreactivities determined by the cortisol-21-HS constituted of unknown unpolar compounds leading to an overestimation of hGC. The cortisol-3-CMO EIA expressed a better specificity, with 32.1-67.4% of immunoreactivity represented by cortisol and cortisone. The LC-MS/MS analyses (gold standard) revealed that the cortisol-3-CMO EIA also resulted in an (up to 3-fold) overestimation of hGC, but EIA results were correlated with LC-MS/MS in the mongoose, the lynx, the spotted hyena and the marmot. No correlation was obtained for Asiatic black bears. As a result of our study, we strongly recommend to test any cortisol EIA for its specificity towards extracted hair components. In all analyzed species, except the Asiatic black bear, cortisone and cortisol were simultaneously present in hair extracts; consequently, an appropriate EIA should cross-react to these two glucocorticoid hormones and express negligible affinity towards substances with less polarity than corticosterone. Choosing the wrong EIA for hGC analyses may lead to overestimations of hGC or-in the worst case-to results that do not mirror real adrenocortical activity.

Dataset Information

Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions.

Publications

Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets