Dataset Information

The assessment of science: the relative merits of post-publication review, the impact factor, and the number of citations.

ABSTRACT: The assessment of scientific publications is an integral part of the scientific process. Here we investigate three methods of assessing the merit of a scientific paper: subjective post-publication peer review, the number of citations gained by a paper, and the impact factor of the journal in which the article was published. We investigate these methods using two datasets in which subjective post-publication assessments of scientific publications have been made by experts. We find that there are moderate, but statistically significant, correlations between assessor scores, when two assessors have rated the same paper, and between assessor score and the number of citations a paper accrues. However, we show that assessor score depends strongly on the journal in which the paper is published, and that assessors tend to over-rate papers published in journals with high impact factors. If we control for this bias, we find that the correlation between assessor scores and between assessor score and the number of citations is weak, suggesting that scientists have little ability to judge either the intrinsic merit of a paper or its likely impact. We also show that the number of citations a paper receives is an extremely error-prone measure of scientific merit. Finally, we argue that the impact factor is likely to be a poor measure of merit, since it depends on subjective assessment. We conclude that the three measures of scientific merit considered here are poor; in particular subjective assessments are an error-prone, biased, and expensive method by which to assess merit. We argue that the impact factor may be the most satisfactory of the methods we have considered, since it is a form of pre-publication review. However, we emphasise that it is likely to be a very error-prone measure of merit that is qualitative, not quantitative.

SUBMITTER: Eyre-Walker A

PROVIDER: S-EPMC3792863 | biostudies-literature | 2013 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

The assessment of science: the relative merits of post-publication review, the impact factor, and the number of citations.

Eyre-Walker Adam A Stoletzki Nina N

PLoS biology 20131008 10

The assessment of scientific publications is an integral part of the scientific process. Here we investigate three methods of assessing the merit of a scientific paper: subjective post-publication peer review, the number of citations gained by a paper, and the impact factor of the journal in which the article was published. We investigate these methods using two datasets in which subjective post-publication assessments of scientific publications have been made by experts. We find that there are ...[more]

PMID: 24115908

Similar Datasets

Project description:Evidence-based decision-making is most effective with comprehensive access to scientific studies. If studies face significant publication delays or barriers, the useful information they contain may not reach decision-makers in a timely manner. This represents a potential problem for mission-oriented disciplines where access to the latest data is required to ensure effective actions are undertaken. We sought to analyse the severity of publication delay in conservation science-a field that requires urgent action to prevent the loss of biodiversity. We used the Conservation Evidence database to assess the length of publication delay (time from finishing data collection to publication) in the literature that tests the effectiveness of conservation interventions. From 7,447 peer-reviewed and non-peer-reviewed studies of conservation interventions published over eleven decades, we find that the raw mean publication delay was 3.2 years (±2SD = 0.1) and varied by conservation subject. A significantly shorter delay was observed for studies focused on Bee Conservation, Sustainable Aquaculture, Management of Captive Animals, Amphibian Conservation, and Control of Freshwater Invasive Species (Estimated Marginal Mean range from 1.4-1.9 years). Publication delay was significantly shorter for the non-peer-reviewed literature (Estimated Marginal Mean delay of 1.9 years ± 0.2) compared to the peer-reviewed literature (i.e., scientific journals; Estimated Marginal Mean delay of 3.0 years ± 0.1). We found publication delay has significantly increased over time (an increase of ~1.2 years from 1912 (1.4 years ± 0.2) to 2020 (2.6 years ± 0.1)), but this change was much weaker and non-significant post-2000s; we found no evidence for any decline. There was also no evidence that studies on more threatened species were subject to a shorter delay-indeed, the contrary was true for mammals, and to a lesser extent for birds. We suggest a range of possible ways in which scientists, funders, publishers, and practitioners can work together to reduce delays at each stage of the publication process.

Project description:BACKGROUND:Social media coverage is increasingly used to spread the message of scientific publications. Traditionally, the scientific impact of an article is measured by the number of citations. At a journal level, this conventionally matures over a 2-year period, and it is challenging to gauge impact around the time of publication. OBJECTIVE:We, therefore, aimed to assess whether Web-based attention is associated with citations and to develop a predictive model that assigns relative importance to different elements of social media coverage: the #SoME_Impact score. METHODS:We included all original articles published in 2015 in a selection of the highest impact journals: The New England Journal of Medicine, The Lancet, the Journal of the American Medical Association, Nature, Cell, and Science. We first characterized the change in Altmetric score over time by taking a single month's sample of recently published articles from the same journals and gathered Altmetric data daily from the time of publication to create a mixed effects spline model. We then obtained the overall weighted Altmetric score for all articles from 2015, the unweighted data for each Altmetric component, and the 2-year citation count from Scopus for each of these articles from 2016 to 2017. We created a stepwise multivariable linear regression model to develop a #SoME_Score that was predictive of 2-year citations. The score was validated using a dataset of articles from the same journals published in 2016. RESULTS:In our unselected sample of 145 recently published articles, social media coverage appeared to plateau approximately 14 days after publication. A total of 3150 articles with a median citation count of 16 (IQR 5-33) and Altmetric score of 72 (IQR 28-169) were included for analysis. On multivariable regression, compared with articles in the lowest quantile of #SoME_Score, articles in the second, third, and upper quantiles had 0.81, 15.20, and 87.67 more citations, respectively. On the validation dataset, #SoME_Score model outperformed the Altmetric score (adjusted R2 0.19 vs 0.09; P<.001). Articles in the upper quantile of #SoME_Score were more than 5 times more likely to be among the upper quantile of those cites (odds ratio 5.61, 95% CI 4.70-6.73). CONCLUSIONS:Social media attention predicts citations and could be used as an early surrogate measure of scientific impact. Owing to the cross-sectional study design, we cannot determine whether correlation relates to causation.

Dataset Information

The assessment of science: the relative merits of post-publication review, the impact factor, and the number of citations.

Publications

The assessment of science: the relative merits of post-publication review, the impact factor, and the number of citations.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets