Dataset Information

Over-optimization of academic publishing metrics: observing Goodhart's Law in action.

ABSTRACT: BACKGROUND:The academic publishing world is changing significantly, with ever-growing numbers of publications each year and shifting publishing patterns. However, the metrics used to measure academic success, such as the number of publications, citation number, and impact factor, have not changed for decades. Moreover, recent studies indicate that these metrics have become targets and follow Goodhart's Law, according to which, "when a measure becomes a target, it ceases to be a good measure." RESULTS:In this study, we analyzed >120 million papers to examine how the academic publishing world has evolved over the last century, with a deeper look into the specific field of biology. Our study shows that the validity of citation-based measures is being compromised and their usefulness is lessening. In particular, the number of publications has ceased to be a good metric as a result of longer author lists, shorter papers, and surging publication numbers. Citation-based metrics, such citation number and h-index, are likewise affected by the flood of papers, self-citations, and lengthy reference lists. Measures such as a journal's impact factor have also ceased to be good metrics due to the soaring numbers of papers that are published in top journals, particularly from the same pool of authors. Moreover, by analyzing properties of >2,600 research fields, we observed that citation-based metrics are not beneficial for comparing researchers in different fields, or even in the same department. CONCLUSIONS:Academic publishing has changed considerably; now we need to reconsider how we measure success.

SUBMITTER: Fire M

PROVIDER: S-EPMC6541803 | biostudies-literature | 2019 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Over-optimization of academic publishing metrics: observing Goodhart's Law in action.

Fire Michael M Guestrin Carlos C

GigaScience 20190601 6

<h4>Background</h4>The academic publishing world is changing significantly, with ever-growing numbers of publications each year and shifting publishing patterns. However, the metrics used to measure academic success, such as the number of publications, citation number, and impact factor, have not changed for decades. Moreover, recent studies indicate that these metrics have become targets and follow Goodhart's Law, according to which, "when a measure becomes a target, it ceases to be a good meas ...[more]

PMID: 31144712

Similar Datasets

Project description:BackgroundTo inform training program development and curricular initiatives, quantitative descriptions of the disciplinary training of research teams publishing in top-tier clinical and epidemiological journals are needed. Our objective was to assess whether interdisciplinary academic training and teamwork of authors publishing original research in 15 top-tier journals varied by year of publication (2000/2010/2020), type of journal (epidemiological/general clinical/specialty clinical), corresponding author gender, and time since the corresponding author completed formal training relative to the article publication date (<5/≥5 years).Methods and findingsWe invited corresponding authors of original research articles to participate in an online survey (n = 103; response rate = 8.3% of 1240 invited authors). In bivariate analyses, year of publication, type of journal, gender, and recency of training were not significantly associated with interdisciplinary team composition, whether a co-author with epidemiological or biostatistical training was involved in any research stage (design/analysis/interpretation/reporting), or with participants' confidence in their own or their co-authors epidemiological or biostatistical expertise (p > 0.05 for each comparison). Exceptions were participants with more recent epidemiological training all had co-author(s) with epidemiological training contribute to study design and interpretation, and participants who published in 2020 were more likely to report being extremely confident in their epidemiological abilities.ConclusionsThis study was the first to quantify interdisciplinary training among research teams publishing in epidemiological and clinical journals. Our quantitative results show research published in top-tier journals generally represents interdisciplinary teamwork and that interdisciplinary training may provide publication type options. Our qualitative results show researchers view interdisciplinary training favorably.

Project description:IntroductionIn academia, many institutions use journal article publication productivity for making decisions on tenure and promotion, funding grants, and rewarding stellar scholars. Although non-alphabetical sequencing of article coauthoring by the spelling of surnames signals the extent to which a scholar has contributed to a project, many disciplines in academia follow the norm of alphabetical ordering of coauthors in journal publications. By assessing business academic publications, this study investigates the hypothesis that author alphabetical ordering disincentivizes teamwork and reduces the overall quality of scholarship.MethodsTo address our objectives, we accessed data from 21,353 articles published over a 20-year period across the four main business subdisciplines. The articles selected are all those published by the four highest-ranked journals (in each year) and four lower-ranked journals (in each year) for accounting, business technology, marketing, and organizational behavior. Poisson regression and binary logistic regression were utilized for hypothesis testing.ResultsThis study finds that, although team size among business scholars is increasing over time, alphabetical ordering as a convention in journal article publishing disincentivizes author teamwork. This disincentive results in fewer authors per publication than for publications using contribution-based ordering of authors. Importantly, article authoring teamwork is related to article quality. Specifically, articles written by a single author typically are of lesser quality than articles published by coauthors, but the number of coauthors exhibits decreasing returns to scale-coauthoring teams of one to three are positively related to high-quality articles, but larger teams are not. Alphabetical ordering itself, however, is positively associated with quality even though it inhibits teamwork, but journal article coauthoring has a greater impact on article quality than does alphabetical ordering.ConclusionsThese findings have important implications for academia. Scholars respond to incentives, yet alphabetical ordering of journal article authors conflicts with what is beneficial for the progress of academic disciplines. Based on these findings, we recommend that, to drive the highest-quality research, teamwork should be incentivized-all fields should adopt a contribution-based journal article author-ordering convention and avoid author ordering based upon the spelling of surnames. Although this study was undertaken using articles from business journals, its findings should generalize across all academia.

Dataset Information

Over-optimization of academic publishing metrics: observing Goodhart's Law in action.

Publications

Over-optimization of academic publishing metrics: observing Goodhart's Law in action.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets