Reputation or peer review? The role of outliers.
Ontology highlight
ABSTRACT:
SUBMITTER: Grimaldo F
PROVIDER: S-EPMC6096687 | biostudies-other | 2018
REPOSITORIES: biostudies-other
Publications
Reputation or peer review? The role of outliers.
Grimaldo Francisco F Paolucci Mario M Sabater-Mir Jordi J
Scientometrics 20180709 3
We present an agent-based model of paper publication and consumption that allows to study the effect of two different evaluation mechanisms, peer review and reputation, on the quality of the manuscripts accessed by a scientific community. The model was empirically calibrated on two data sets, mono- and multi-disciplinary. Our results point out that disciplinary settings differ in the rapidity with which they deal with extreme events-papers that have an extremely high quality, that we call outlie ...[more]
Similar Datasets
Project description:Recent empirical evidence shows that working in an unsupervised, isolated situation under competition, can increase dishonest behavior to achieve prestige. However, could working in a common space, in the presence of colleagues affect cheating? Here, we examine how familiar-peer influence, supervision and social incentives affect worker performance and dishonest behavior. First, we show that working in the presence of peers is an effective mechanism to constrain honest/dishonest behavior compared to an isolated work situation (experiment 1). Second, we demonstrate that the mere suspicion of dishonesty from another peer is not enough to affect individual cheating behavior (experiment 2), suggesting that reputation holds great importance in a worker's self-image acting as a strong social incentives. Third, we show that when the suspicion of dishonesty increases with multiple peers behaving dishonestly, the desire to increase standing is sufficient to nudge individuals' behavior back to cheating at the same levels as isolated situations (experiment 3).
| S-EPMC4390288 | biostudies-literature
Project description:Peer review is the gold standard for scientific communication, but its ability to guarantee the quality of published research remains difficult to verify. Recent modeling studies suggest that peer review is sensitive to reviewer misbehavior, and it has been claimed that referees who sabotage work they perceive as competition may severely undermine the quality of publications. Here we examine which aspects of suboptimal reviewing practices most strongly impact quality, and test different mitigating strategies that editors may employ to counter them. We find that the biggest hazard to the quality of published literature is not selfish rejection of high-quality manuscripts but indifferent acceptance of low-quality ones. Bypassing or blacklisting bad reviewers and consulting additional reviewers to settle disagreements can reduce but not eliminate the impact. The other editorial strategies we tested do not significantly improve quality, but pairing manuscripts to reviewers unlikely to selfishly reject them and allowing revision of rejected manuscripts minimize rejection of above-average manuscripts. In its current form, peer review offers few incentives for impartial reviewing efforts. Editors can help, but structural changes are more likely to have a stronger impact.
| S-EPMC5633172 | biostudies-other
Project description:BackgroundWe sought to develop a more reliable structured implicit chart review instrument for use in assessing the quality of care for chronic disease and to examine if ratings are more reliable for conditions in which the evidence base for practice is more developed.MethodsWe conducted a reliability study in a cohort with patient records including both outpatient and inpatient care as the objects of measurement. We developed a structured implicit review instrument to assess the quality of care over one year of treatment. 12 reviewers conducted a total of 496 reviews of 70 patient records selected from 26 VA clinical sites in two regions of the country. Each patient had between one and four conditions specified as having a highly developed evidence base (diabetes and hypertension) or a less developed evidence base (chronic obstructive pulmonary disease or a collection of acute conditions). Multilevel analysis that accounts for the nested and cross-classified structure of the data was used to estimate the signal and noise components of the measurement of quality and the reliability of implicit review.ResultsFor COPD and a collection of acute conditions the reliability of a single physician review was quite low (intra-class correlation = 0.16-0.26) but comparable to most previously published estimates for the use of this method in inpatient settings. However, for diabetes and hypertension the reliability is significantly higher at 0.46. The higher reliability is a result of the reviewers collectively being able to distinguish more differences in the quality of care between patients (p < 0.007) and not due to less random noise or individual reviewer bias in the measurement. For these conditions the level of true quality (i.e. the rating of quality of care that would result from the full population of physician reviewers reviewing a record) varied from poor to good across patients.ConclusionsFor conditions with a well-developed quality of care evidence base, such as hypertension and diabetes, a single structured implicit review to assess the quality of care over a period of time is moderately reliable. This method could be a reasonable complement or alternative to explicit indicator approaches for assessing and comparing quality of care. Structured implicit review, like explicit quality measures, must be used more cautiously for illnesses for which the evidence base is less well developed, such as COPD and acute, short-course illnesses.
| S-EPMC434524 | biostudies-literature
Project description:Publishing peer review materials alongside research articles promises to make the peer review process more transparent as well as making it easier to recognise these contributions and give credit to peer reviewers. Traditionally, the peer review reports, editors letters and author responses are only shared between the small number of people in those roles prior to publication, but there is a growing interest in making some or all of these materials available. A small number of journals have been publishing peer review materials for some time, others have begun this practice more recently, and significantly more are now considering how they might begin. This article outlines the outcomes from a recent workshop among journals with experience in publishing peer review materials, in which the specific operation of these workflows, and the challenges, were discussed. Here, we provide a draft as to how to represent these materials in the JATS and Crossref data models to facilitate the coordination and discoverability of peer review materials, and seek feedback on these initial recommendations.
| S-EPMC6206614 | biostudies-literature
Project description:Open peer review (OPR), as with other elements of open science and open research, is on the rise. It aims to bring greater transparency and participation to formal and informal peer review processes. But what is meant by `open peer review', and what advantages and disadvantages does it have over standard forms of review? How do authors or reviewers approach OPR? And what pitfalls and opportunities should you look out for? Here, we propose ten considerations for OPR, drawing on discussions with authors, reviewers, editors, publishers and librarians, and provide a pragmatic, hands-on introduction to these issues. We cover basic principles and summarise best practices, indicating how to use OPR to achieve best value and mutual benefits for all stakeholders and the wider research community.
| S-EPMC6073088 | biostudies-other
Project description:The field of law has retained its distinctiveness regarding peer review to this day, and reviews are often conducted without following standardized rules and principles. External and independent evaluation of submissions has recently become adopted by European law journals, and peer review procedures are still poorly defined, investigated, and attuned to the legal science publishing landscape. The aim of our study was to gain a better insight into current editorial policies on peer review in law journals by exploring editorial documents (instructions, guidelines, policies) issued by 119 Croatian, Italian, and Spanish law journals. We relied on automatic content analysis of 135 publicly available documents collected from the journal websites to analyze the basic features of the peer review processes, manuscript evaluation criteria, and related ethical issues using WordStat8. Differences in covered topics between the countries were compared using the chi-square test. Our findings reveal that most law journals have adopted a traditional approach, in which the editorial board manages mostly anonymized peer review (104, 77%) engaging independent/external reviewers (65, 48%). Submissions are evaluated according to their originality and relevance (113, 84%), quality of writing and presentation (94, 70%), comprehensiveness of literature references (93, 69%), and adequacy of methods (57, 42%). The main ethical issues related to peer review addressed by these journals are reviewer's competing interests (42, 31%), plagiarism (35, 26%), and biases (30, 22%). We observed statistically significant differences between countries in mentioning key concepts such as "Peer review ethics", "Reviewer", "Transparency of identities", "Publication type", and "Research misconduct". Spanish journals favor reviewers' "Independence" and "Competence" and "Anonymized" peer review process. Also, some manuscript types popular in one country are rarely mentioned in other countries. Even though peer review is equally conventional in all three countries, high transparency in Croatian law journals, respect for research integrity in Spanish ones, and diversity and inclusion in Italian are promising indicators of future development.
| S-EPMC8692876 | biostudies-literature
Project description:Diabetes mellitus is a group of metabolic diseases in which blood sugar levels are too high. About 8.8% of the world was diabetic in 2017. It is projected that this will reach nearly 10% by 2045. The major challenge is that when machine learning-based classifiers are applied to such data sets for risk stratification, leads to lower performance. Thus, our objective is to develop an optimized and robust machine learning (ML) system under the assumption that missing values or outliers if replaced by a median configuration will yield higher risk stratification accuracy. This ML-based risk stratification is designed, optimized and evaluated, where: (i) the features are extracted and optimized from the six feature selection techniques (random forest, logistic regression, mutual information, principal component analysis, analysis of variance, and Fisher discriminant ratio) and combined with ten different types of classifiers (linear discriminant analysis, quadratic discriminant analysis, naïve Bayes, Gaussian process classification, support vector machine, artificial neural network, Adaboost, logistic regression, decision tree, and random forest) under the hypothesis that both missing values and outliers when replaced by computed medians will improve the risk stratification accuracy. Pima Indian diabetic dataset (768 patients: 268 diabetic and 500 controls) was used. Our results demonstrate that on replacing the missing values and outliers by group median and median values, respectively and further using the combination of random forest feature selection and random forest classification technique yields an accuracy, sensitivity, specificity, positive predictive value, negative predictive value and area under the curve as: 92.26%, 95.96%, 79.72%, 91.14%, 91.20%, and 0.93, respectively. This is an improvement of 10% over previously developed techniques published in literature. The system was validated for its stability and reliability. RF-based model showed the best performance when outliers are replaced by median values.
| S-EPMC5893681 | biostudies-other
Project description:"Classical peer review" has been subject to intense criticism for slowing down the publication process, bias against specific categories of paper and author, unreliability, inability to detect errors and fraud, unethical practices, and the lack of recognition for unpaid reviewers. This paper surveys innovative forms of peer review that attempt to address these issues. Based on an initial literature review, we construct a sample of 82 channels of scientific communication covering all forms of review identified by the survey, and analyze the review mechanisms used by each channel. We identify two major trends: the rapidly expanding role of preprint servers (e.g., ArXiv) that dispense with traditional peer review altogether, and the growth of "non-selective review," focusing on papers' scientific quality rather than their perceived importance and novelty. Other potentially important developments include forms of "open review," which remove reviewer anonymity, and interactive review, as well as new mechanisms for post-publication review and out-of-channel reader commentary, especially critical commentary targeting high profile papers. One of the strongest findings of the survey is the persistence of major differences between the peer review processes used by different disciplines. None of these differences is likely to disappear in the foreseeable future. The most likely scenario for the coming years is thus continued diversification, in which different review mechanisms serve different author, reader, and publisher needs. Relatively little is known about the impact of these innovations on the problems they address. These are important questions for future quantitative research.
| S-EPMC4444765 | biostudies-other
Project description:The advancement of various fields of science depends on the actions of individual scientists via the peer review process. The referees' work patterns and stochastic nature of decision making both relate to the particular features of refereeing and to the universal aspects of human behavior. Here, we show that the time a referee takes to write a report on a scientific manuscript depends on the final verdict. The data is compared to a model, where the review takes place in an ongoing competition of completing an important composite task with a large number of concurrent ones - a Deadline -effect. In peer review human decision making and task completion combine both long-range predictability and stochastic variation due to a large degree of ever-changing external "friction".
| S-EPMC3628219 | biostudies-other