Dataset Information

Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study.

ABSTRACT: Background: Decisions about which applications to fund are generally based on the mean scores of a panel of peer reviewers. As well as the mean, a large disagreement between peer reviewers may also be worth considering, as it may indicate a high-risk application with a high return. Methods: We examined the peer reviewers' scores for 227 funded applications submitted to the American Institute of Biological Sciences between 1999 and 2006. We examined the mean score and two measures of reviewer disagreement: the standard deviation and range. The outcome variable was the relative citation ratio, which is the number of citations from all publications associated with the application, standardised by field and publication year. Results: There was a clear increase in relative citations for applications with a better mean. There was no association between relative citations and either of the two measures of disagreement. Conclusions: We found no evidence that reviewer disagreement was able to identify applications with a higher than average return. However, this is the first study to empirically examine this association, and it would be useful to examine whether reviewer disagreement is associated with research impact in other funding schemes and in larger sample sizes.

SUBMITTER: Barnett AG

PROVIDER: S-EPMC6171721 | biostudies-literature | 2018

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study.

Barnett Adrian G AG Glisson Scott R SR Gallo Stephen S

F1000Research 20180709

<b>Background</b>: Decisions about which applications to fund are generally based on the mean scores of a panel of peer reviewers. As well as the mean, a large disagreement between peer reviewers may also be worth considering, as it may indicate a high-risk application with a high return. <b>Methods</b>: We examined the peer reviewers' scores for 227 funded applications submitted to the American Institute of Biological Sciences between 1999 and 2006. We examined the mean score and two measures o ...[more]

PMID: 30345025

Similar Datasets

Project description:Systematic reviews evaluating the impact of interventions to improve the quality of peer review for biomedical publications highlighted that interventions were limited and have little impact. This study aims to compare the accuracy of early career peer reviewers who use an innovative online tool to the usual peer reviewer process in evaluating the completeness of reporting and switched primary outcomes in completed reports.This is a cross-sectional study of individual two-arm parallel-group randomised controlled trials (RCTs) published in the BioMed Central series medical journals, BMJ, BMJ Open and Annals of Emergency Medicine and indexed with the publication type 'Randomised Controlled Trial'. First, we will develop an online tool and training module based (a) on the Consolidated Standards of Reporting Trials (CONSORT) 2010 checklist and the Explanation and Elaboration document that would be dedicated to junior peer reviewers for assessing the completeness of reporting of key items and (b) the Centre for Evidence-Based Medicine Outcome Monitoring Project process used to identify switched outcomes in completed reports of the primary results of RCTs when initially submitted. Then, we will compare the performance of early career peer reviewers who use the online tool to the usual peer review process in identifying inadequate reporting and switched outcomes in completed reports of RCTs at initial journal submission. The primary outcome will be the mean number of items accurately classified per manuscript. The secondary outcomes will be the mean number of items accurately classified per manuscript for the CONSORT items and the sensitivity, specificity and likelihood ratio to detect the item as adequately reported and to identify a switch in outcomes. We aim to include 120 RCTs and 120 early career peer reviewers.The research protocol was approved by the ethics committee of the INSERM Institutional Review Board (21 January 2016). The study is based on voluntary participation and informed written consent.NCT03119376.

Project description:BackgroundThe peer review process has been questioned as it may fail to allow the publication of high-quality articles. This study aimed to evaluate the accuracy in identifying inadequate reporting in RCT reports by early career researchers (ECRs) using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process.MethodsWe performed a cross-sectional diagnostic study of 119 manuscripts, from BMC series medical journals, BMJ, BMJ Open, and Annals of Emergency Medicine reporting the results of two-arm parallel-group RCTs. One hundred and nineteen ECRs who had never reviewed an RCT manuscript were recruited from December 2017 to January 2018. Each ECR assessed one manuscript. To assess accuracy in identifying inadequate reporting, we used two tests: (1) ECRs assessing a manuscript using the COBPeer tool (after completing an online training module) and (2) the usual peer-review process. The reference standard was the assessment of the manuscript by two systematic reviewers. Inadequate reporting was defined as incomplete reporting or a switch in primary outcome and considered nine domains: the eight most important CONSORT domains and a switch in primary outcome(s). The primary outcome was the mean number of domains accurately classified (scale from 0 to 9).ResultsThe mean (SD) number of domains (0 to 9) accurately classified per manuscript was 6.39 (1.49) for ECRs using COBPeer versus 5.03 (1.84) for the journal's usual peer-review process, with a mean difference [95% CI] of 1.36 [0.88-1.84] (p < 0.001). Concerning secondary outcomes, the sensitivity of ECRs using COBPeer versus the usual peer-review process in detecting incompletely reported CONSORT items was 86% [95% CI 82-89] versus 20% [16-24] and in identifying a switch in primary outcome 61% [44-77] versus 11% [3-26]. The specificity of ECRs using COBPeer versus the usual process to detect incompletely reported CONSORT domains was 61% [57-65] versus 77% [74-81] and to identify a switch in primary outcome 77% [67-86] versus 98% [92-100].ConclusionsTrained ECRs using the COBPeer tool were more likely to detect inadequate reporting in RCTs than the usual peer review processes used by journals. Implementing a two-step peer-review process could help improve the quality of reporting.Trial registrationClinical.Trials.gov NCT03119376 (Registered April, 18, 2017).

Project description:OBJECTIVES:Innovations resulting from research have both national and global impact, so selecting the most promising research studies to fund is crucial. Peer review of research funding applications is part of the selection process, and requires considerable resources. This study aimed to elicit stakeholder opinions about which factors contribute to and influence effective peer review of funding applications to the UK National Institute for Health Research (NIHR), and to identify possible minor improvements to current processes and any major changes or potential innovations to achieve a more efficient peer review process. DESIGN:Qualitative interviews with 30 stakeholders involved in the peer review process. PARTICIPANTS:Participants were drawn from three NIHR coordinating centres and represented four types of stakeholders: board members with responsibility for making funding decisions, applicants, external peer reviewers and NIHR staff. METHODS:All interviews were conducted by telephone apart from three that were face to face with NIHR staff. Data were analysed using a thematic template method. RESULTS:The responses from NIHR staff, board members and reviewers differed from those received from applicants. The first three groups focused on how well the process of peer review did or did not function. The applicants mentioned these points but in addition often reflected on how their personal application was assessed. Process improvements suggested included: developing a more proportionate review process; providing greater guidance, feedback, training, acknowledgement or incentives for peer reviewers; reducing the time commitment and amount of paperwork; and asking reviewers to comment on the importance, strengths and weaknesses of applications and flaws which are potentially 'fixable'. CONCLUSIONS:Overall, participants were supportive of the need for peer review in evaluating applications for research funding. This study revealed which parts of the process are working well and are valued, and barriers, difficulties and potential areas for improvement and development.

Project description:Synthesizing outcomes of underreported primary studies can pose a serious threat to the validity of outcomes and conclusions of systematic reviews. To address this problem, the Cochrane Collaboration recommends reviewers to contact authors of eligible primary studies to obtain additional information on poorly reported items. In this protocol, we present a cross-sectional study and a survey to assess (1) how reviewers of new Cochrane intervention reviews report on procedures and outcomes of contacting of authors of primary studies to obtain additional data, (2) how authors reply, and (3) the consequences of these additional data on the outcomes and quality scores in the review. All research questions and methods were pilot tested on 2 months of Cochrane reviews and were subsequently fine-tuned.Eligibility criteria are (1) all new (not-updates) Cochrane intervention reviews published in 2016, (2) reviews that included one or more primary studies, and (3) eligible interventions refer to contacting of authors of the eligible primary studies included in the review to obtain additional research data (e.g., information on unreported or missing data, individual patient data, research methods, and bias issues). Searching for eligible reviews and data extraction will be conducted by two authors independently. The cross-sectional study will primarily focus on how contacting of authors is conducted and reported, how contacted authors reply, and how reviewers report on obtained additional data and their consequences for the review.The same eligible reviews for the cross-sectional study will also be eligible for the survey. Surveys will be sent to the contact addresses of these reviews according to a pre-defined protocol. We will use Google Forms as our survey platform. Surveyees are asked to answer eight questions. The survey will primarily focus on the consequences of contacting authors of eligible primary studies for the risk of bias and Grading of Recommendations, Assessment, Development and Evaluation scores and the primary and secondary outcomes of the review.The findings of this study could help improve methods of contacting authors and reporting of these procedures and their outcomes. Patients, clinicians, researchers, guideline developers, research sponsors, and the general public will all be beneficiaries.

Dataset Information

Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study.

Publications

Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets