Ranking versus rating in peer review of research grant applications.
Ontology highlight
ABSTRACT:
SUBMITTER: Tamblyn R
PROVIDER: S-EPMC10553257 | biostudies-literature | 2023
REPOSITORIES: biostudies-literature
Publications
Ranking versus rating in peer review of research grant applications.
Tamblyn Robyn R Girard Nadyne N Hanley James J Habib Bettina B Mota Adrian A Khan Karim M KM Ardern Clare L CL
PloS one 20231005 10
The allocation of public funds for research has been predominantly based on peer review where reviewers are asked to rate an application on some form of ordinal scale from poor to excellent. Poor reliability and bias of peer review rating has led funding agencies to experiment with different approaches to assess applications. In this study, we compared the reliability and potential sources of bias associated with application rating with those of application ranking in 3,156 applications to the C ...[more]
Similar Datasets
Project description:BackgroundPeer review of grant applications has been criticized as lacking reliability. Studies showing poor agreement among reviewers supported this possibility but usually focused on reviewers' scores and failed to investigate reasons for disagreement. Here, our goal was to determine how reviewers rate applications, by investigating reviewer practices and grant assessment criteria.Methods and findingsWe first collected and analyzed a convenience sample of French and international calls for proposals and assessment guidelines, from which we created an overall typology of assessment criteria comprising nine domains relevance to the call for proposals, usefulness, originality, innovativeness, methodology, feasibility, funding, ethical aspects, and writing of the grant application. We then performed a qualitative study of reviewer practices, particularly regarding the use of assessment criteria, among reviewers of the French Academic Hospital Research Grant Agencies (Programmes Hospitaliers de Recherche Clinique, PHRCs). Semi-structured interviews and observation sessions were conducted. Both the time spent assessing each grant application and the assessment methods varied across reviewers. The assessment criteria recommended by the PHRCs were listed by all reviewers as frequently evaluated and useful. However, use of the PHRC criteria was subjective and varied across reviewers. Some reviewers gave the same weight to each assessment criterion, whereas others considered originality to be the most important criterion (12/34), followed by methodology (10/34) and feasibility (4/34). Conceivably, this variability might adversely affect the reliability of the review process, and studies evaluating this hypothesis would be of interest.ConclusionsVariability across reviewers may result in mistrust among grant applicants about the review process. Consequently, ensuring transparency is of the utmost importance. Consistency in the review process could also be improved by providing common definitions for each assessment criterion and uniform requirements for grant application submissions. Further research is needed to assess the feasibility and acceptability of these measures.
| S-EPMC3460995 | biostudies-literature
Project description:Grant proposals submitted for funding are usually selected by a peer-review rating process. Some proposals may result in discordant peer-review ratings and therefore require discussion by the selection committee members. The issue is which peer-review ratings are considered as discordant. We propose a simple method to identify such proposals. Our approach is based on the intraclass correlation coefficient, which is usually used in assessing agreement in studies with continuous ratings.
| S-EPMC3215721 | biostudies-literature
Project description:BackgroundResearch funders use a wide variety of application assessment processes yet there is little evidence on their relative advantages and disadvantages. A broad distinction can be made between processes with a single stage assessment of full proposals and those that first invite an outline, with full proposals invited at a second stage only for those which are shortlisted. This paper examines the effects of changing from a one-stage to a two-stage process within the UK's National Institute for Health Research's (NIHR) Research for Patient Benefit (RfPB) Programme which made this change in 2015.MethodsA retrospective comparative design was used to compare eight one-stage funding competitions (912 applications) with eight two-stage funding competitions (1090 applications). Comparisons were made between the number of applications submitted, number of peer and lay reviews required, the duration of the funding round, average external peer review scores, and the total costs involved.ResultsThere was a mean number of 114 applications per funding round for the one-stage process and 136 for the two-stage process. The one-stage process took a mean of 274 days and the two-stage process 348 days to complete, although those who were not funded (i.e. the majority) were informed at a mean of 195 days (mean 79 days earlier) under the two-stage process. The mean peer review score for full applications using the one-stage process was 6.46 and for the two-stage process 6.82 (5.6% difference using a 1-10 scale (with 10 being the highest), but there was no significant difference between the lay reviewer scores. The one-stage process required a mean of 423 peer reviews and 102 lay reviewers and the two-stage process required a mean of 208 peer reviews and 50 lay reviews (mean difference of 215 peer reviews and 52 lay reviews) per funding round. Overall cost per funding round changed from £148,908 for the one-stage process to £105,342 for the two-stage process saving approximately £43,566 per round.ConclusionWe conclude that a two-stage application process increases the number of applications submitted to a funding round, is less burdensome and more efficient for all those involved with the process, is cost effective and has a small increase in peer reviewer scores. For the addition of fewer than 11 weeks to the process substantial efficiencies are gained which benefit funders, applicants and science. Funding agencies should consider adopting a two-stage application assessment process.
| S-EPMC7067561 | biostudies-literature
Project description:BackgroundDifferential participation and success in grant applications may contribute to women's lesser representation in the sciences. This study's objective was to conduct a systematic review and meta-analysis to address the question of gender differences in grant award acceptance rates and reapplication award acceptance rates (potential bias in peer review outcomes) and other grant outcomes.MethodsThe review was registered on PROSPERO (CRD42021232153) and conducted in accordance with PRISMA 2020 standards. We searched Academic Search Complete, PubMed, and Web of Science for the timeframe 1 January 2005 to 31 December 2020, and forward and backward citations. Studies were included that reported data, by gender, on any of the following: grant applications or reapplications, awards, award amounts, award acceptance rates, or reapplication award acceptance rates. Studies that duplicated data reported in another study were excluded. Gender differences were investigated by meta-analyses and generalized linear mixed models. Doi plots and LFK indices were used to assess reporting bias.ResultsThe searches identified 199 records, of which 13 were eligible. An additional 42 sources from forward and backward searches were eligible, for a total of 55 sources with data on one or more outcomes. The data from these studies ranged from 1975 to 2020: 49 sources were published papers and six were funders' reports (the latter were identified by forwards and backwards searches). Twenty-nine studies reported person-level data, 25 reported application-level data, and one study reported both: person-level data were used in analyses. Award acceptance rates were 1% higher for men, which was not significantly different from women (95% CI 3% more for men to 1% more for women, k = 36, n = 303,795 awards and 1,277,442 applications, I2 = 84%). Reapplication award acceptance rates were significantly higher for men (9%, 95% CI 18% to 1%, k = 7, n = 7319 applications and 3324 awards, I2 = 63%). Women received smaller award amounts (g = -2.28, 95% CI -4.92 to 0.36, k = 13, n = 212,935, I2 = 100%).ConclusionsThe proportions of women that applied for grants, re-applied, accepted awards, and accepted awards after reapplication were less than the proportion of eligible women. However, the award acceptance rate was similar for women and men, implying no gender bias in this peer reviewed grant outcome. Women received smaller awards and fewer awards after re-applying, which may negatively affect continued scientific productivity. Greater transparency is needed to monitor and verify these data globally.
| S-EPMC10155348 | biostudies-literature
Project description:Purpose: This study examined the gender composition of career development award applicants and grant review panels during the pandemic compared with that beforehand. Methods: Data were collected from 14 Health Research Alliance (HRA) organizations, which fund biomedical research and training. HRA members provided the gender of grant applicants and grant reviewers during the pandemic (April 1, 2020, to February 28, 2021) and prepandemic (April 1, 2019, to February 29, 2020). The signed-rank test compared medians and the chi square test compared the overall gender distribution. Results: The total number of applicants was similar during the pandemic (N = 3,724) and prepandemic (N = 3,882) periods, as was the percentage of women applicants (45.2% pandemic vs. 44.9% prepandemic, p = 0.78). The total number of men and women grant reviewers declined during the pandemic (N = 856) compared with that pre-pandemic (N = 1,689); this decrease was driven by a change for the largest funder. Also driven by changes for this one funder, the percentage of total grant reviewers who were women increased significantly during the pandemic (45.9%) compared with that during prepandemic (38.8%; p = 0.001), but the median percentage of women grant reviewers across organizations remained similar during the pandemic (43.6%) and prepandemic periods (38.2%; p = 0.53). Conclusions: In a sample of research organizations, the gender composition of grant applicants and grant review panels remained similar, except for the review panel composition for one large funder. Given evidence from other studies that have revealed gender differences in other career and life experiences of scientists during the pandemic, ongoing evaluation of women's representation in grant submission and review mechanisms is essential.
| S-EPMC10329151 | biostudies-literature
Project description:Background and objectivesPeer review is ubiquitous in evaluating scientific research. While peer review of manuscripts submitted to journals has been widely studied, there has been relatively less attention paid to peer review of grant applications (despite how crucial peer review is to researchers having the means and capacity to conduct research). There is spirited debate in academic community forums (including on social media) about the perceived benefits and limitations of grant peer review. The aim of our study was to understand the experiences and challenges faced by grant peer reviewers.MethodsTherefore, we conducted qualitative interviews with 18 members of grant review panels-the Chairs, peer reviewers and Scientific Officers of a national funding agency-that highlight threats to the integrity of grant peer review.ResultsWe identified three threats: (1) lack of training and limited opportunities to learn, (2) challenges in differentiating and rating applications of similar strength, and (3) reviewers weighting reputations and relationships in the review process to differentiate grant applications of a similar strength. These threats were compounded by reviewers' stretched resources or lack of time. Our data also highlighted the essential role of the Chair in ensuring transparency and rigorous grant peer review.ConclusionsAs researchers continue to evaluate the threats to grant peer review, the reality of stretched resources and time must be considered. We call on funders and academic institutions to implement practices that reduce reviewer burden.
| S-EPMC11842976 | biostudies-literature
Project description:BackgroundVast sums are distributed based on grant peer review, but studies show that interrater reliability is often low. In this study, we tested the effect of receiving two short individual feedback reports compared to one short general feedback report on the agreement between reviewers.MethodsA total of 42 reviewers at the Norwegian Foundation Dam were randomly assigned to receive either a general feedback report or an individual feedback report. The general feedback group received one report before the start of the reviews that contained general information about the previous call in which the reviewers participated. In the individual feedback group, the reviewers received two reports, one before the review period (based on the previous call) and one during the period (based on the current call). In the individual feedback group, the reviewers were presented with detailed information on their scoring compared with the review committee as a whole, both before and during the review period. The main outcomes were the proportion of agreement in the eligibility assessment and the average difference in scores between pairs of reviewers assessing the same proposal. The outcomes were measured in 2017 and after the feedback was provided in 2018.ResultsA total of 2398 paired reviews were included in the analysis. There was a significant difference between the two groups in the proportion of absolute agreement on whether the proposal was eligible for the funding programme, with the general feedback group demonstrating a higher rate of agreement. There was no difference between the two groups in terms of the average score difference. However, the agreement regarding the proposal score remained critically low for both groups.ConclusionsWe did not observe changes in proposal score agreement between 2017 and 2018 in reviewers receiving different feedback. The low levels of agreement remain a major concern in grant peer review, and research to identify contributing factors as well as the development and testing of interventions to increase agreement rates are still needed.Trial registrationThe study was preregistered at OSF.io/n4fq3 .
| S-EPMC8485516 | biostudies-literature
Project description:BackgroundThe Delphi technique has steeply grown in popularity in health research as a structured approach to group communication process. Rating and ranking are two different procedures commonly used to quantify participants' opinions in Delphi surveys. We explored the influence of using a rating or ranking approach on item prioritization (main outcome), questionnaire completion time, and evaluation of task difficulty in a Delphi survey aimed at identifying priorities for the organization of primary cardiovascular care.MethodsA randomized controlled parallel group trial was embedded in a three-round online Delphi survey. After an "open" first round, primary care patients, trained patient partners, and primary care clinicians from seven primary care practices were allocated 1:1 to a rating or ranking assessment group for the remainder of the study by stratified permuted block randomization, with strata based on participants' gender and status. Agreement on item prioritization between the experimental groups was measured by calculating Krippendorff's alpha reliability coefficient on the aggregate rank order of items in each group after the final round. Self-reported ease or difficulty with the assessment task was measured with the Single Ease Question.ResultsThirty-six panelists (13 clinic patients, 7 patient partners, 16 clinicians; 60% females) were randomized to the rating (n = 18) or ranking (n = 18) group, with 30 (83%) completing all rounds. Both groups identified the same highest priorities from a set of 41 items, but significant discrepancies were found as early as the seventh top item. There was moderately strong agreement between the priority ordering of top items common to both groups (Krippendorff's alpha = 0.811, 95% CI = 0.669-0.920). A 9-min mean difference to complete the third-round questionnaire in favor of the rating group failed to achieve statistical significance (p = 0.053). Ranking was perceived as more difficult (p < 0.001).ConclusionsA rating or ranking procedure led to modestly similar item prioritization in a Delphi survey, but ranking was more difficult. This study should be replicated with a larger number of participants and with variations in the ranking and rating procedures.Trial registrationNot applicable.
| S-EPMC10436639 | biostudies-literature
Project description:Adverse events are poor patient outcomes caused by medical care. Their identification requires the peer-review of poor outcomes, which may be unreliable. Combining physician ratings might improve the accuracy of adverse event classification.To evaluate the variation in peer-reviewer ratings of adverse outcomes; determine the impact of this variation on estimates of reviewer accuracy; and determine the number of reviewers who judge an adverse event occurred that is required to ensure that the true probability of an adverse event exceeded 50%, 75% or 95%.Thirty physicians rated 319 case reports giving details of poor patient outcomes following hospital discharge. They rated whether medical management caused the outcome using a six-point ordinal scale. We conducted latent class analyses to estimate the prevalence of adverse events as well as the sensitivity and specificity of each reviewer. We used this model and Bayesian calculations to determine the probability that an adverse event truly occurred to each patient as function of their number of positive ratings.The overall median score on the 6-point ordinal scale was 3 (IQR 2,4) but the individual rater median score ranged from a minimum of 1 (in four reviewers) to a maximum median score of 5. The overall percentage of cases rated as an adverse event was 39.7% (3798/9570). The median kappa for all pair-wise combinations of the 30 reviewers was 0.26 (IQR 0.16, 0.42; Min?=?-0.07, Max?=?0.62). Reviewer sensitivity and specificity for adverse event classification ranged from 0.06 to 0.93 and 0.50 to 0.98, respectively. The estimated prevalence of adverse events using a latent class model with a common sensitivity and specificity for all reviewers (0.64 and 0.83 respectively) was 47.6%. For patients to have a 95% chance of truly having an adverse event, at least 3 of 3 reviewers are required to deem the outcome an adverse event.Adverse event classification is unreliable. To be certain that a case truly represents an adverse event, there needs to be agreement among multiple reviewers.
| S-EPMC3406022 | biostudies-literature
Project description:PurposeTo compare two techniques for eliciting and clarifying patient values for decision making about colorectal cancer (CRC) screening: choice-based conjoint analysis and a rating and ranking task.MethodsUsing our decision lab registry and university e-mail lists, we recruited average risk adults ages 48-75 for a written, mailed survey. Eligible participants were given basic information about CRC screening and six attributes of CRC screening tests, then randomized to complete either a choice-based conjoint analysis with 16 discrete choice tasks or a rating and ranking task. The main outcome was the most important attribute, as determined from conjoint analysis or participant ranking. Conjoint analysis-based most important attribute was determined from individual patient-level utilities generated using multinomial logistic regression and hierarchical Bayesian modeling.ResultsOf the 114 eligible participants, 104 completed and returned questionnaires. Mean age was 57 (range 48-73), 70% were female, 88% were white, 71% were college graduates, and 62% were up to date with CRC screening. Ability to reduce CRC incidence and mortality was the most frequent most important attribute for both the conjoint analysis (56% of respondents) and rating/ranking (76% of respondents) groups, and these proportions differed significantly between groups (absolute difference 20%, 95% CI 3%, 37%, p =0.03). There were no significant differences between groups in proportion with clear values (p = 0.352), intent to be screened (p = 0.226) or unlabelled test preference (p = 0.521)ConclusionsChoice-based conjoint analysis produced somewhat different patterns of attribute importance than a rating and ranking task, but had little effect on other outcomes.
| S-EPMC3250548 | biostudies-literature