Dataset Information

Grant Review Feedback: Appropriateness and Usefulness.

ABSTRACT: The primary goal of the peer review of research grant proposals is to evaluate their quality for the funding agency. An important secondary goal is to provide constructive feedback to applicants for their resubmissions. However, little is known about whether review feedback achieves this goal. In this paper, we present a multi-methods analysis of responses from grant applicants regarding their perceptions of the effectiveness and appropriateness of peer review feedback they received from grant submissions. Overall, 56-60% of applicants determined the feedback to be appropriate (fair, well-written, and well-informed), although their judgments were more favorable if their recent application was funded. Importantly, independent of funding success, women found the feedback better written than men, and more white applicants found the feedback to be fair than non-white applicants. Also, perceptions of a variety of biases were specifically reported in respondents' feedback. Less than 40% of applicants found the feedback to be very useful in informing their research and improving grantsmanship and future submissions. Further, negative perceptions of the appropriateness of review feedback were positively correlated with more negative perceptions of feedback usefulness. Importantly, respondents suggested that highly competitive funding pay-lines and poor inter-panel reliability limited the usefulness of review feedback. Overall, these results suggest that more effort is needed to ensure that appropriate and useful feedback is provided to all applicants, bolstering the equity of the review process and likely improving the quality of resubmitted proposals.

SUBMITTER: Gallo SA

PROVIDER: S-EPMC7969534 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundVast sums are distributed based on grant peer review, but studies show that interrater reliability is often low. In this study, we tested the effect of receiving two short individual feedback reports compared to one short general feedback report on the agreement between reviewers.MethodsA total of 42 reviewers at the Norwegian Foundation Dam were randomly assigned to receive either a general feedback report or an individual feedback report. The general feedback group received one report before the start of the reviews that contained general information about the previous call in which the reviewers participated. In the individual feedback group, the reviewers received two reports, one before the review period (based on the previous call) and one during the period (based on the current call). In the individual feedback group, the reviewers were presented with detailed information on their scoring compared with the review committee as a whole, both before and during the review period. The main outcomes were the proportion of agreement in the eligibility assessment and the average difference in scores between pairs of reviewers assessing the same proposal. The outcomes were measured in 2017 and after the feedback was provided in 2018.ResultsA total of 2398 paired reviews were included in the analysis. There was a significant difference between the two groups in the proportion of absolute agreement on whether the proposal was eligible for the funding programme, with the general feedback group demonstrating a higher rate of agreement. There was no difference between the two groups in terms of the average score difference. However, the agreement regarding the proposal score remained critically low for both groups.ConclusionsWe did not observe changes in proposal score agreement between 2017 and 2018 in reviewers receiving different feedback. The low levels of agreement remain a major concern in grant peer review, and research to identify contributing factors as well as the development and testing of interventions to increase agreement rates are still needed.Trial registrationThe study was preregistered at OSF.io/n4fq3 .

Project description:BackgroundImprecise interpretation of coronary angiograms was reported and resulted in inappropriate revascularization. Synergy Between Percutaneous Coronary Intervention with Taxus and Cardiac Surgery (SYNTAX) score is a comprehensive system to evaluate the complexity of the overall lesions. We hypothesized that a real-time SYNTAX score feedback from image analysts may rectify the mis-estimation and improve revascularization appropriateness in patients with stable coronary artery disease (CAD).MethodsIn this single-center, historical control study, patients with stable CAD with coronary lesion stenosis ≥50% were consecutively recruited. During the control period, SYNTAX scores were calculated by treating cardiologists. During the intervention period, SYNTAX scores were calculated by image analysts immediately after coronary angiography and were provided to cardiologists in real-time to aid decision-making. The primary outcome was revascularization deemed inappropriate by Chinese appropriate use criteria for coronary revascularization.ResultsA total of 3245 patients were enrolled and assigned to the control group (08/2016-03/2017, n = 1525) or the intervention group (03/2017-09/2017, n = 1720). For SYNTAX score tertiles, 17.9% patients were overestimated and 4.3% were underestimated by cardiologists in the control group. After adjustment, inappropriate revascularization significantly decreased in the intervention group compared with the control group (adjusted odds ratio [OR]: 0.83; 95% confidence interval [CI]: 0.73-0.95; P = 0.007). Both inappropriate percutaneous coronary intervention (adjusted OR: 0.82; 95% CI: 0.74-0.92; P < 0.001) and percutaneous coronary intervention utilization (adjusted OR: 0.88; 95% CI: 0.79-0.98; P = 0.016) decreased significantly in the intervention group. There was no significant difference in 1-year adverse cardiac events between the control group and the intervention group.ConclusionsReal-time SYNTAX score feedback significantly reduced inappropriate coronary revascularization in stable patients with CAD.Clinical trial registrationNos. NCT03068858 and NCT02880605; https://www.clinicaltrials.gov.

Project description:BackgroundPeer review of grant applications has been criticized as lacking reliability. Studies showing poor agreement among reviewers supported this possibility but usually focused on reviewers' scores and failed to investigate reasons for disagreement. Here, our goal was to determine how reviewers rate applications, by investigating reviewer practices and grant assessment criteria.Methods and findingsWe first collected and analyzed a convenience sample of French and international calls for proposals and assessment guidelines, from which we created an overall typology of assessment criteria comprising nine domains relevance to the call for proposals, usefulness, originality, innovativeness, methodology, feasibility, funding, ethical aspects, and writing of the grant application. We then performed a qualitative study of reviewer practices, particularly regarding the use of assessment criteria, among reviewers of the French Academic Hospital Research Grant Agencies (Programmes Hospitaliers de Recherche Clinique, PHRCs). Semi-structured interviews and observation sessions were conducted. Both the time spent assessing each grant application and the assessment methods varied across reviewers. The assessment criteria recommended by the PHRCs were listed by all reviewers as frequently evaluated and useful. However, use of the PHRC criteria was subjective and varied across reviewers. Some reviewers gave the same weight to each assessment criterion, whereas others considered originality to be the most important criterion (12/34), followed by methodology (10/34) and feasibility (4/34). Conceivably, this variability might adversely affect the reliability of the review process, and studies evaluating this hypothesis would be of interest.ConclusionsVariability across reviewers may result in mistrust among grant applicants about the review process. Consequently, ensuring transparency is of the utmost importance. Consistency in the review process could also be improved by providing common definitions for each assessment criterion and uniform requirements for grant application submissions. Further research is needed to assess the feasibility and acceptability of these measures.

Dataset Information

Grant Review Feedback: Appropriateness and Usefulness.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets