Dataset Information

Individual versus general structured feedback to improve agreement in grant peer review: a randomized controlled trial.

ABSTRACT:

Background

Vast sums are distributed based on grant peer review, but studies show that interrater reliability is often low. In this study, we tested the effect of receiving two short individual feedback reports compared to one short general feedback report on the agreement between reviewers.

Methods

A total of 42 reviewers at the Norwegian Foundation Dam were randomly assigned to receive either a general feedback report or an individual feedback report. The general feedback group received one report before the start of the reviews that contained general information about the previous call in which the reviewers participated. In the individual feedback group, the reviewers received two reports, one before the review period (based on the previous call) and one during the period (based on the current call). In the individual feedback group, the reviewers were presented with detailed information on their scoring compared with the review committee as a whole, both before and during the review period. The main outcomes were the proportion of agreement in the eligibility assessment and the average difference in scores between pairs of reviewers assessing the same proposal. The outcomes were measured in 2017 and after the feedback was provided in 2018.

Results

A total of 2398 paired reviews were included in the analysis. There was a significant difference between the two groups in the proportion of absolute agreement on whether the proposal was eligible for the funding programme, with the general feedback group demonstrating a higher rate of agreement. There was no difference between the two groups in terms of the average score difference. However, the agreement regarding the proposal score remained critically low for both groups.

Conclusions

We did not observe changes in proposal score agreement between 2017 and 2018 in reviewers receiving different feedback. The low levels of agreement remain a major concern in grant peer review, and research to identify contributing factors as well as the development and testing of interventions to increase agreement rates are still needed.

Trial registration

The study was preregistered at OSF.io/n4fq3 .

SUBMITTER: Hesselberg JO

PROVIDER: S-EPMC8485516 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundMuch research worldwide is focussed on cost containment and better adherence to guidelines in healthcare. The research focussing on professional behaviour is often performed in a well-controlled research setting. In this study a large-scale implementation of a peer review strategy was tested on both test ordering and prescribing behaviour in primary care in the normal quality improvement setting.MethodsWe planned a cluster-RCT in existing local quality improvement collaboratives (LQICs) in primary care. The study ran from January 2008 to January 2011. LQICs were randomly assigned to one of two trial arms, with each arm receiving the same intervention of audit and feedback combined with peer review. Both arms were offered five different clinical topics and acted as blind controls for the other arm. The differences in test ordering rates and prescribing rates between both arms were analysed in an intention-to-treat pre-post analysis and a per-protocol analysis.ResultsTwenty-one LQIC groups, including 197 GPs working in 88 practices, entered the trial. The intention-to-treat analysis did not show a difference in the changes in test ordering or prescribing performance between intervention and control groups. The per-protocol analysis showed positive results for half of the clinical topics. The increase in total tests ordered was 3% in the intervention arm and 15% in the control arm. For prescribing the increase in prescriptions was 20% in the intervention arm and 66% in the control group. It was observed that the groups with the highest baseline test ordering and prescription volumes showed the largest improvements.ConclusionsOur study shows that the results from earlier work could not be confirmed by our attempt to implement the strategy in the field. We did not see a decrease in the volumes of tests ordered or of the drugs prescribed but were able to show a lesser increase instead. Implementing the peer review with audit and feedback proved to be not feasible in primary care in the Netherlands.Trial registrationThis trial was registered at the Dutch trial register under number ISRCTN40008171 on August 7th 2007.

Project description:BackgroundPatients receiving in-center hemodialysis experience disproportionate morbidity and incur high healthcare-related costs. Much of this cost stems from potentially avoidable hospitalizations. Peer mentorship has been used effectively to improve outcomes for patients with complex chronic diseases. We propose testing the efficacy of peer mentorship on hospitalization rates among patients receiving hemodialysis.MethodsThis is a multicenter parallel group randomized controlled pragmatic trial of patients treated at hemodialysis facilities in Bronx, NY and Nashville, TN. The study has two phases. Phase 1 will enroll and train 16 hemodialysis patients (10 in Bronx, NY and 6 in Nashville TN) to be mentors using a program focused on enhancing self-efficacy, dialysis self-management and autonomy-supportive communication skills. Phase 2 will enroll 200 high risk adults receiving hemodialysis (140 in Bronx, NY and 60 in Nashville, TN), half of whom will be randomized to intervention and half to usual care. Intervention participants are assigned to weekly telephone calls with trained mentors (see Phase 1) for a 3-month period. The primary outcome of Phase 1 will be engagement of mentors with training and change in knowledge scores and autonomy skills from pre- to post-training. The primary outcome of Phase 2 will be the composite count of ED visits and hospitalizations at the end of study follow-up in patient participants assigned to intervention as compared to those assigned to usual care. Secondary outcomes for Phase 2 include the change over the trial period in validated survey scores measuring perception of social support and self-efficacy, and dialysis adherence metrics, among intervention participants as compared to usual care participants.DiscussionThe PEER-HD study will test the feasibility and efficacy of a pragmatic peer-mentorship program designed for patients receiving hemodialysis on ED visit and hospitalization rates. If effective, peer-mentorship holds promise as a scalable patient-centered intervention to decrease hospital resource utilization, and by extension morbidity and cost, for patients receiving maintenance in-center hemodialysis.Trial registrationClinicaltrials.gov identifier: NCT03595748 ; 7/23/2018.Trial sponsorNational Institutes of Diabetes, Digestive and Kidney Disease (NIDDK) 5R18DK118471.FundingFunding for this study was provided by the National Institutes of Diabetes, Digestive and Kidney Disease: R18DK118471.Study statusThis is an ongoing study and not complete. We are still collecting data for observational follow-up on participants.Related articlesNo related articles for this study have been submitted to any journal. The study sponsor and funders had no role in the design, analysis or interpretation of this data. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Project description:ObjectiveSince many years, teaching and training in communication skills are cornerstones in the medical education curriculum. Although video recording in a real-time consultation is expected to positively contribute to the learning process, research on this topic is scarce. This study will focus on the feasibility and acceptability of video recording during real-time patient encounters performed by general practitioner (GP) trainees.MethodThe primary research question addressed the experiences (defined as feasibility and acceptability) of GP trainees in video-recorded vocational training in a general practice. The second research question addressed the appraisal of this training. The procedure of video-recorded training is developed, refined and validated by the Academic Teaching Practice of Leuven since 1974 (Faculty of Medicine of the University of Leuven). The study is set up as a cross-sectional survey without follow-up. Outcome measures were defined as 'feasibility and acceptability' (experiences of trainees) of the video-recorded training and were approached by a structured questionnaire with the opportunity to add free text comments. The studied sample consisted of all first-phase trainees of the GP Master 2011-2012 at the University of Leuven.ResultsAlmost 70% of the trainees were positive about recording consultations. Nevertheless, over 60% believed that patients felt uncomfortable during the video-recorded encounter. Almost 90% noticed an improvement of own communication skills through the observation and evaluation of. Most students (85%) experienced the logistical issues as major barrier to perform video consultations on a regular base.ConclusionsThis study lays the foundation stone for further exploration of the video training in real-time consultations. Both students and teachers on the field acknowledge that the power of imaging is underestimated in the training of communication and vocational skills. The development of supportive material and protocols will lower thresholds.Practice implicationsTime investment for teachers could be tempered by bringing up students to peer tutors and by an accurate scheduling of the video training. The development of supportive material and protocols will lower thresholds. Further research should finally focus on long-term efficacy and efficiency in terms of learning outcomes and on the facilitation of the technical process.