Dataset Information

Face-to-face panel meetings versus remote evaluation of fellowship applications: simulation study at the Swiss National Science Foundation.

ABSTRACT:

Objectives

To trial a simplified, time and cost-saving method for remote evaluation of fellowship applications and compare this with existing panel review processes by analysing concordance between funding decisions, and the use of a lottery-based decision method for proposals of similar quality.

Design

The study involved 134 junior fellowship proposals for postdoctoral research ('Postdoc.Mobility'). The official method used two panel reviewers who independently scored the application, followed by triage and discussion of selected applications in a panel. Very competitive/uncompetitive proposals were directly funded/rejected without discussion. The simplified procedure used the scores of the two panel members, with or without the score of an additional, third expert. Both methods could further use a lottery to decide on applications of similar quality close to the funding threshold. The same funding rate was applied, and the agreement between the two methods analysed.

Setting

Swiss National Science Foundation (SNSF).

Participants

Postdoc.Mobility panel reviewers and additional expert reviewers.

Primary outcome measure

Per cent agreement between the simplified and official evaluation method with 95% CIs.

Results

The simplified procedure based on three reviews agreed in 80.6% (95% CI: 73.9% to 87.3%) of applicants with the official funding outcome. The agreement was 86.6% (95% CI: 80.6% to 91.8%) when using the two reviews of the panel members. The agreement between the two methods was lower for the group of applications discussed in the panel (64.2% and 73.1%, respectively), and higher for directly funded/rejected applications (range: 96.7%-100%). The lottery was used in 8 (6.0%) of 134 applications (official method), 19 (14.2%) applications (simplified, three reviewers) and 23 (17.2%) applications (simplified, two reviewers). With the simplified procedure, evaluation costs could have been halved and 31 hours of meeting time saved for the two 2019 calls.

Conclusion

Agreement between the two methods was high. The simplified procedure could represent a viable evaluation method for the Postdoc.Mobility early career instrument at the SNSF.

SUBMITTER: Bieri M

PROVIDER: S-EPMC8103360 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:PurposeTo understand the perceptions of program directors (PDs) and fellowship applicants regarding the virtual interview process for orthopaedic surgery sports medicine fellowship programs.MethodsAn anonymous online survey was distributed through the American Orthopaedic Society for Sports Medicine (AOSSM) to applicants and PDs of orthopaedic surgery sports medicine fellowship programs following the 2020-2021 fellowship application cycle.ResultsA total of 40 responses were received from PDs for a response rate of 47% (40 of 85) and 72 responses were received from applicants for a response rate of 27% (72 of 271). All of the surveyed PDs (40/40, 100%) agreed/strongly agreed that the applicant's interview carries significant weight in determining where an applicant is ranked on the match list. Fifty-eight percent (23 of 40) of PDs agreed/strongly agreed that virtual interviews negatively affected their personal connection with the fellowship interviewee. The presence of virtual interviews allowed 80% (57 of 71) of applicants to go on more interviews. Seventy-three percent (51 of 70) of applicants were able to save greater than $5,000 on travel expenses and 63% (25 of 40) of fellowship programs were able to save greater than $2,500 by conducting virtual interviews.ConclusionsVirtual interviews allowed fellowship programs and applicants to complete more interviews, but both PDs and applicants stated that interviewing in-person was important for applicants to meet faculty and tour the institution where they may be spending a year. In contrast, significant financial savings resulted due to the transition to virtual interviews. Finally, both PDs and applicants were in favor of having the option of interviewing virtually, suggesting that virtual interviews may continue to play a role in future application cycles.Clinical relevanceThis study may be valuable to fellowship programs that will continue to implement virtual interviews into future application cycles.