Dataset Information

Statistical analysis of the National Institutes of Health peer review system.

ABSTRACT: A statistical model is proposed for the analysis of peer-review ratings of R01 grant applications submitted to the National Institutes of Health. Innovations of this model include parameters that reflect differences in reviewer scoring patterns, a mechanism to account for the transfer of information from an application's preliminary ratings and group discussion to final ratings provided by all panel members and posterior estimates of the uncertainty associated with proposal ratings. Application of this model to recent R01 rating data suggests that statistical adjustments to panel rating data would lead to a 25% change in the pool of funded proposals. Viewed more broadly, the methodology proposed in this article provides a general framework for the analysis of data collected interactively from expert panels through the use of the Delphi method and related procedures.

SUBMITTER: Johnson VE

PROVIDER: S-EPMC2488382 | biostudies-literature | 2008 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Statistical analysis of the National Institutes of Health peer review system.

Johnson Valen E VE

Proceedings of the National Academy of Sciences of the United States of America 20080728 32

A statistical model is proposed for the analysis of peer-review ratings of R01 grant applications submitted to the National Institutes of Health. Innovations of this model include parameters that reflect differences in reviewer scoring patterns, a mechanism to account for the transfer of information from an application's preliminary ratings and group discussion to final ratings provided by all panel members and posterior estimates of the uncertainty associated with proposal ratings. Application ...[more]

PMID: 18663221

Similar Datasets

Project description:ImportanceDespite the rapid growth of interest and diversity in applications of artificial intelligence (AI) to biomedical research, there are limited objective ways to characterize the potential for use of AI in clinical practice.ObjectiveTo examine what types of medical AI have the greatest estimated translational impact (ie, ability to lead to development that has measurable value for human health) potential.Design, setting, and participantsIn this cohort study, research grants related to AI awarded between January 1, 1985, and December 31, 2020, were identified from a National Institutes of Health (NIH) award database. The text content for each award was entered into a Natural Language Processing (NLP) clustering algorithm. An NIH database was also used to extract citation data, including the number of citations and approximate potential to translate (APT) score for published articles associated with the granted awards to create proxies for translatability.ExposuresUnsupervised assignment of AI-related research awards to application topics using NLP.Main outcomes and measuresAnnualized citations per $1 million funding (ACOF) and average APT score for award-associated articles, grouped by application topic. The APT score is a machine-learning based metric created by the NIH Office of Portfolio Analysis that quantifies the likelihood of future citation by a clinical article.ResultsA total of 16 629 NIH awards related to AI were included in the analysis, and 75 applications of AI were identified. Total annual funding for AI grew from $17.4 million in 1985 to $1.43 billion in 2020. By average APT, interpersonal communication technologies (0.488; 95% CI, 0.472-0.504) and population genetics (0.463; 95% CI, 0.453-0.472) had the highest translatability; environmental health (ACOF, 1038) and applications focused on the electronic health record (ACOF, 489) also had high translatability. The category of applications related to biochemical analysis was found to have low translatability by both metrics (average APT, 0.393; 95% CI, 0.388-0.398; ACOF, 246).Conclusions and relevanceBased on this study's findings, data on grants from the NIH can apparently be used to identify and characterize medical applications of AI to understand changes in academic productivity, funding support, and potential for translational impact. This method may be extended to characterize other research domains.

Dataset Information

Statistical analysis of the National Institutes of Health peer review system.

Publications

Statistical analysis of the National Institutes of Health peer review system.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets