Dataset Information

Developing and evaluating criteria to help reviewers of biomedical informatics manuscripts.

ABSTRACT: Peer-reviewed publication of scientific research results represents the most important means of their communication. The authors have annually reviewed a large heterogeneous set of papers to produce the International Medical Informatics Association (IMIA) Yearbook of Medical Informatics. To support an objective and high-quality review process, the authors attempted to provide reviewers with a set of refined quality criteria, comprised of 80 general criteria and an additional 60 criteria for specific types of manuscripts. Authors conducted a randomized controlled trial, with 18 reviewers, to evaluate application of the refined criteria on review outcomes. Whereas the trial found that reviewers applying the criteria graded papers more strictly (lower overall scores), and that junior reviewers appreciated the availability of the criteria, there was no overall change in the interrater variability in reviewing the manuscripts. The authors describe their experience as a "case report" and provide a reference to the refined quality review criteria without claiming that the criteria represent a validated instrument for quantitative quality measurement.

SUBMITTER: Ammenwerth E

PROVIDER: S-EPMC212789 | biostudies-literature | 2003 Sep-Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Developing and evaluating criteria to help reviewers of biomedical informatics manuscripts.

Ammenwerth Elske E Wolff Astrid C AC Knaup Petra P Ulmer Hanno H Skonetzki Stefan S van Bemmel Jan H JH McCray Alexa T AT Haux Reinhold R Kulikowski Casimir C

Journal of the American Medical Informatics Association : JAMIA 20030604 5

Peer-reviewed publication of scientific research results represents the most important means of their communication. The authors have annually reviewed a large heterogeneous set of papers to produce the International Medical Informatics Association (IMIA) Yearbook of Medical Informatics. To support an objective and high-quality review process, the authors attempted to provide reviewers with a set of refined quality criteria, comprised of 80 general criteria and an additional 60 criteria for spec ...[more]

PMID: 12807814

Similar Datasets

Project description:ObjectivesTo describe gender and geographical inequalities in invitations to review and the response to these invitations and to assess whether inequalities increased during the covid-19 pandemic.DesignRetrospective cohort study.Setting19 specialist medical journals and two large general medical journals from BMJ Publishing Group.PopulationReviewers invited to review manuscripts submitted between 1 January 2018 and 31 May 2021. The cohort was followed up to 28 February 2022.Main outcome measuresReviewer's agreement to review.ResultsA total of 257 025 reviewers were invited (38.6% (88 454/228 869) women), and 90 467 (35.2%) agreed to review. Invited reviewers were mainly (217 682; 84.7%) affiliated with high income countries: Europe (122 414; 47.6%), North America (66 931; 26.0%), Africa (25 735; 10.0%), Asia (22 693; 8.8%), Oceania (16 175; 6.3%), and South America (3076; 1.2%). Independent factors associated with agreement to review were gender (odds ratio 0.89, 95% confidence interval 0.87 to 0.92, for women compared with men), geographical affiliation (2.89, 2.73 to 3.06, for Asia; 3.32, 2.94 to 3.75 for South America; 1.35, 1.27 to 1.43, for Oceania; and 0.35, 0.33 to 0.37, for Africa compared with Europe), and country income (0.47, 0.45 to 0.49, for upper middle income; 5.12, 4.67 to 5.61, for lower middle income; and 4.66, 3.79 to 5.73, for low income compared with high income country). Agreement was also independently associated with editor's gender (0.96, 0.93 to 0.99, for women compared with men), last author's geographical affiliation (0.80, 0.78 to 0.83, for Asia; 0.89, 0.85 to 0.94, for Oceania compared with Europe), impact factor (1.78, 1.27 to 2.50, for >10 compared with <5), and type of peer review process (0.52, 0.35 to 0.77, for open compared with anonymised). During the first and second phases of the pandemic, agreement was lower than in the pre-pandemic period (P<0.001). The interaction between time periods and covid-19 related topic and reviewer's gender was non-significant. However, significant interaction was found between time periods and covid-19 related topic and reviewer's geographical affiliation.ConclusionsTo reduce bias and improve diversity, editors need to identify and implement effective strategies and continually evaluate progress against these to ensure that more women and researchers from upper middle income and low income countries are involved in review.

Project description:BACKGROUND:Peer review is at the heart of the scientific process. With the advent of digitisation, journals started to offer electronic articles or publishing online only. A new philosophy regarding the peer review process found its way into academia: the open peer review. Open peer review as practiced by BioMed Central (BMC) is a type of peer review where the names of authors and reviewers are disclosed and reviewer comments are published alongside the article. A number of articles have been published to assess peer reviews using quantitative research. However, no studies exist that used qualitative methods to analyse the content of reviewers' comments. METHODS:A focused mapping review and synthesis (FMRS) was undertaken of manuscripts reporting qualitative research submitted to BMC open access journals from 1 January - 31 March 2018. Free-text reviewer comments were extracted from peer review reports using a 77-item classification system organised according to three key dimensions that represented common themes and sub-themes. A two stage analysis process was employed. First, frequency counts were undertaken that allowed revealing patterns across themes/sub-themes. Second, thematic analysis was conducted on selected themes of the narrative portion of reviewer reports. RESULTS:A total of 107 manuscripts submitted to nine open-access journals were included in the FMRS. The frequency analysis revealed that among the 30 most frequently employed themes "writing criteria" (dimension II) is the top ranking theme, followed by comments in relation to the "methods" (dimension I). Besides that, some results suggest an underlying quantitative mindset of reviewers. Results are compared and contrasted in relation to established reporting guidelines for qualitative research to inform reviewers and authors of frequent feedback offered to enhance the quality of manuscripts. CONCLUSIONS:This FMRS has highlighted some important issues that hold lessons for authors, reviewers and editors. We suggest modifying the current reporting guidelines by including a further item called "Degree of data transformation" to prompt authors and reviewers to make a judgment about the appropriateness of the degree of data transformation in relation to the chosen analysis method. Besides, we suggest that completion of a reporting checklist on submission becomes a requirement.

Project description:BackgroundAlthough peer review is widely considered to be the most credible way of selecting manuscripts and improving the quality of accepted papers in scientific journals, there is little evidence to support its use. Our aim was to estimate the effects on manuscript quality of either adding a statistical peer reviewer or suggesting the use of checklists such as CONSORT or STARD to clinical reviewers or both.Methodology and principal findingsInterventions were defined as 1) the addition of a statistical reviewer to the clinical peer review process, and 2) suggesting reporting guidelines to reviewers; with "no statistical expert" and "no checklist" as controls. The two interventions were crossed in a 2x2 balanced factorial design including original research articles consecutively selected, between May 2004 and March 2005, by the Medicina Clinica (Barc) editorial committee. We randomized manuscripts to minimize differences in terms of baseline quality and type of study (intervention, longitudinal, cross-sectional, others). Sample-size calculations indicated that 100 papers provide an 80% power to test a 55% standardized difference. We specified the main outcome as the increment in quality of papers as measured on the Goodman Scale. Two blinded evaluators rated the quality of manuscripts at initial submission and final post peer review version. Of the 327 manuscripts submitted to the journal, 131 were accepted for further review, and 129 were randomized. Of those, 14 that were lost to follow-up showed no differences in initial quality to the followed-up papers. Hence, 115 were included in the main analysis, with 16 rejected for publication after peer review. 21 (18.3%) of the 115 included papers were interventions, 46 (40.0%) were longitudinal designs, 28 (24.3%) cross-sectional and 20 (17.4%) others. The 16 (13.9%) rejected papers had a significantly lower initial score on the overall Goodman scale than accepted papers (difference 15.0, 95% CI: 4.6-24.4). The effect of suggesting a guideline to the reviewers had no effect on change in overall quality as measured by the Goodman scale (0.9, 95% CI: -0.3-+2.1). The estimated effect of adding a statistical reviewer was 5.5 (95% CI: 4.3-6.7), showing a significant improvement in quality.Conclusions and significanceThis prospective randomized study shows the positive effect of adding a statistical reviewer to the field-expert peers in improving manuscript quality. We did not find a statistically significant positive effect by suggesting reviewers use reporting guidelines.

Dataset Information

Developing and evaluating criteria to help reviewers of biomedical informatics manuscripts.

Publications

Developing and evaluating criteria to help reviewers of biomedical informatics manuscripts.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets