Developing and evaluating criteria to help reviewers of biomedical informatics manuscripts.
Ontology highlight
ABSTRACT:
SUBMITTER: Ammenwerth E
PROVIDER: S-EPMC212789 | biostudies-literature | 2003 Sep-Oct
REPOSITORIES: biostudies-literature
Publications
Developing and evaluating criteria to help reviewers of biomedical informatics manuscripts.
Ammenwerth Elske E Wolff Astrid C AC Knaup Petra P Ulmer Hanno H Skonetzki Stefan S van Bemmel Jan H JH McCray Alexa T AT Haux Reinhold R Kulikowski Casimir C
Journal of the American Medical Informatics Association : JAMIA 20030604 5
Peer-reviewed publication of scientific research results represents the most important means of their communication. The authors have annually reviewed a large heterogeneous set of papers to produce the International Medical Informatics Association (IMIA) Yearbook of Medical Informatics. To support an objective and high-quality review process, the authors attempted to provide reviewers with a set of refined quality criteria, comprised of 80 general criteria and an additional 60 criteria for spec ...[more]
Similar Datasets
Project description:MotivationMany researchers with domain expertise are unable to easily apply machine learning (ML) to their bioinformatics data due to a lack of ML and/or coding expertise. Methods that have been proposed thus far to automate ML mostly require programming experience as well as expert knowledge to tune and apply the algorithms correctly. Here, we study a method of automating biomedical data science using a web-based AI platform to recommend model choices and conduct experiments. We have two goals in mind: first, to make it easy to construct sophisticated models of biomedical processes; and second, to provide a fully automated AI agent that can choose and conduct promising experiments for the user, based on the user's experiments as well as prior knowledge. To validate this framework, we conduct an experiment on 165 classification problems, comparing to state-of-the-art, automated approaches. Finally, we use this tool to develop predictive models of septic shock in critical care patients.ResultsWe find that matrix factorization-based recommendation systems outperform metalearning methods for automating ML. This result mirrors the results of earlier recommender systems research in other domains. The proposed AI is competitive with state-of-the-art automated ML methods in terms of choosing optimal algorithm configurations for datasets. In our application to prediction of septic shock, the AI-driven analysis produces a competent ML model (AUROC 0.85±0.02) that performs on par with state-of-the-art deep learning results for this task, with much less computational effort.Availability and implementationPennAI is available free of charge and open-source. It is distributed under the GNU public license (GPL) version 3.Supplementary informationSupplementary data are available at Bioinformatics online.
| S-EPMC8055228 | biostudies-literature
Project description:Authors endure considerable hardship carrying out biomedical research, from generating ideas to completing their manuscripts and submitting their findings and data (as is increasingly required) to a journal. When researchers submit to journals, they entrust their findings and ideas to editors and peer reviewers who are expected to respect the confidentiality of peer review. Inherent trust in peer review is built on the ethical conduct of authors, editors and reviewers, and on the respect of this confidentiality. If such confidentiality is breached by unethical reviewers who might steal or plagiarize the authors' ideas, researchers will lose trust in peer review and may resist submitting their findings to that journal. Science loses as a result, scientific and medical advances slow down, knowledge may become scarce, and it is unlikely that increasing bias in the literature will be detected or eliminated. In such a climate, society will ultimately be deprived from scientific and medical advances. Despite a rise in documented cases of abused peer review, there is still a relative lack of qualitative and quantitative studies on reviewer-related misconduct, most likely because evidence is difficult to come by. Our paper presents an assessment of editors' and reviewers' responsibilities in preserving the confidentiality of manuscripts during the peer review process, in response to a 2016 case of intellectual property theft by a reviewer. Our main objectives are to propose additional measures that would offer protection of authors' intellectual ideas from predatory reviewers, and increase researchers' awareness of the responsible reviewing of journal articles and reporting of biomedical research.
| S-EPMC6310661 | biostudies-literature
Project description:ObjectivesTo describe gender and geographical inequalities in invitations to review and the response to these invitations and to assess whether inequalities increased during the covid-19 pandemic.DesignRetrospective cohort study.Setting19 specialist medical journals and two large general medical journals from BMJ Publishing Group.PopulationReviewers invited to review manuscripts submitted between 1 January 2018 and 31 May 2021. The cohort was followed up to 28 February 2022.Main outcome measuresReviewer's agreement to review.ResultsA total of 257 025 reviewers were invited (38.6% (88 454/228 869) women), and 90 467 (35.2%) agreed to review. Invited reviewers were mainly (217 682; 84.7%) affiliated with high income countries: Europe (122 414; 47.6%), North America (66 931; 26.0%), Africa (25 735; 10.0%), Asia (22 693; 8.8%), Oceania (16 175; 6.3%), and South America (3076; 1.2%). Independent factors associated with agreement to review were gender (odds ratio 0.89, 95% confidence interval 0.87 to 0.92, for women compared with men), geographical affiliation (2.89, 2.73 to 3.06, for Asia; 3.32, 2.94 to 3.75 for South America; 1.35, 1.27 to 1.43, for Oceania; and 0.35, 0.33 to 0.37, for Africa compared with Europe), and country income (0.47, 0.45 to 0.49, for upper middle income; 5.12, 4.67 to 5.61, for lower middle income; and 4.66, 3.79 to 5.73, for low income compared with high income country). Agreement was also independently associated with editor's gender (0.96, 0.93 to 0.99, for women compared with men), last author's geographical affiliation (0.80, 0.78 to 0.83, for Asia; 0.89, 0.85 to 0.94, for Oceania compared with Europe), impact factor (1.78, 1.27 to 2.50, for >10 compared with <5), and type of peer review process (0.52, 0.35 to 0.77, for open compared with anonymised). During the first and second phases of the pandemic, agreement was lower than in the pre-pandemic period (P<0.001). The interaction between time periods and covid-19 related topic and reviewer's gender was non-significant. However, significant interaction was found between time periods and covid-19 related topic and reviewer's geographical affiliation.ConclusionsTo reduce bias and improve diversity, editors need to identify and implement effective strategies and continually evaluate progress against these to ensure that more women and researchers from upper middle income and low income countries are involved in review.
| S-EPMC10471900 | biostudies-literature
Project description:BACKGROUND:Peer review is at the heart of the scientific process. With the advent of digitisation, journals started to offer electronic articles or publishing online only. A new philosophy regarding the peer review process found its way into academia: the open peer review. Open peer review as practiced by BioMed Central (BMC) is a type of peer review where the names of authors and reviewers are disclosed and reviewer comments are published alongside the article. A number of articles have been published to assess peer reviews using quantitative research. However, no studies exist that used qualitative methods to analyse the content of reviewers' comments. METHODS:A focused mapping review and synthesis (FMRS) was undertaken of manuscripts reporting qualitative research submitted to BMC open access journals from 1 January - 31 March 2018. Free-text reviewer comments were extracted from peer review reports using a 77-item classification system organised according to three key dimensions that represented common themes and sub-themes. A two stage analysis process was employed. First, frequency counts were undertaken that allowed revealing patterns across themes/sub-themes. Second, thematic analysis was conducted on selected themes of the narrative portion of reviewer reports. RESULTS:A total of 107 manuscripts submitted to nine open-access journals were included in the FMRS. The frequency analysis revealed that among the 30 most frequently employed themes "writing criteria" (dimension II) is the top ranking theme, followed by comments in relation to the "methods" (dimension I). Besides that, some results suggest an underlying quantitative mindset of reviewers. Results are compared and contrasted in relation to established reporting guidelines for qualitative research to inform reviewers and authors of frequent feedback offered to enhance the quality of manuscripts. CONCLUSIONS:This FMRS has highlighted some important issues that hold lessons for authors, reviewers and editors. We suggest modifying the current reporting guidelines by including a further item called "Degree of data transformation" to prompt authors and reviewers to make a judgment about the appropriateness of the degree of data transformation in relation to the chosen analysis method. Besides, we suggest that completion of a reporting checklist on submission becomes a requirement.
| S-EPMC7236308 | biostudies-literature
Project description:ObjectivesTo examine roles for summer internship programs in expanding pathways into biomedical informatics, based on 10 years of the Vanderbilt Department of Biomedical Informatics (DBMI) Summer Research Internship Program.Materials and methodsVanderbilt DBMI's internship program is a research-intensive paid 8-10 week program for high school, undergraduate, and graduate students. The program is grounded in a "Windows, Mirrors, and Open Doors" educational framework, and is guided by an evolving set of design principles, including providing meaningful research experiences, applying a multi-factor approach to diversity, and helping interns build peer connections.ResultsOver 10 years, 90 individuals have participated in the internship program, with nine students participating for more than one summer. Of 90 participants, 38 were women and 52 were men. Participants represented a range of racial/ethnic groups. A total of 39 faculty members have served as primary mentor for one or more interns. Five key lessons emerged from our program experience: Festina Lente ("Make haste slowly"), The Power of Community, Learning by Doing, Thoughtful Partnerships Lead to Innovation, and The Whole is More Than the Sum of Its Parts.DiscussionBased on our experience, we suggest that internship programs should become a core element of the biomedical informatics educational ecosystem. Continued development and growth of this important educational outreach approach requires stable funding sources and building connections between programs to share best practices.ConclusionInternship programs can play a substantial role in the biomedical informatics educational ecosystem, helping introduce individuals to the field earlier in their educational trajectories.
| S-EPMC6951956 | biostudies-literature
Project description:BackgroundAlthough peer review is widely considered to be the most credible way of selecting manuscripts and improving the quality of accepted papers in scientific journals, there is little evidence to support its use. Our aim was to estimate the effects on manuscript quality of either adding a statistical peer reviewer or suggesting the use of checklists such as CONSORT or STARD to clinical reviewers or both.Methodology and principal findingsInterventions were defined as 1) the addition of a statistical reviewer to the clinical peer review process, and 2) suggesting reporting guidelines to reviewers; with "no statistical expert" and "no checklist" as controls. The two interventions were crossed in a 2x2 balanced factorial design including original research articles consecutively selected, between May 2004 and March 2005, by the Medicina Clinica (Barc) editorial committee. We randomized manuscripts to minimize differences in terms of baseline quality and type of study (intervention, longitudinal, cross-sectional, others). Sample-size calculations indicated that 100 papers provide an 80% power to test a 55% standardized difference. We specified the main outcome as the increment in quality of papers as measured on the Goodman Scale. Two blinded evaluators rated the quality of manuscripts at initial submission and final post peer review version. Of the 327 manuscripts submitted to the journal, 131 were accepted for further review, and 129 were randomized. Of those, 14 that were lost to follow-up showed no differences in initial quality to the followed-up papers. Hence, 115 were included in the main analysis, with 16 rejected for publication after peer review. 21 (18.3%) of the 115 included papers were interventions, 46 (40.0%) were longitudinal designs, 28 (24.3%) cross-sectional and 20 (17.4%) others. The 16 (13.9%) rejected papers had a significantly lower initial score on the overall Goodman scale than accepted papers (difference 15.0, 95% CI: 4.6-24.4). The effect of suggesting a guideline to the reviewers had no effect on change in overall quality as measured by the Goodman scale (0.9, 95% CI: -0.3-+2.1). The estimated effect of adding a statistical reviewer was 5.5 (95% CI: 4.3-6.7), showing a significant improvement in quality.Conclusions and significanceThis prospective randomized study shows the positive effect of adding a statistical reviewer to the field-expert peers in improving manuscript quality. We did not find a statistically significant positive effect by suggesting reviewers use reporting guidelines.
| S-EPMC1824709 | biostudies-literature
Project description:Analysis of high-throughput experiments in the life sciences frequently relies upon standardized information about genes, gene products, and other biological entities. To provide this information, expert curators are increasingly relying on text mining tools to identify, extract and harmonize statements from biomedical journal articles that discuss findings of interest. For determining reliability of the statements, curators need the evidence used by the authors to support their assertions. It is important to annotate the evidence directly used by authors to qualify their findings rather than simply annotating mentions of experimental methods without the context of what findings they support. Text mining tools require tuning and adaptation to achieve accurate performance. Many annotated corpora exist to enable developing and tuning text mining tools; however, none currently provides annotations of evidence based on the extensive and widely used Evidence and Conclusion Ontology. We present the ECO-CollecTF corpus, a novel, freely available, biomedical corpus of 84 documents that captures high-quality, evidence-based statements annotated with the Evidence and Conclusion Ontology.
| S-EPMC8313968 | biostudies-literature
Project description:The primary aims of peer review are to detect flaws and deficiencies in the design and interpretation of studies, and ensure the clarity and quality of their presentation. However, it has been questioned whether peer review fulfils this function. Studies have highlighted a stronger focus of reviewers on critiquing methodological aspects of studies and the quality of writing in biomedical sciences, with less focus on theoretical grounding. In contrast, reviewers in the social sciences appear more concerned with theoretical underpinnings. These studies also found the effect of peer review on manuscripts' content to be variable, but generally modest and positive. I qualitatively analysed 1430 peer reviewers' comments for a sample of 40 social science preprint-publication pairs to identify the key foci of reviewers' comments. I then quantified the effect of peer review on manuscripts by examining differences between the preprint and published versions using the normalised Levenshtein distance, cosine similarity, and word count ratios for titles, abstracts, document sections and full-texts. I also examined changes in references used between versions and linked changes to reviewers' comments. Reviewers' comments were nearly equally split between issues of methodology (30.7%), theory (30.0%), and writing quality (29.2%). Titles, abstracts, and the semantic content of documents remained similar, although publications were typically longer than preprints. Two-thirds of citations were unchanged, 20.9% were added during review and 13.1% were removed. These findings indicate reviewers equally attended to the theoretical and methodological details and communication style of manuscripts, although the effect on quantitative measures of the manuscripts was limited.Supplementary informationThe online version contains supplementary material available at 10.1007/s11192-022-04357-y.
| S-EPMC8993676 | biostudies-literature
Project description:BackgroundPeer review is widely used in academic fields to assess a manuscript's significance and to improve its quality for publication. This scoping review will assess existing peer review guidelines and/or checklists intended for reviewers of biomedical journals and provide an overview on the review guidelines.MethodsPubMed, Embase, and Allied and Complementary Medicine (AMED) databases were searched for review guidelines from the date of inception until February 19, 2021. There was no date restriction nor article type restriction. In addition to the database search, websites of journal publishers and non-publishers were additionally hand-searched.ResultsOf 14,633 database publication records and 24 website records, 65 publications and 14 websites met inclusion criteria for the review (78 records in total). From the included records, a total of 1,811 checklist items were identified. The items related to Methods, Results, and Discussion were found to be the highly discussed in reviewer guidelines.ConclusionThis review identified existing literature on peer review guidelines and provided an overview of the current state of peer review guides. Review guidelines were varying by journals and publishers. This calls for more research to determine the need to use uniform review standards for transparent and standardized peer review.Protocol registrationThe protocol for this study has been registered at Research Registry (www.researchregistry.com): reviewregistry881.
| S-EPMC8136639 | biostudies-literature
Project description:The Function Biomedical Informatics Research Network (FBIRN) developed methods and tools for conducting multi-scanner functional magnetic resonance imaging (fMRI) studies. Method and tool development were based on two major goals: 1) to assess the major sources of variation in fMRI studies conducted across scanners, including instrumentation, acquisition protocols, challenge tasks, and analysis methods, and 2) to provide a distributed network infrastructure and an associated federated database to host and query large, multi-site, fMRI and clinical data sets. In the process of achieving these goals the FBIRN test bed generated several multi-scanner brain imaging data sets to be shared with the wider scientific community via the BIRN Data Repository (BDR). The FBIRN Phase 1 data set consists of a traveling subject study of 5 healthy subjects, each scanned on 10 different 1.5 to 4 T scanners. The FBIRN Phase 2 and Phase 3 data sets consist of subjects with schizophrenia or schizoaffective disorder along with healthy comparison subjects scanned at multiple sites. In this paper, we provide concise descriptions of FBIRN's multi-scanner brain imaging data sets and details about the BIRN Data Repository instance of the Human Imaging Database (HID) used to publicly share the data.
| S-EPMC4651841 | biostudies-literature