Dataset Information

Tools used to assess the quality of peer review reports: a methodological systematic review.

ABSTRACT: BACKGROUND:A strong need exists for a validated tool that clearly defines peer review report quality in biomedical research, as it will allow evaluating interventions aimed at improving the peer review process in well-performed trials. We aim to identify and describe existing tools for assessing the quality of peer review reports in biomedical research. METHODS:We conducted a methodological systematic review by searching PubMed, EMBASE (via Ovid) and The Cochrane Methodology Register (via The Cochrane Library) as well as Google® for all reports in English describing a tool for assessing the quality of a peer review report in biomedical research. Data extraction was performed in duplicate using a standardized data extraction form. We extracted information on the structure, development and validation of each tool. We also identified quality components across tools using a systematic multi-step approach and we investigated quality domain similarities among tools by performing hierarchical, complete-linkage clustering analysis. RESULTS:We identified a total number of 24 tools: 23 scales and 1 checklist. Six tools consisted of a single item and 18 had several items ranging from 4 to 26. None of the tools reported a definition of 'quality'. Only 1 tool described the scale development and 10 provided measures of validity and reliability. Five tools were used as an outcome in a randomized controlled trial (RCT). Moreover, we classified the quality components of the 18 tools with more than one item into 9 main quality domains and 11 subdomains. The tools contained from two to seven quality domains. Some domains and subdomains were considered in most tools such as the detailed/thorough (11/18) nature of reviewer's comments. Others were rarely considered, such as whether or not the reviewer made comments on the statistical methods (1/18). CONCLUSION:Several tools are available to assess the quality of peer review reports; however, the development and validation process is questionable and the concepts evaluated by these tools vary widely. The results from this study and from further investigations will inform the development of a new tool for assessing the quality of peer review reports in biomedical research.

SUBMITTER: Superchi C

PROVIDER: S-EPMC6402095 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundSystematic reviews (SRs) are considered the highest level of evidence to answer research questions; however, they are time and resource intensive.ObjectiveWhen comparing SR tasks done manually, using standard methods, versus those same SR tasks done using automated tools, (1) what is the difference in time to complete the SR task and (2) what is the impact on the error rate of the SR task?MethodsA case study compared specific tasks done during the conduct of an SR on prebiotic, probiotic, and synbiotic supplementation in chronic kidney disease. Two participants (manual team) conducted the SR using current methods, comprising a total of 16 tasks. Another two participants (automation team) conducted the tasks where a systematic review automation (SRA) tool was available, comprising of a total of six tasks. The time taken and error rate of the six tasks that were completed by both teams were compared.ResultsThe approximate time for the manual team to produce a draft of the background, methods, and results sections of the SR was 126 hours. For the six tasks in which times were compared, the manual team spent 2493 minutes (42 hours) on the tasks, compared to 708 minutes (12 hours) spent by the automation team. The manual team had a higher error rate in two of the six tasks-regarding Task 5: Run the systematic search, the manual team made eight errors versus three errors made by the automation team; regarding Task 12: Assess the risk of bias, 25 assessments differed from a reference standard for the manual team compared to 20 differences for the automation team. The manual team had a lower error rate in one of the six tasks-regarding Task 6: Deduplicate search results, the manual team removed one unique study and missed zero duplicates versus the automation team who removed two unique studies and missed seven duplicates. Error rates were similar for the two remaining compared tasks-regarding Task 7: Screen the titles and abstracts and Task 9: Screen the full text, zero relevant studies were excluded by both teams. One task could not be compared between groups-Task 8: Find the full text.ConclusionsFor the majority of SR tasks where an SRA tool was used, the time required to complete that task was reduced for novice researchers while methodological quality was maintained.

Project description:There is a lack of standardized measurement tools globally to assess knowledge, attitudes, and perceptions of expecting women toward prenatal screening. The purpose of this systematic review was to identify reasons women pursue or decline prenatal screening and compare the strengths and limitations of available measurement tools used to assess pregnant women's perceptions, knowledge, and attitudes toward prenatal screening. This review followed the five-step York methodology by Arksey and O'Malley and incorporated recommendations from the Preferred Reporting Items for Systematic Reviews and Meta-Analysis checklist for the extraction, analysis, and presentation of results. The five steps consisted of: (1) identification of the research questions; (2) searching for relevant studies; (3) selection of studies relevant to the research questions; (4) data charting; and (5) collation, summarization, and reporting of results. Four online databases (PubMed, Embase, Web of Science, and Cochrane Library) were selected after the librarian's development of a detailed search strategy. The Rayyan platform was used between June 2023 and August 2023 to epitomize the articles produced from our search. A total of 68 eligible studies were included in the analysis. The top five major reasons for declining prenatal screening uptake included (1) being unsure of the risk of prenatal screening and harm to the baby or miscarriage (n = 15), (2) not considering action such as termination of pregnancy for prenatal screening to be considered as necessary (n = 14), (3) high cost (n = 12), (4) lack of knowledge about testing procedures and being anxious about the test (n = 10), and (5) being worried about probability of false negative or false positive results (n = 6). Only 32 studies utilized scientifically validated instruments. Difficulties in capturing representative, adequately sized samples inclusive of diverse ethnicities and demographics were pervasive. Findings highlight the need for rigorous validation of research measurement methodologies to ensure the accuracy and applicability of resulting data regarding the assessment of prenatal screening perceptions, knowledge, and attitudes across diverse female populations.Registration: N/A.

Project description:ObjectiveTo examine whether network meta-analyses, increasingly used to assess comparative effectiveness of healthcare interventions, follow the key methodological recommendations for reporting and conduct of systematic reviews.DesignMethodological systematic review of reports of network meta-analyses.Data sourcesCochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, Medline, and Embase, searched from inception to 12 July 2012.Review methodsAll network meta-analyses comparing clinical efficacy of three or more interventions based on randomised controlled trials, excluding meta-analyses with an open loop network of three interventions. We assessed the reporting of general characteristics and key methodological components of the systematic review process using two composite outcomes. For some components, if reporting was adequate, we assessed their conduct quality.ResultsOf 121 network meta-analyses covering a wide range of medical areas, 100 (83%) assessed pharmacological interventions and 11 (9%) non-pharmacological interventions; 56 (46%) were published in journals with a high impact factor. The electronic search strategy for each database was not reported in 88 (73%) network meta-analyses; for 36 (30%), the primary outcome was not clearly identified. Overall, 61 (50%) network meta-analyses did not report any information regarding the assessment of risk of bias of individual studies, and 103 (85%) did not report any methods to assess the likelihood of publication bias. Overall, 87 (72%) network meta-analyses did not report the literature search, searched only one database, did not search other sources, or did not report an assessment of risk of bias of individual studies. These methodological components did not differ by publication in a general or specialty journal or by public or private funding.ConclusionsEssential methodological components of the systematic review process-conducting a literature search and assessing risk of bias of individual studies-are frequently lacking in reports of network meta-analyses, even when published in journals with high impact factors.

Project description:BACKGROUND:A crucial element in the systematic review (SR) methodology is the appraisal of included primary studies, using tools for assessment of methodological quality or risk of bias (RoB). SR authors can conduct sensitivity analyses to explore whether their results are sensitive to exclusion of low quality studies or a high RoB. However, it is unknown which tools do SR authors use for assessing quality/RoB, and how they set threshold for quality/RoB in sensitivity analyses. The aim of this study was to assess quality/RoB assessment tools, the types of sensitivity analyses and quality/RoB thresholds for sensitivity analyses used within SRs published in high-impact pain/anesthesiology journals. METHODS:This was a methodological study. We analyzed SRs published from January 2005 to June 2018 in the 25% highest-ranking journals within the Journal Citation Reports (JCR) "Anesthesiology" category. We retrieved the SRs from PubMed. Two authors independently screened records, full texts, and extracted data on quality/RoB tools and sensitivity analyses. We extracted data about quality/RoB tools, types of sensitivity analyses and the thresholds for quality/RoB used in them. RESULTS:Out of 678 analyzed SRs, 513 (76%) reported the use of quality/RoB assessments. The most commonly reported tools for assessing quality/RoB in the studies were the Cochrane tool for risk of bias assessment (N?=?251; 37%) and Jadad scale (N?=?99; 15%). Meta-analysis was conducted in 451 (66%) of SRs and sensitivity analysis in 219/451 (49%). Most commonly, sensitivity analysis was conducted to explore the influence of study quality/RoB (90/219; 41%) on the results. Quality/RoB thresholds used for sensitivity analysis for those studies were clearly reported in 47 (52%) articles that used them. The quality/RoB thresholds used for sensitivity analyses were highly heterogeneous and inconsistent, even when the same tool was used. CONCLUSIONS:A quarter of SRs reported using quality/RoB assessments, and some of them cited tools that are not meant for assessing quality/RoB. Authors who use quality/RoB to explore the robustness of their results in meta-analyses use highly heterogeneous quality/RoB thresholds in sensitivity analyses. Better methodological consistency for quality/RoB sensitivity analyses is needed.

Dataset Information

Tools used to assess the quality of peer review reports: a methodological systematic review.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets