Dataset Information

Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review.

ABSTRACT: BACKGROUND:Several scales, checklists and domain-based tools for assessing risk of reporting biases exist, but it is unclear how much they vary in content and guidance. We conducted a systematic review of the content and measurement properties of such tools. METHODS:We searched for potentially relevant articles in Ovid MEDLINE, Ovid Embase, Ovid PsycINFO and Google Scholar from inception to February 2017. One author screened all titles, abstracts and full text articles, and collected data on tool characteristics. RESULTS:We identified 18 tools that include an assessment of the risk of reporting bias. Tools varied in regard to the type of reporting bias assessed (eg, bias due to selective publication, bias due to selective non-reporting), and the level of assessment (eg, for the study as a whole, a particular result within a study or a particular synthesis of studies). Various criteria are used across tools to designate a synthesis as being at 'high' risk of bias due to selective publication (eg, evidence of funnel plot asymmetry, use of non-comprehensive searches). However, the relative weight assigned to each criterion in the overall judgement is unclear for most of these tools. Tools for assessing risk of bias due to selective non-reporting guide users to assess a study, or an outcome within a study, as 'high' risk of bias if no results are reported for an outcome. However, assessing the corresponding risk of bias in a synthesis that is missing the non-reported outcomes is outside the scope of most of these tools. Inter-rater agreement estimates were available for five tools. CONCLUSION:There are several limitations of existing tools for assessing risk of reporting biases, in terms of their scope, guidance for reaching risk of bias judgements and measurement properties. Development and evaluation of a new, comprehensive tool could help overcome present limitations.

SUBMITTER: Page MJ

PROVIDER: S-EPMC5857645 | biostudies-literature | 2018 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review.

Page Matthew J MJ McKenzie Joanne E JE Higgins Julian P T JPT

BMJ open 20180314 3

<h4>Background</h4>Several scales, checklists and domain-based tools for assessing risk of reporting biases exist, but it is unclear how much they vary in content and guidance. We conducted a systematic review of the content and measurement properties of such tools.<h4>Methods</h4>We searched for potentially relevant articles in Ovid MEDLINE, Ovid Embase, Ovid PsycINFO and Google Scholar from inception to February 2017. One author screened all titles, abstracts and full text articles, and collec ...[more]

PMID: 29540417

Similar Datasets

Project description:BACKGROUND:Systematic reviews are increasingly prevalent in environmental health due to their ability to synthesize evidence while reducing bias. Different systematic review methods have been developed by the US National Toxicology Program's Office of Health Assessment and Translation (OHAT), the US Environmental Protection Agency's (EPA) Integrated Risk Information System (IRIS), and by the US EPA under the Toxic Substances Control Act (TSCA), including the approach to assess risk of bias (ROB), one of the most vital steps which is used to evaluate internal validity of the studies. Our objective was to compare the performance of three tools (OHAT, IRIS, TSCA) in assessing ROB. METHODS:We selected a systematic review on polybrominated diphenyl ethers and intelligence quotient and/or attention deficit hyperactivity disorder because it had been endorsed by the National Academy of Sciences. Two reviewers followed verbatim instructions from the tools and independently applied each tool to assess ROB in 15 studies previously identified. We documented the time to apply each tool and the impact the ROB ratings for each tool had on the final rating of the quality of the overall body of evidence. RESULTS:The time to complete the ROB assessments varied widely (mean = 20, 32, and 40?min per study for the OHAT, IRIS, and TSCA tools, respectively). All studies were rated overall "low" or "uninformative" using IRIS, due to "deficient" or "critically deficient" ratings in one or two domains. Similarly, all studies were rated "unacceptable" using the TSCA tool because of one "unacceptable" rating in a metric related to statistical power. Approximately half of the studies had "low" or "probably low ROB" ratings across all domains with the OHAT and Navigation Guide tools. CONCLUSIONS:Tools that use overall ROB or study quality ratings, such as IRIS and TSCA, may reduce the available evidence to assess the harms of environmental exposures by erroneously excluding studies, which leads to inaccurate conclusions about the quality of the body of evidence. We recommend using ROB tools that circumvents these issues, such as OHAT and Navigation Guide. SYSTEMATIC REVIEW REGISTRATION:This review has not been registered as it is not a systematic review.

Project description:ObjectiveTo identify and assess the globally available valid, reliable and acceptable tools for assessing health research partnership outcomes and impacts.MethodsWe searched Ovid MEDLINE, Embase, CINAHL Plus and PsycINFO from origin to 2 June 2021, without limits, using an a priori strategy and registered protocol. We screened citations independently and in duplicate, resolving discrepancies by consensus and retaining studies involving health research partnerships, the development, use and/or assessment of tools to evaluate partnership outcomes and impacts, and reporting empirical psychometric evidence. Study, tool, psychometric and pragmatic characteristics were abstracted using a hybrid approach, then synthesized using descriptive statistics and thematic analysis. Study quality was assessed using the quality of survey studies in psychology (Q-SSP) checklist.ResultsFrom 56 123 total citations, we screened 36 027 citations, assessed 2784 full-text papers, abstracted data from 48 studies and one companion report, and identified 58 tools. Most tools comprised surveys, questionnaires and scales. Studies used cross-sectional or mixed-method/embedded survey designs and employed quantitative and mixed methods. Both studies and tools were conceptually well grounded, focusing mainly on outcomes, then process, and less frequently on impact measurement. Multiple forms of empirical validity and reliability evidence was present for most tools; however, psychometric characteristics were inconsistently assessed and reported. We identified a subset of studies (22) and accompanying tools distinguished by their empirical psychometric, pragmatic and study quality characteristics. While our review demonstrated psychometric and pragmatic improvements over previous reviews, challenges related to health research partnership assessment and the nascency of partnership science persist.ConclusionThis systematic review identified multiple tools demonstrating empirical psychometric evidence, pragmatic strength and moderate study quality. Increased attention to psychometric and pragmatic requirements in tool development, testing and reporting is key to advancing health research partnership assessment and partnership science. PROSPERO CRD42021137932.

Project description:BackgroundThe last decade has seen growing interest in scaling up of innovations to strengthen healthcare systems. However, the lack of appropriate methods for determining their potential for scale-up is an unfortunate global handicap. Thus, we aimed to review tools proposed for assessing the scalability of innovations in health.MethodsWe conducted a systematic review following the COSMIN methodology. We included any empirical research which aimed to investigate the creation, validation or interpretability of a scalability assessment tool in health. We searched Embase, MEDLINE, CINAHL, Web of Science, PsycINFO, Cochrane Library and ERIC from their inception to 20 March 2019. We also searched relevant websites, screened the reference lists of relevant reports and consulted experts in the field. Two reviewers independently selected and extracted eligible reports and assessed the methodological quality of tools. We summarized data using a narrative approach involving thematic syntheses and descriptive statistics.ResultsWe identified 31 reports describing 21 tools. Types of tools included criteria (47.6%), scales (33.3%) and checklists (19.0%). Most tools were published from 2010 onwards (90.5%), in open-access sources (85.7%) and funded by governmental or nongovernmental organizations (76.2%). All tools were in English; four were translated into French or Spanish (19.0%). Tool creation involved single (23.8%) or multiple (19.0%) types of stakeholders, or stakeholder involvement was not reported (57.1%). No studies reported involving patients or the public, or reported the sex of tool creators. Tools were created for use in high-income countries (28.6%), low- or middle-income countries (19.0%), or both (9.5%), or for transferring innovations from low- or middle-income countries to high-income countries (4.8%). Healthcare levels included public or population health (47.6%), primary healthcare (33.3%) and home care (4.8%). Most tools provided limited information on content validity (85.7%), and none reported on other measurement properties. The methodological quality of tools was deemed inadequate (61.9%) or doubtful (38.1%).ConclusionsWe inventoried tools for assessing the scalability of innovations in health. Existing tools are as yet of limited utility for assessing scalability in health. More work needs to be done to establish key psychometric properties of these tools. Trial registration We registered this review with PROSPERO (identifier: CRD42019107095).

Project description:Introduction: Intensive monitoring (IM) is one of the methods of post-marketing active surveillance based upon event monitoring, which has received interest in the current medicines regulatory landscape. For a specific period of time, IM involves primary data collection and is actively focused on gathering longitudinal information, mainly safety, since the first day of drug use. Objectives: To describe IM systems and studies' data published over 11-years period (2006-2016). Specifically, we reviewed study population/event surveillance, methodological approaches, limitations, and its applications in the real-world evidence generation data. Methods: We completed a systematic search of MEDLINE and EMBASE to identify studies published from 2006 to 2016, that used IM methodology. We extracted data using a standardized form and results were analyzed descriptively. The methodological quality of selected studies was assessed using the modified Downs and Black checklist. Results: From 1,400 screened citations, we identified 86 papers, corresponding to 69 different studies. Seventy percent of reviewed studies corresponded to established IM systems, of which, more than half were prescription event monitoring (PEM) and modified-PEM. Among non-established IM systems, vaccines were the most common studied drugs (n = 14). The median cohort size ranged from 488 (hospitals) to 10,479 (PEM) patients. Patients and caregivers were the event data source in 39.1% of studies. The mean overall quality score was similar between established and non-established IM. Conclusions: Over the study period, IM studies were implemented in 26 countries with different maturity levels of post-marketing surveillance systems. We identified two major limitations: only 20% of studies were conducted at hospital-level, which is a matter of concern, insofar as healthcare systems are facing a lack of access to new medicines at ambulatory care level. Additionally, IM access to data of drug exposure cohorts, either at identification or at follow-up stages, could somehow constitute a barrier, given the complexity of managerial, linkable, and privacy data issues.

Dataset Information

Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review.

Publications

Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets