Dataset Information

Managing the incidence of selective reporting bias: a survey of Cochrane review groups.

ABSTRACT: Selective reporting bias (SRB), the incomplete publication of outcomes measured or of analyses performed in a study, may lead to the over- or underestimation of treatment effects or harms. Cochrane systematic reviews of interventions are required to assess the risk of SRB, achieved in part by applying the Cochrane risk of bias tool to each included randomised trial. The Cochrane Handbook outlines strategies for a comprehensive risk of bias assessment, but the extent to which these are followed by Cochrane review groups (CRGs) has not been assessed to date. The objective of this study was to determine the methods which CRGs require of their authors to address SRB within systematic reviews, and how SRB risk assessments are verified.A cross-sectional survey was developed and distributed electronically to the 52 CRGs involved in intervention reviews.Responses from 42 CRGs show that the majority refer their authors to the Cochrane Handbook for specific instruction regarding assessments of SRB. The handbook strategies remain variably enforced, with 57 % (24/42) of CRGs not requiring review authors to search for included trial protocols and 31 % (13/42) not requiring that contact with individual study authors be attempted. Only half (48 %, 20/42) of the groups consistently verify review authors' assessments of the risk of SRB to ensure completeness.A range of practices are used by CRGs for addressing SRB, with many steps outlined in the Cochrane Handbook being encouraged but not required. The majority of CRGs do not consider their review authors to be sufficiently competent to assess for SRB, yet risk of bias assessments are not always verified by editors before publication. The implications of SRB may not be fully appreciated by all CRGs, and resolving the identified issues may require an approach targeting several steps in the systematic review process.

SUBMITTER: Reid EK

PROVIDER: S-EPMC4472158 | biostudies-literature | 2015 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Managing the incidence of selective reporting bias: a survey of Cochrane review groups.

Reid Emma K EK Tejani Aaron M AM Huan Lawrence N LN Egan Gregory G O'Sullivan Cait C Mayhew Alain D AD Kabir Monisha M

Systematic reviews 20150613

<h4>Background</h4>Selective reporting bias (SRB), the incomplete publication of outcomes measured or of analyses performed in a study, may lead to the over- or underestimation of treatment effects or harms. Cochrane systematic reviews of interventions are required to assess the risk of SRB, achieved in part by applying the Cochrane risk of bias tool to each included randomised trial. The Cochrane Handbook outlines strategies for a comprehensive risk of bias assessment, but the extent to which t ...[more]

PMID: 26071043

Similar Datasets

Project description:IntroductionSelective reporting bias occurs when chance or selective outcome reporting rather than the intervention contributes to group differences. The prevailing concern about selective reporting bias is the possibility of results being modified towards specific conclusions. In this study, we evaluate randomized controlled trials (RCTs) published in hematology journals, a group in which selective outcome reporting has not yet been explored.MethodsOur primary goal was to examine discrepancies between the reported primary and secondary outcomes in registered and published RCTs concerning hematological malignancies reported in hematology journals with a high impact factor. The secondary goals were to address whether outcome reporting discrepancies favored statistically significant outcomes, whether a pattern existed between the funding source and likelihood of outcome reporting bias, and whether temporal trends were present in outcome reporting bias. For trials with major outcome discrepancies, we contacted trialists to determine reasons for these discrepancies. Trials published between January 1, 2010 and December 31, 2015 in Blood; British Journal of Haematology; American Journal of Hematology; Leukemia; and Haematologica were included.ResultsOf 499 RCTs screened, 109 RCTs were included. Our analysis revealed 118 major discrepancies and 629 total discrepancies. Among the 118 discrepancies, 30 (25.4%) primary outcomes were demoted, 47 (39.8%) primary outcomes were omitted, and 30 (25.4%) primary outcomes were added. Three (2.5%) secondary outcomes were upgraded to a primary outcome. The timing of assessment for a primary outcome changed eight (6.8%) times. Thirty-one major discrepancies were published with a P-value and twenty-five (80.6%) favored statistical significance. A majority of authors whom we contacted cited a pre-planned subgroup analysis as a reason for outcome changes.ConclusionOur results suggest that outcome changes occur frequently in hematology trials. Because RCTs ultimately underpin clinical judgment and guide policy implementation, selective reporting could pose a threat to medical decision making.

Project description:BackgroundIn 2008, the Cochrane Collaboration introduced a tool for assessing the risk of bias in clinical trials included in Cochrane reviews. The risk of bias (RoB) tool is based on narrative descriptions of evidence-based methodological features known to increase the risk of bias in trials.MethodsTo assess the usability of this tool, we conducted an evaluation by means of focus groups, online surveys and a face-to-face meeting. We obtained feedback from a range of stakeholders within The Cochrane Collaboration regarding their experiences with, and perceptions of, the RoB tool and associated guidance materials. We then assessed this feedback in a face-to-face meeting of experts and stakeholders and made recommendations for improvements and further developments of the RoB tool.ResultsThe survey attracted 380 responses. Respondents reported taking an average of between 10 and 60 minutes per study to complete their RoB assessments, which 83% deemed acceptable. Most respondents (87% of authors and 95% of editorial staff) thought RoB assessments were an improvement over past approaches to trial quality assessment. Most authors liked the standardized approach (81%) and the ability to provide quotes to support judgements (74%). A third of participants disliked the increased workload and found the wording describing RoB judgements confusing. The RoB domains reported to be the most difficult to assess were incomplete outcome data and selective reporting of outcomes. Authors expressed the need for more guidance on how to incorporate RoB assessments into meta-analyses and review conclusions. Based on this evaluation, recommendations were made for improvements to the RoB tool and the associated guidance. The implementation of these recommendations is currently underway.ConclusionsOverall, respondents identified positive experiences and perceptions of the RoB tool. Revisions of the tool and associated guidance made in response to this evaluation, and improved provision of training, may improve implementation.

Project description:Reporting bias represents a major problem in the assessment of health care interventions. Several prominent cases have been described in the literature, for example, in the reporting of trials of antidepressants, Class I anti-arrhythmic drugs, and selective COX-2 inhibitors. The aim of this narrative review is to gain an overview of reporting bias in the medical literature, focussing on publication bias and selective outcome reporting. We explore whether these types of bias have been shown in areas beyond the well-known cases noted above, in order to gain an impression of how widespread the problem is. For this purpose, we screened relevant articles on reporting bias that had previously been obtained by the German Institute for Quality and Efficiency in Health Care in the context of its health technology assessment reports and other research work, together with the reference lists of these articles.We identified reporting bias in 40 indications comprising around 50 different pharmacological, surgical (e.g. vacuum-assisted closure therapy), diagnostic (e.g. ultrasound), and preventive (e.g. cancer vaccines) interventions. Regarding pharmacological interventions, cases of reporting bias were, for example, identified in the treatment of the following conditions: depression, bipolar disorder, schizophrenia, anxiety disorder, attention-deficit hyperactivity disorder, Alzheimer's disease, pain, migraine, cardiovascular disease, gastric ulcers, irritable bowel syndrome, urinary incontinence, atopic dermatitis, diabetes mellitus type 2, hypercholesterolaemia, thyroid disorders, menopausal symptoms, various types of cancer (e.g. ovarian cancer and melanoma), various types of infections (e.g. HIV, influenza and Hepatitis B), and acute trauma. Many cases involved the withholding of study data by manufacturers and regulatory agencies or the active attempt by manufacturers to suppress publication. The ascertained effects of reporting bias included the overestimation of efficacy and the underestimation of safety risks of interventions.In conclusion, reporting bias is a widespread phenomenon in the medical literature. Mandatory prospective registration of trials and public access to study data via results databases need to be introduced on a worldwide scale. This will allow for an independent review of research data, help fulfil ethical obligations towards patients, and ensure a basis for fully-informed decision making in the health care system.

Project description:BACKGROUND:The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias and outcome reporting bias have been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. METHODOLOGY/PRINCIPAL FINDINGS:In this update, we review and summarise the evidence from cohort studies that have assessed study publication bias or outcome reporting bias in randomised controlled trials. Twenty studies were eligible of which four were newly identified in this update. Only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Fifteen of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40-62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies. CONCLUSIONS:This update does not change the conclusions of the review in which 16 studies were included. Direct empirical evidence for the existence of study publication bias and outcome reporting bias is shown. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.

Project description:Meta-analyses play an important role in synthesizing evidence from diverse studies and datasets that address similar questions. A major obstacle for meta-analyses arises from biases in reporting. In particular, it is speculated that findings which do not achieve formal statistical significance are less likely reported than statistically significant findings. Moreover, the patterns of bias can be complex and may also depend on the timing of the research results and their relationship with previously published work. In this paper, we present an approach that is specifically designed to analyze large-scale datasets on published results. Such datasets are currently emerging in diverse research fields, particularly in molecular medicine. We use our approach to investigate a dataset on Alzheimer's disease (AD) that covers 1167 results from case-control studies on 102 genetic markers. We observe that initial studies on a genetic marker tend to be substantially more biased than subsequent replications. The chances for initial, statistically non-significant results to be published are estimated to be about 44% (95% CI, 32% to 63%) relative to statistically significant results, while statistically non-significant replications have almost the same chance to be published as statistically significant replications (84%; 95% CI, 66% to 107%). Early replications tend to be biased against initial findings, an observation previously termed Proteus phenomenon: The chances for non-significant studies going in the same direction as the initial result are estimated to be lower than the chances for non-significant studies opposing the initial result (73%; 95% CI, 55% to 96%). Such dynamic patterns in bias are difficult to capture by conventional methods, where typically simple publication bias is assumed to operate. Our approach captures and corrects for complex dynamic patterns of bias, and thereby helps generating conclusions from published results that are more robust against the presence of different coexisting types of selective reporting.

Project description:BackgroundClinical decisions are made based on Cochrane reviews, but the implementation of results of evidence syntheses such as Cochrane reviews is problematic if the evidence is not prepared consistently. All systematic reviews should assess the risk of bias (RoB) in included studies, and in Cochrane reviews, this is done by using Cochrane RoB tool. However, the tool is not necessarily applied according to the instructions. In this study, we aimed to determine the types of bias and their corresponding judgements noted in the 'other bias' domain of Cochrane RoB tool.MethodsWe analyzed Cochrane reviews that included randomized controlled trials (RCTs) and extracted data regarding 'other bias' from the RoB table and accompanying support for the judgment. We categorized different types of other bias.ResultsWe analyzed 768 Cochrane reviews that included 11,369 RCTs. There were 602 (78%) Cochrane reviews that had 'other bias' domain in the RoB tool, and they included a total of 7811 RCTs. In the RoB table of 337 Cochrane reviews for at least one of the included trials it was indicated that no other bias was found and supporting explanations were inconsistently judged as low, unclear or high RoB. In the 524 Cochrane reviews that described various sources of other bias, there were 5762 individual types of explanations which we categorized into 31 groups. The judgments of the same supporting explanations were highly inconsistent. We found numerous other inconsistencies in reporting of sources of other bias in Cochrane reviews.ConclusionCochrane authors mention a wide range of sources of other bias in the RoB tool and they inconsistently judge the same supporting explanations. Inconsistency in appraising risk of other bias hinders reliability and comparability of Cochrane systematic reviews. Discrepant and erroneous judgments of bias in evidence synthesis may hinder implementation of evidence in routine clinical practice and reduce confidence in otherwise trustworthy sources of information. These results can help authors of Cochrane and non-Cochrane reviews to gain insight into various sources of other bias that can be found in trials, and also to help them avoid mistakes that were recognized in published Cochrane reviews.

Dataset Information

Managing the incidence of selective reporting bias: a survey of Cochrane review groups.

Publications

Managing the incidence of selective reporting bias: a survey of Cochrane review groups.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets