Dataset Information

Disagreements in risk of bias assessment for randomised controlled trials included in more than one Cochrane systematic reviews: a research on research study using cross-sectional design.

ABSTRACT: OBJECTIVES:Assess the frequency and reasons for disagreements in risk of bias assessments for randomised controlled trials (RCTs) included in more than one Cochrane review. DESIGN:Research on research study, using cross-sectional design. DATA SOURCES:2796 Cochrane reviews published between March 2011 and September 2014. DATA SELECTION:RCTs included in more than one review. DATA EXTRACTION:Risk of bias assessment and support for judgement for five key risk of bias items. DATA SYNTHESIS:For each item, we compared risk of bias assessment made in each review and calculated proportion of agreement. Two reviewers independently analysed 50% of all disagreements by comparing support for each judgement with information from study report to evaluate whether disagreements were related to a difference in information (eg, contact the study author) or a difference in interpretation (same support for judgement but different interpretation). They also identified main reasons for different interpretation. RESULTS:1604 RCTs were included in more than one review. Proportion of agreement ranged from 57% (770/1348 trials) for incomplete outcome data to 81% for random sequence generation (1193/1466). Most common source of disagreement was difference in interpretation of the same information, ranging from 65% (88/136) for random sequence generation to 90% (56/62) for blinding of participants and personnel. Access to different information explained 32/136 (24%) disagreements for random sequence generation and 38/205 (19%) for allocation concealment. Disagreements related to difference in interpretation were frequently related to incomplete or unclear reporting in the study report (83% of disagreements related to different interpretation for random sequence generation). CONCLUSIONS:Risk of bias judgements of RCTs included in more than one Cochrane review differed substantially. Most disagreements were related to a difference in interpretation of an incomplete or unclear description in the study report. A clearer guidance on common causes of incomplete information may improve agreement.

SUBMITTER: Bertizzolo L

PROVIDER: S-EPMC6500379 | biostudies-literature | 2019 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Disagreements in risk of bias assessment for randomised controlled trials included in more than one Cochrane systematic reviews: a research on research study using cross-sectional design.

Bertizzolo Lorenzo L Bossuyt Patrick P Atal Ignacio I Ravaud Philippe P Dechartres Agnes A

BMJ open 20190401 4

<h4>Objectives</h4>Assess the frequency and reasons for disagreements in risk of bias assessments for randomised controlled trials (RCTs) included in more than one Cochrane review.<h4>Design</h4>Research on research study, using cross-sectional design.<h4>Data sources</h4>2796 Cochrane reviews published between March 2011 and September 2014.<h4>Data selection</h4>RCTs included in more than one review.<h4>Data extraction</h4>Risk of bias assessment and support for judgement for five key risk of b ...[more]

PMID: 30940766

Similar Datasets

Project description:BackgroundThe inter-reviewer reliability of the risk of bias (RoB) assessment lacked agreement in previous studies. It is important to analyse these disagreements to improve the repeatability of RoB assessment. The objective of the study was to evaluate the frequency and reasons for disagreements in RoB assessments for randomised controlled trials (RCTs) that were included in multiple Cochrane reviews in the field of hypertension.MethodsA cross-sectional study was employed. We retrieved any RCTs that had been included in multiple Cochrane reviews in the field of hypertension from ARCHIE. The results of the RoB assessments were extracted, and the distributions of agreements and possible reasons for disagreement were analyzed.ResultsTwenty-six Cochrane reviews were included in this study. A total of 78 RCTs appeared in more than one Cochrane review. The level of agreement ranged from domain to domain. "Blinding of outcome assessment" showed a reasonably high level of agreement (94.9%), while "incomplete outcome data", "selective outcome reporting" and "other sources of bias" showed moderate levels of agreement (74.6%, 79.2% and 75.6%, respectively). However, the domains of "allocation concealment", "random sequence generation" and "blinding of participants and personnel" showed low levels of agreement (24.4%, 23.5%, and 47.4%, respectively). In the domains of "allocation concealment" and "blinding of participants and personnel", the agreement group had higher proportion of publication year ≤ 1996 than the disagreement group (P = 0.008 and P < 0.001, respectively). In the "blinding of participants and personnel", the impact factor was higher in the agreement group (P < 0.001). By analyzing the support text, we found that the most likely reason for disagreement was extracting different information from the same RCT.ConclusionFor Cochrane reviews in the field of hypertension using the 2011 version of the RoB tool, there was a large disagreement in the RoB assessment. It is suggested that the results of RoB assessments in systematic reviews that used the 2011 version of the RoB tool need to be interpreted with caution. More accurate information from RCTs needs to be collected when we synthesize clinical evidence.

Project description:IntroductionThe validity of systematic reviews and meta-analysis depends on methodological quality and unbiased dissemination of trials. Our objective is to evaluate the association of estimates of treatment effects with different bias-related study characteristics in meta-analyses of interventions used for treating pain in osteoarthritis (OA). From the findings, we hope to consolidate guidance on interpreting OA trials in systematic reviews based on empirical evidence from Cochrane reviews.Methods and analysisOnly systematic reviews that compare experimental interventions with sham, placebo or no intervention control will be considered eligible. Bias will be assessed with the risk of bias tool, used according to the Cochrane Collaboration's recommendations. Furthermore, center status, trial size and funding will be assessed. The primary outcome (pain) will be abstracted from the first appearing forest plot for overall pain in the Cochrane review. Treatment effect sizes will be expressed as standardised mean differences (SMDs), where the difference in mean values available from the forest plots is divided by the pooled SD. To empirically assess the risk of bias in treatment benefits, we will perform stratified analyses of the trials from the included meta-analyses and assess the interaction between trial characteristics and treatment effect. A relevant study-level covariate is defined as one that decreases the between-study variance (τ(2), estimated as Tau-squared) as a consequence of inclusion in the mixed effects statistical model.Ethics and disseminationMeta-analyses and randomised controlled trials provide the most reliable basis for treatment of patients with OA, but the actual impact of bias is unclear. This study will systematically examine the methodological quality in OA Cochrane reviews and explore the effect estimates behind possible bias. Since our study does not collect primary data, no formal ethical assessment and informed consent are required.Trial registration numberPROSPERO (CRD42013006924).

Project description:BackgroundClinical decisions are made based on Cochrane reviews, but the implementation of results of evidence syntheses such as Cochrane reviews is problematic if the evidence is not prepared consistently. All systematic reviews should assess the risk of bias (RoB) in included studies, and in Cochrane reviews, this is done by using Cochrane RoB tool. However, the tool is not necessarily applied according to the instructions. In this study, we aimed to determine the types of bias and their corresponding judgements noted in the 'other bias' domain of Cochrane RoB tool.MethodsWe analyzed Cochrane reviews that included randomized controlled trials (RCTs) and extracted data regarding 'other bias' from the RoB table and accompanying support for the judgment. We categorized different types of other bias.ResultsWe analyzed 768 Cochrane reviews that included 11,369 RCTs. There were 602 (78%) Cochrane reviews that had 'other bias' domain in the RoB tool, and they included a total of 7811 RCTs. In the RoB table of 337 Cochrane reviews for at least one of the included trials it was indicated that no other bias was found and supporting explanations were inconsistently judged as low, unclear or high RoB. In the 524 Cochrane reviews that described various sources of other bias, there were 5762 individual types of explanations which we categorized into 31 groups. The judgments of the same supporting explanations were highly inconsistent. We found numerous other inconsistencies in reporting of sources of other bias in Cochrane reviews.ConclusionCochrane authors mention a wide range of sources of other bias in the RoB tool and they inconsistently judge the same supporting explanations. Inconsistency in appraising risk of other bias hinders reliability and comparability of Cochrane systematic reviews. Discrepant and erroneous judgments of bias in evidence synthesis may hinder implementation of evidence in routine clinical practice and reduce confidence in otherwise trustworthy sources of information. These results can help authors of Cochrane and non-Cochrane reviews to gain insight into various sources of other bias that can be found in trials, and also to help them avoid mistakes that were recognized in published Cochrane reviews.

Project description:ObjectiveSystematic reviews can include cluster-randomised controlled trials (C-RCTs), which require different analysis compared with standard individual-randomised controlled trials. However, it is not known whether review authors follow the methodological and reporting guidance when including these trials. The aim of this study was to assess the methodological and reporting practice of Cochrane reviews that included C-RCTs against criteria developed from existing guidance.MethodsCriteria were developed, based on methodological literature and personal experience supervising review production and quality. Criteria were grouped into four themes: identifying, reporting, assessing risk of bias, and analysing C-RCTs. The Cochrane Database of Systematic Reviews was searched (2nd December 2013), and the 50 most recent reviews that included C-RCTs were retrieved. Each review was then assessed using the criteria.ResultsThe 50 reviews we identified were published by 26 Cochrane Review Groups between June 2013 and November 2013. For identifying C-RCTs, only 56% identified that C-RCTs were eligible for inclusion in the review in the eligibility criteria. For reporting C-RCTs, only eight (24%) of the 33 reviews reported the method of cluster adjustment for their included C-RCTs. For assessing risk of bias, only one review assessed all five C-RCT-specific risk-of-bias criteria. For analysing C-RCTs, of the 27 reviews that presented unadjusted data, only nine (33%) provided a warning that confidence intervals may be artificially narrow. Of the 34 reviews that reported data from unadjusted C-RCTs, only 13 (38%) excluded the unadjusted results from the meta-analyses.ConclusionsThe methodological and reporting practices in Cochrane reviews incorporating C-RCTs could be greatly improved, particularly with regard to analyses. Criteria developed as part of the current study could be used by review authors or editors to identify errors and improve the quality of published systematic reviews incorporating C-RCTs.

Project description:The Assessment of SpondyloArthritis international Society (ASAS) has defined core sets for (i) symptom-modifying anti-rheumatic drugs (SM-ARD), (ii) clinical record keeping, and (iii) disease-controlling anti-rheumatic therapy (DC-ART). These include the following domains for all three core sets: "physical function," "pain," "spinal mobility," "spinal stiffness," and "patient's global assessment" (PGA). The core set for clinical record keeping further includes the domains "peripheral joints/entheses" and "acute phase reactants," and the core set for DC-ART further includes the domains "fatigue" and "spine radiographs/hip radiographs." The Outcome Measures in Rheumatology (OMERACT) endorsed the core sets in 1998.Using empirical evidence from axSpA trials, we investigated the efficacy (i.e., net benefit) according to the ASAS/OMERACT core outcome set for axSpA across all interventions tested in trials included in subsequent Cochrane reviews. For all continuous scales, we combined data using the standardized mean difference (SMD) to meta-analyze outcomes involving the same domains. Also, through meta-regression analysis, we examined the effect of the separate SMD measures (independent variables) on the primary endpoint (log [OR], dependent variable) across all trials.Based on 11 eligible Cochrane reviews, from these, 85 articles were screened; we included 43 trials with 63 randomized comparisons. Mean (SD) number of ASAS/OMERACT core outcome domains measured for SM-ARD/physical therapy trials was 4.2 (1.7). Six trials assessed all proposed domains. Mean (SD) for number of core outcome domains for DC-ART trials was 5.8 (1.7). No trials assessed all nine domains. Eight trials (16%) were judged to have inadequate (i.e., high risk of) selective outcome reporting bias. The most responsible core domains for achieving success in meeting the primary objective per trial were pain, OR (95% CI) 5.19 (2.28, 11.77), and PGA, OR (95% CI) 1.87 (1.14, 3.07). In conclusion, selective outcome reporting (and "missing data") should be reduced by encouraging the use of the endorsed ASAS/OMERACT outcome domains in clinical trials. Overall outcome reporting was good for SM-ARD/physical therapy trials and poor for DC-ART trials. Our findings suggest that both PGA and pain provide a valuable holistic construct for the assessment of improvement beyond more objective measures of spinal inflammation.

Project description:ObjectiveTo examine the association between clinical trial registration and risk of bias in clinical trials that have been included in systematic reviews. As a secondary objective, we evaluated the risk of bias among trials registered prospectively vs. retrospectively.MethodClinical trials published in 2005 or after included in a sample of 100 Cochrane systematic reviews published from 2014-2019.ResultsOf 1,177 clinical trials identified, we verified 368 (31%) had been registered, of which 135 (36.7%) were registered prospectively (i.e., before or up to 1 month after enrollment of the first participant). Across the bias domains (one bias assessment for each domain per trial), the percentage of trials at low risk ranged from 29% to 58%; unclear risk ranged from to 26% to 61% and high risk ranged from 2% to 38%. Trials that had been registered had less high or unclear risk of bias in five domains: random sequence generation (univariate risk ratio [RR] 0.69, 95% confidence interval [95% CI] 0.58-0.81), allocation concealment (RR 0.64, 95% CI 0.57-0.72), performance bias (RR 0.65, 95% CI 0.58-0.72), detection bias (RR 0.70, 95% CI 0.62-0.78), and reporting bias (RR 0.62, 95% CI 0.53-0.73). An association between clinical trial registration and high or unclear risk of attrition bias could not be demonstrated nor refuted (RR 1.02, 95% CI 0.89-1.17). It also was observed in terms of overall risk of bias, that registered trials had less high or unclear overall risk of bias than trials that had not been registered (univariate RR 0.29, 95% CI 0.19-0.46). Prospective clinical trial registration was associated with low risks of selection bias due to inadequate allocation concealment, performance bias, and detection bias compared with retrospective clinical trial registration.ConclusionIn a large sample of clinical trials included in recently published systematic reviews of interventions, clinical trial registration was associated with low risk of bias for five of the six domains examined.

Project description:BackgroundBias in randomized controlled trials (RCTs) can lead to underestimation or overestimation of the true effects of interventions. Surgical RCTs may suffer from the risk of bias (RoB) that is avoidable in trials of other interventions, and vice versa. We aimed to compare the adequacy of RoB assessments in surgical versus non-surgical RCTs included in Cochrane reviews and to assess the most common differences in those RoB assessments. Due to specificities of surgical trials, i.e. difficulties associated with blinding of surgical interventions, we hypothesized that assessments of surgical trials may be more adequate, compared to RCTs of non-surgical interventions.MethodsThis was a methodological study, analyzing methods of published Cochrane systematic reviews. Data were extracted from RoB tables in Cochrane reviews (judgments and accompanying explanatory comment) for the following four RoB domains used in the 2011 Cochrane RoB tool: randomization, allocation concealment, blinding of participants and personnel, and blinding of outcome assessors. We defined adequate assessments as those that were in line with instructions from the Cochrane Handbook for Systematic Reviews of Interventions. The prevalence of adequate assessments was compared in surgical versus non-surgical trials. The most common differences in both groups of reviews were presented.ResultsIn 729 analyzed Cochrane reviews, there were 10,537 included trials. The prevalence of adequate RoB judgments made by Cochrane authors ranged from 87.9, 95%CI (87.3 to 88.6%) for randomization to 70.7, 95%CI (69.8 to 71.5%) for blinding of participants and personnel. For all analyzed RoB domains, the prevalence of adequate RoB domains was higher in surgical trials than in non-surgical trials. For two RoB domains assessing blinding, this difference between surgical and non-surgical trials was statistically significant (P < 0.001), while the difference was not significant for the RoB domain regarding randomization (P = 0.124) and allocation concealment (P = 0.039, β < 0.8).ConclusionsRoB judgments were more in line with instructions from the Cochrane Handbook when Cochrane reviews assessed surgical trials, compared to those that analyzed non-surgical interventions. However, further steps are warranted to scrutinize RoB assessment in trials of both surgical and non-surgical interventions.

Dataset Information

Disagreements in risk of bias assessment for randomised controlled trials included in more than one Cochrane systematic reviews: a research on research study using cross-sectional design.

Publications

Disagreements in risk of bias assessment for randomised controlled trials included in more than one Cochrane systematic reviews: a research on research study using cross-sectional design.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets