Adequacy of risk of bias assessment in surgical vs non-surgical trials in Cochrane reviews: a methodological study
Ontology highlight
ABSTRACT:
SUBMITTER: Barcot O
PROVIDER: S-EPMC7526117 | biostudies-literature | 2020 Jan
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:ObjectiveSystematic reviews can include cluster-randomised controlled trials (C-RCTs), which require different analysis compared with standard individual-randomised controlled trials. However, it is not known whether review authors follow the methodological and reporting guidance when including these trials. The aim of this study was to assess the methodological and reporting practice of Cochrane reviews that included C-RCTs against criteria developed from existing guidance.MethodsCriteria were developed, based on methodological literature and personal experience supervising review production and quality. Criteria were grouped into four themes: identifying, reporting, assessing risk of bias, and analysing C-RCTs. The Cochrane Database of Systematic Reviews was searched (2nd December 2013), and the 50 most recent reviews that included C-RCTs were retrieved. Each review was then assessed using the criteria.ResultsThe 50 reviews we identified were published by 26 Cochrane Review Groups between June 2013 and November 2013. For identifying C-RCTs, only 56% identified that C-RCTs were eligible for inclusion in the review in the eligibility criteria. For reporting C-RCTs, only eight (24%) of the 33 reviews reported the method of cluster adjustment for their included C-RCTs. For assessing risk of bias, only one review assessed all five C-RCT-specific risk-of-bias criteria. For analysing C-RCTs, of the 27 reviews that presented unadjusted data, only nine (33%) provided a warning that confidence intervals may be artificially narrow. Of the 34 reviews that reported data from unadjusted C-RCTs, only 13 (38%) excluded the unadjusted results from the meta-analyses.ConclusionsThe methodological and reporting practices in Cochrane reviews incorporating C-RCTs could be greatly improved, particularly with regard to analyses. Criteria developed as part of the current study could be used by review authors or editors to identify errors and improve the quality of published systematic reviews incorporating C-RCTs.
| S-EPMC4794236 | biostudies-literature
Project description:IntroductionThe validity of systematic reviews and meta-analysis depends on methodological quality and unbiased dissemination of trials. Our objective is to evaluate the association of estimates of treatment effects with different bias-related study characteristics in meta-analyses of interventions used for treating pain in osteoarthritis (OA). From the findings, we hope to consolidate guidance on interpreting OA trials in systematic reviews based on empirical evidence from Cochrane reviews.Methods and analysisOnly systematic reviews that compare experimental interventions with sham, placebo or no intervention control will be considered eligible. Bias will be assessed with the risk of bias tool, used according to the Cochrane Collaboration's recommendations. Furthermore, center status, trial size and funding will be assessed. The primary outcome (pain) will be abstracted from the first appearing forest plot for overall pain in the Cochrane review. Treatment effect sizes will be expressed as standardised mean differences (SMDs), where the difference in mean values available from the forest plots is divided by the pooled SD. To empirically assess the risk of bias in treatment benefits, we will perform stratified analyses of the trials from the included meta-analyses and assess the interaction between trial characteristics and treatment effect. A relevant study-level covariate is defined as one that decreases the between-study variance (τ(2), estimated as Tau-squared) as a consequence of inclusion in the mixed effects statistical model.Ethics and disseminationMeta-analyses and randomised controlled trials provide the most reliable basis for treatment of patients with OA, but the actual impact of bias is unclear. This study will systematically examine the methodological quality in OA Cochrane reviews and explore the effect estimates behind possible bias. Since our study does not collect primary data, no formal ethical assessment and informed consent are required.Trial registration numberPROSPERO (CRD42013006924).
| S-EPMC4187994 | biostudies-literature
Project description:BackgroundTo ensure evidence-based decision-making in pediatric oral health, Cochrane systematic reviews that address topics pertinent to this field are necessary. We aimed to identify all systematic reviews of paediatric dentistry and oral health by the Cochrane Oral Health Group (COHG), summarize their characteristics and assess their methodological quality. Our second objective was to assess implications for practice in the review conclusions and provide an overview of clinical implications about the usefulness of paediatric oral health interventions in practice.MethodsWe conducted a methodological survey including all paediatric dentistry reviews from the COHG. We extracted data on characteristics of included reviews, then assessed the methodological quality using a validated 11-item quality assessment tool (AMSTAR). Finally, we coded each review to indicate whether its authors concluded that an intervention should be implemented in practice, was not supported or was refuted by the evidence, or should be used only in research (inconclusive evidence).ResultsWe selected 37 reviews; most concerned the prevention of caries. The methodological quality was high, except for the assessment of reporting bias. In 7 reviews (19%), the research showed that benefits outweighed harms; in 1, the experimental intervention was found ineffective; and in 29 (78%), evidence was insufficient to assess benefits and harms. In the 7 reviews, topical fluoride treatments (with toothpaste, gel or varnish) were found effective for permanent and deciduous teeth in children and adolescents, and sealants for occlusal tooth surfaces of permanent molars.ConclusionsCochrane reviews of paediatric dentistry were of high quality. They provided strong evidence that topical fluoride treatments and sealants are effective for children and adolescents and thus should be implemented in practice. However, a substantial number of reviews yielded inconclusive evidence.
| S-EPMC4108002 | biostudies-literature
Project description:BACKGROUND: Temporomandibular joint disorders (TMJD) are multifactor, complex clinical problems affecting approximately 60-70% of the general population, with considerable controversy about the most effective treatment. For example, reports claim success rates of 70% and 83% for non-surgical and surgical treatment, whereas other reports claim success rates of 40% to 70% for self-improvement without treatment. Therefore, the purpose of this study was to (1) identify systematic reviews comparing temporomandibular joint disorder surgical and non-surgical treatment, (2) evaluate their methodological quality, and (3) evaluate the evidence grade within the systematic reviews. METHODS: A search strategy was developed and implemented for MEDLINE, Cochrane Library, LILACS, and Brazilian Dentistry Bibliography databases. Inclusion criteria were: systematic reviews (+/- meta-analysis) comparing surgical and non-surgical TMJD treatment, published in English, Spanish, Portuguese, Italian, or German between the years 1966 and 2007(up to July). Exclusion criteria were: in vitro or animal studies; narrative reviews or editorials or editorial letters; and articles published in other languages. Two investigators independently selected and evaluated systematic reviews. Three different instruments (AMSTAR, OQAQ and CASP) were used to evaluate methodological quality, and the results averaged. The GRADE instrument was used to evaluate the evidence grade within the reviews. RESULTS: The search strategy identified 211 reports; of which 2 were systematic reviews meeting inclusion criteria. The first review met 23.5 +/- 6.0% and the second met 77.5 +/- 12.8% of the methodological quality criteria (mean +/- sd). In these systematic reviews between 9 and 15% of the trials were graded as high quality, and 2 and 8% of the total number of patients were involved in these studies. CONCLUSION: The results indicate that in spite of the widespread impact of TMJD, and the multitude of potential interventions, clinicians have expended sparse attention to systematically implementing clinical trial methodology that would improve validity and reliability of outcome measures. With some 20 years of knowledge of evidence-based healthcare, the meager attention to these issues begins to raise ethical issues about TMJD trial conduct and clinical care.
| S-EPMC2576167 | biostudies-literature
Project description:ObjectiveOsteoarthritis generates substantial health and socioeconomic burden, which is particularly marked in marginalized groups. It is imperative that practitioners have ready access to summaries of evidence-based interventions for osteoarthritis that incorporate equity considerations. Summaries of systematic reviews can provide this. The present study surveyed experts to prioritize a selection ofinterventions, from which equity focused summaries will be generated. Specifically, the prioritized interventions will be developed into Cochrane Evidence4Equity (E4E) summaries.MethodsTwenty-seven systematic reviews of OA interventions were found. From these, twenty-nine non-surgical treatments for osteoarthritis were identified, based on statistically significant findings for desired outcome variables or adverse events. Key findings from these studies were summarised and provided to 9 experts in the field of osteoarthritis.. Expert participants were asked to rate interventions based on feasibility, health system effects, universality, impact on inequities, and priority for translation into equity based E4E summaries. Expert participants were also encouraged to make comments to provide context for each rating. Free text responses were coded inductively and grouped into subthemes and themes.ResultsExpert participants rated the intervention home land-based exercise for knee OA highest for priority for translation into an E4E summaries, followed by the interventions individual land-based exercise for knee OA, class land-based exercise for knee OA, exercise for hand OA and land-based exercise for hip OA. Upon qualitative analysis of the expert participants' comments, fifteen subthemes were identified and grouped into three overall themes: (1) this intervention or an aspect of this intervention is unnecessary or unsafe; (2) this intervention or an aspect of this intervention may increase health inequities; and (3) experts noted difficulties completing rating exercise.ConclusionThe list of priority interventions and corresponding expert commentary generated information that will be used to direct and support knowledge translation efforts.
| S-EPMC8193871 | biostudies-literature
Project description:BackgroundDiscrepancies in outcome reporting (DOR) between protocol and published studies include inclusions of new outcomes, omission of prespecified outcomes, upgrade and downgrade of secondary and primary outcomes, and changes in definitions of prespecified outcomes. DOR can result in outcome reporting bias (ORB) when changes in outcomes occur after knowledge of results. This has potential to overestimate treatment effects and underestimate harms. This can also occur at the level of systematic reviews when changes in outcomes occur after knowledge of results of included studies. The prevalence of DOR and ORB in systematic reviews is unknown in systematic reviews published post-2007.ObjectiveTo estimate the prevalence of DOR and risk of ORB in all Cochrane reviews between the years 2007 and 2014.MethodsA stratified random sampling approach was applied to collect a representative sample of Cochrane systematic reviews from each Cochrane review group. DOR was assessed by matching outcomes in each systematic review with their respective protocol. When DOR occurred, reviews were further assessed if there was a risk of ORB (unclear, low or high risk). We classified DOR as a high risk for ORB if the discrepancy occurred after knowledge of results in the systematic review.Results150 of 350 (43%) review and protocol pairings contained DOR. When reviews were further scrutinised, 23% (35 of 150) of reviews with DOR contained a high risk of ORB, with changes being made after knowledge of results from individual trials.ConclusionsIn our study, we identified just under a half of Cochrane reviews with at least one DOR. Of these, a fifth were at high risk of ORB. The presence of DOR and ORB in Cochrane reviews is of great concern; however, a solution is relatively simple. Authors are encouraged to be transparent where outcomes change and to describe the legitimacy of changing outcomes in order to prevent suspicion of bias.
| S-EPMC7076244 | biostudies-literature
Project description:Flaws in the design of randomized trials may bias intervention effect estimates and increase between-trial heterogeneity. Empirical evidence suggests that these problems are greatest for subjectively assessed outcomes. For the Risk of Bias in Evidence Synthesis (ROBES) Study, we extracted risk-of-bias judgements (for sequence generation, allocation concealment, blinding, and incomplete data) from a large collection of meta-analyses published in the Cochrane Library (issue 4; April 2011). We categorized outcome measures as mortality, other objective outcome, or subjective outcome, and we estimated associations of bias judgements with intervention effect estimates using Bayesian hierarchical models. Among 2,443 randomized trials in 228 meta-analyses, intervention effect estimates were, on average, exaggerated in trials with high or unclear (versus low) risk-of-bias judgements for sequence generation (ratio of odds ratios (ROR) = 0.91, 95% credible interval (CrI): 0.86, 0.98), allocation concealment (ROR = 0.92, 95% CrI: 0.86, 0.98), and blinding (ROR = 0.87, 95% CrI: 0.80, 0.93). In contrast to previous work, we did not observe consistently different bias for subjective outcomes compared with mortality. However, we found an increase in between-trial heterogeneity associated with lack of blinding in meta-analyses with subjective outcomes. Inconsistency in criteria for risk-of-bias judgements applied by individual reviewers is a likely limitation of routinely collected bias assessments. Inadequate randomization and lack of blinding may lead to exaggeration of intervention effect estimates in randomized trials.
| S-EPMC5928453 | biostudies-other
Project description:OBJECTIVE:To assess the quality of Cochrane reviews. DESIGN:Ten methodologists affiliated with the Cochrane Collaboration independently examined, in a semistructured way, the quality of reviews first published in 1998. Each review was assessed by two people; if one of them noted any major problems, they agreed on a common assessment. Predominant types of problem were categorised. SETTING:Cyberspace collaboration coordinated from the Nordic Cochrane Centre. STUDIES:All 53 reviews first published in issue 4 of the Cochrane Library in 1998. MAIN OUTCOME MEASURE:Proportion of reviews with various types of major problem. RESULTS:No problems or only minor ones were found in most reviews. Major problems were identified in 15 reviews (29%). The evidence did not fully support the conclusion in nine reviews (17%), the conduct or reporting was unsatisfactory in 12 reviews (23%), and stylistic problems were identified in 12 reviews (23%). The problematic conclusions all gave too favourable a picture of the experimental intervention. CONCLUSIONS:Cochrane reviews have previously been shown to be of higher quality and less biased on average than other systematic reviews, but improvement is always possible. The Cochrane Collaboration has taken steps to improve editorial processes and the quality of its reviews. Meanwhile, the Cochrane Library remains a key source of evidence about the effects of healthcare interventions. Its users should interpret reviews cautiously, particularly those with conclusions favouring experimental interventions and those with many typographical errors.
| S-EPMC57800 | biostudies-literature
Project description:BackgroundSystematic reviews with meta-analyses often contain many statistical tests. This multiplicity may increase the risk of type I error. Few attempts have been made to address the problem of statistical multiplicity in systematic reviews. Before the implications are properly considered, the size of the issue deserves clarification. Because of the emphasis on bias evaluation and because of the editorial processes involved, Cochrane reviews may contain more multiplicity than their non-Cochrane counterparts. This study measured the quantity of statistical multiplicity present in a population of systematic reviews and aimed to assess whether this quantity is different in Cochrane and non-Cochrane reviews.Methods/principal findingsWe selected all the systematic reviews published by the Cochrane Anaesthesia Review Group containing a meta-analysis and matched them with comparable non-Cochrane reviews. We counted the number of statistical tests done in each systematic review. The median number of tests overall was 10 (interquartile range (IQR) 6 to 18). The median was 12 in Cochrane and 8 in non-Cochrane reviews (difference in medians 4 (95% confidence interval (CI) 2.0-19.0). The proportion that used an assessment of risk of bias as a reason for doing extra analyses was 42% in Cochrane and 28% in non-Cochrane reviews (difference in proportions 14% (95% CI -8 to 36). The issue of multiplicity was addressed in 6% of all the reviews.Conclusion/significanceStatistical multiplicity in systematic reviews requires attention. We found more multiplicity in Cochrane reviews than in non-Cochrane reviews. Many of the reasons for the increase in multiplicity may well represent improved methodological approaches and greater transparency, but multiplicity may also cause an increased risk of spurious conclusions. Few systematic reviews, whether Cochrane or non-Cochrane, address the issue of multiplicity.
| S-EPMC3229598 | biostudies-literature
Project description:IntroductionPrevalence of non-communicable diseases (NCDs) is increasing globally, with the greatest projected increases in low-income and middle-income countries. We sought to quantify the proportion of Cochrane evidence relating to NCDs derived from such countries.MethodsWe searched the Cochrane database of systematic reviews for reviews relating to NCDs highlighted in the WHO NCD action plan (cardiovascular, cancers, diabetes and chronic respiratory diseases). We excluded reviews at the protocol stage and those that were repeated or had been withdrawn. For each review, two independent researchers extracted data relating to the country of the corresponding author and the number of trials and participants from countries, using the World Bank classification of gross national income per capita.Results797 reviews were analysed, with a reported total number of 12 340 trials and 10 937 306 participants. Of the corresponding authors 90% were from high-income countries (41% from the UK). Of the 746 reviews in which at least one trial had met the inclusion criteria, only 55% provided a summary of the country of included trials. Analysis of the 633 reviews in which country of trials could be established revealed that almost 90% of trials and over 80% of participants were from high-income countries. 438 (5%) trials including 1 145 013 (11.7%) participants were undertaken in low-middle income countries. We found that only 13 (0.15%) trials with 982 (0.01%) participants were undertaken in low-income countries. Other than the five Cochrane NCD corresponding authors from South Africa, only one other corresponding author was from Africa (Gambia).DiscussionThe overwhelming body of evidence for NCDs pertains to high-income countries, with only a small number of review authors based in low-income settings. As a consequence, there is an urgent need for research infrastructure and funding for the undertaking of high-quality trials in this area.
| S-EPMC3703573 | biostudies-literature