Dataset Information

Systematic Differences between Cochrane and Non-Cochrane Meta-Analyses on the Same Topic: A Matched Pair Analysis.

ABSTRACT: Meta-analyses conducted via the Cochrane Collaboration adhere to strict methodological and reporting standards aiming to minimize bias, maximize transparency/reproducibility, and improve the accuracy of summarized data. Whether this results in differences in the results reported by meta-analyses on the same topic conducted outside the Cochrane Collaboration is an open question.We conducted a matched-pair analysis with individual meta-analyses as the unit of analysis, comparing Cochrane and non-Cochrane reviews. Using meta-analyses from the cardiovascular literature, we identified pairs that matched on intervention and outcome. The pairs were contrasted in terms of how frequently results disagreed between the Cochrane and non-Cochrane reviews, whether effect sizes and statistical precision differed systematically, and how these differences related to the frequency of secondary citations of those reviews.Our search yielded 40 matched pairs of reviews. The two sets were similar in terms of which was first to publication, how many studies were included, and average sample sizes. The paired reviews included a total of 344 individual clinical trials: 111 (32.3%) studies were included only in a Cochrane review, 104 (30.2%) only in a non-Cochrane review, and 129 (37.5%) in both. Stated another way, 62.5% of studies were only included in one or the other meta-analytic literature. Overall, 37.5% of pairs had discrepant results. The most common involved shifts in the width of 95% confidence intervals that would yield a different statistical interpretation of the significance of results (7 pairs). Additionally, 20% differed in the direction of the summary effect size (5 pairs) or reported greater than a 2-fold difference in its magnitude (3 pairs). Non-Cochrane reviews reported significantly higher effect sizes (P< 0.001) and lower precision (P<0.001) than matched Cochrane reviews. Reviews reporting an effect size at least 2-fold greater than their matched pair were cited more frequently.Though results between topic-matched Cochrane and non-Cochrane reviews were quite similar, discrepant results were frequent, and the overlap of included studies was surprisingly low. Non-Cochrane reviews report larger effect sizes with lower precision than Cochrane reviews, indicating systematic differences, likely reflective of methodology, between the two types of reviews that could generate different interpretations of the interventions under question.

SUBMITTER: Useem J

PROVIDER: S-EPMC4686011 | biostudies-literature | 2015

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Systematic Differences between Cochrane and Non-Cochrane Meta-Analyses on the Same Topic: A Matched Pair Analysis.

Useem Johanna J Brennan Alana A LaValley Michael M Vickery Michelle M Ameli Omid O Reinen Nichole N Gill Christopher J CJ

PloS one 20151215 12

<h4>Background</h4>Meta-analyses conducted via the Cochrane Collaboration adhere to strict methodological and reporting standards aiming to minimize bias, maximize transparency/reproducibility, and improve the accuracy of summarized data. Whether this results in differences in the results reported by meta-analyses on the same topic conducted outside the Cochrane Collaboration is an open question.<h4>Methods</h4>We conducted a matched-pair analysis with individual meta-analyses as the unit of ana ...[more]

PMID: 26671213

Similar Datasets

Project description:Objective To evaluate the frequency, validity, and relevance of statistically significant (P<0.05) sex-treatment interactions in randomized controlled trials in Cochrane meta-analyses.Design Meta-epidemiological study.Data sources Cochrane Database of Systematic Reviews (CDSR) and PubMed.Eligibility criteria for study selection Reviews published in the CDSR with sex-treatment subgroup analyses in the forest plots, using data from randomized controlled trials.Data extraction Information on the study design and sex subgroup data were extracted from reviews and forest plots that met inclusion criteria. For each statistically significant sex-treatment interaction, the potential for biological plausibility and clinical significance was considered.Results Among the 41 reviews with relevant data, there were 109 separate treatment-outcome analyses ("topics"). Among the 109 topics, eight (7%) had a statistically significant sex-treatment interaction. The 109 topics included 311 randomized controlled trials (162 with both sexes, 46 with males only, 103 with females only). Of the 162 individual randomized controlled trials that included both sexes, 15 (9%) had a statistically significant sex-treatment interaction. Of four topics where the first published randomized controlled trial had a statistically significant sex-treatment interaction, no meta-analyses that included other randomized controlled trials retained the statistical significance and no meta-analyses showed statistical significance when data from the first published randomized controlled trial were excluded. Of the eight statistically significant sex-treatment interactions from the overall analyses, only three were discussed by the CDSR reviewers for a potential impact on different clinical management for males compared with females. None of these topics had a sex-treatment interaction that influenced treatment recommendations in recent guidelines. UpToDate, an online physician-authored clinical decision support resource, suggested differential management of men and women for one of these sex-treatment interactions.Conclusion Statistically significant sex-treatment interactions are only slightly more frequent than what would be expected by chance and there is little evidence of subsequent corroboration or clinical relevance of sex-treatment interactions.

Project description:BACKGROUND:Information on subgroup assessments in systematic reviews (SR) of atrial fibrillation (AF) is limited. This review aims to describe subgroup analyses in AF SRs to inform the design of SRs and randomized trials as well as clinical practice. METHODS:We conducted a cross sectional meta-epidemiological study of Cochrane AF reviews by searching AF (including variants) in the title, abstract, or keyword field without date or language restrictions (Issue 9; September 2018). Two reviewers independently extracted study characteristics to summarize frequency of subgroups pre-specified and conducted and report credibility of subgroup effects claimed. RESULTS:Of 39 Cochrane reviews identified, 17 met inclusion criteria (including 168 reports of 127 randomized trials) and the majority (16; 94.1%) conducted meta-analysis of outcomes. Most (13; 76.5%) planned pre-specified subgroup analyses; 7 of which (41.2%) conducted subgroups. In these 7 reviews, 56 subgroups were planned, 17 (30.4%) conducted and 6 (10.7%) yielded subgroup effects. Variables such as co-morbid disease, stroke risk factors, prior stroke/transient ischemic attack, age, race, and sex represented 44% (24 subgroups) of all planned subgroups (8 conducted; 14.3%); however, information on covariate selection was lacking. Overall, more subgroups were planned than conducted (mean difference (95% CI) 2.3 (1.2-3.5, p?<?0.001)). Of all subgroups conducted, anticoagulant characteristics comprised a third of all subgroup effects (n?=?5, 35.7%). The credibility of subgroups identified (n?=?14) was assessed and less than half (43%) represented one of a small number of pre-specified hypothesis and rarely were effects seen within studies (7%). Of 5 reviews that reported subgroup effects, only 3 discussed subgroup effects as part of the overall conclusions; none discussed credibility of subgroup effects. CONCLUSIONS:This meta-epidemiological review of a subset of Cochrane AF reviews suggests that planning and reporting of subgroup analyses in AF reviews can be improved to better inform clinical management. Most pre-specified subgroup analyses were not performed, important variables (such as stroke, bleeding risk, and other comorbidities) were rarely examined and credibility of subgroup effects claimed was low. Future reviews should aim to identify important subgroups in their protocols and use recommended approaches to test subgroup effects in order to better support clinical decision-making.

Project description:ObjectiveTo (1) investigate the extent to which recently published meta-analyses report trial funding, author-industry financial ties and author-industry employment from included randomised controlled trials (RCTs), comparing Cochrane and non-Cochrane meta-analyses; (2) examine characteristics of meta-analyses independently associated with reporting funding sources of included RCTs; and (3) compare reporting among recently published Cochrane meta-analyses to Cochrane reviews published in 2010.DesignReview of consecutive sample of recently published meta-analyses.Data sourcesMEDLINE database via PubMed searched on 19 October 2018.Eligibility criteria for selecting articlesWe selected the 250 most recent meta-analyses listed in PubMed that included a documented search of at least one database, statistically combined results from ≥2 RCTs and evaluated the effects of a drug or class of drugs.Results90 of 107 (84%) Cochrane meta-analyses reported funding sources for some or all included trials compared with 21 of 143 (15%) non-Cochrane meta-analyses, a difference of 69% (95% CI 59% to 77%). Percent reporting was also higher for Cochrane meta-analyses compared with non-Cochrane meta-analyses for trial author-industry financial ties (44% versus 1%; 95% CI for difference 33% to 52%) and employment (17% versus 1%; 95% CI for difference 9% to 24%). In multivariable analysis, compared with Cochrane meta-analyses, the odds ratio (OR) for reporting trial funding was ≤0.11 for all other journal category and impact factor combinations. Compared with Cochrane reviews from 2010, reporting of funding sources of included RCTs among recently published Cochrane meta-analyses improved by 54% (95% CI 42% to 63%), and reporting of trial author-industry financial ties and employment improved by 37% (95% CI 26% to 47%) and 10% (95% CI 2% to 19%).ConclusionsReporting of trial funding sources, trial author-industry financial ties and trial author-industry employment in Cochrane meta-analyses has improved since 2010 and is higher than in non-Cochrane meta-analyses.

Project description:BACKGROUND:There is growing interest in evaluating differences in healthcare interventions across routinely collected demographic characteristics. However, individual subgroup analyses in randomized controlled trials are often not prespecified, adjusted for multiple testing, or conducted using the appropriate statistical test for interaction, and therefore frequently lack credibility. Meta-analyses can be used to examine the validity of potential subgroup differences by collating evidence across trials. Here, we characterize the conduct and clinical translation of age-treatment subgroup analyses in Cochrane reviews. METHODS:For a random sample of 928 Cochrane intervention reviews of randomized trials, we determined how often subgroup analyses of age are reported, how often these analyses have a P?<?0.05 from formal interaction testing, how frequently subgroup differences first observed in an individual trial are later corroborated by other trials in the same meta-analysis, and how often statistically significant results are included in commonly used clinical management resources (BMJ Best Practice, UpToDate, Cochrane Clinical Answers, Google Scholar, and Google search). RESULTS:Among 928 Cochrane intervention reviews, 189 (20.4%) included plans to conduct age-treatment subgroup analyses. The vast majority (162 of 189, 85.7%) of the planned analyses were not conducted, commonly because of insufficient trial data. There were 22 reviews that conducted their planned age-treatment subgroup analyses, and another 3 reviews appeared to perform unplanned age-treatment subgroup analyses. These 25 (25 of 928, 2.7%) reviews conducted a total of 97 age-treatment subgroup analyses, of which 65 analyses (in 20 reviews) had non-overlapping subgroup levels. Among the 65 age-treatment subgroup analyses, 14 (21.5%) did not report any formal interaction testing. Seven (10.8%) reported P?<?0.05 from formal age-treatment interaction testing; however, none of these seven analyses were in reviews that discussed the potential biological rationale or clinical significance of the subgroup findings or had results that were included in common clinical practice resources. CONCLUSION:Age-treatment subgroup analyses in Cochrane intervention reviews were frequently planned but rarely conducted, and implications of detected interactions were not discussed in the reviews or mentioned in common clinical resources. When subgroup analyses are performed, authors should report the findings, compare the results to previous studies, and outline any potential impact on clinical care.

Project description:BackgroundStudies have shown that industry-sponsored meta-analyses of drugs lack scientific rigour and have biased conclusions. However, these studies have been restricted to certain medical specialities. We compared all industry-supported meta-analyses of drug-drug comparisons with those without industry support.MethodsWe searched PubMed for all meta-analyses that compared different drugs or classes of drugs published in 2004. Two authors assessed the meta-analyses and independently extracted data. We used a validated scale for judging the methodological quality and a binary scale for judging conclusions. We divided the meta-analyses according to the type of support in 3 categories: industry-supported, non-profit support or no support, and undeclared support.ResultsWe included 39 meta-analyses. Ten had industry support, 18 non-profit or no support, and 11 undeclared support. On a 0-7 scale, the median quality score was 6 for meta-analyses with non-profit or no support and 2.5 for the industry-supported meta-analyses (P < 0.01). Compared with industry-supported meta-analyses, more meta-analyses with non-profit or no support avoided bias in the selection of studies (P = 0.01), more often stated the search methods used to find studies (P = 0.02), searched comprehensively (P < 0.01), reported criteria for assessing the validity of the studies (P = 0.02), used appropriate criteria (P = 0.04), described methods of allocation concealment (P = 0.05), described methods of blinding (P = 0.05), and described excluded patients (P = 0.08) and studies (P = 0.15). Forty percent of the industry-supported meta-analyses recommended the experimental drug without reservations, compared with 22% of the meta-analyses with non-profit or no support (P = 0.57).In a sensitivity analysis, we contacted the authors of the meta-analyses with undeclared support. Eight who replied that they had not received industry funding were added to those with non-profit or no support, and 3 who did not reply were added to those with industry support. This analysis did not change the results much.ConclusionTransparency is essential for readers to make their own judgment about medical interventions guided by the results of meta-analyses. We found that industry-supported meta-analyses are less transparent than meta-analyses with non-profit support or no support.

Dataset Information

Systematic Differences between Cochrane and Non-Cochrane Meta-Analyses on the Same Topic: A Matched Pair Analysis.

Publications

Systematic Differences between Cochrane and Non-Cochrane Meta-Analyses on the Same Topic: A Matched Pair Analysis.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets