Dataset Information

Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review.

ABSTRACT:

Objective

To compare the methodological quality and conclusions in Cochrane reviews with those in industry supported meta-analyses and other meta-analyses of the same drugs.

Design

Systematic review comparing pairs of meta-analyses that studied the same two drugs in the same disease and were published within two years of each other.

Data sources

Cochrane Database of Systematic Reviews (2003, issue 1), PubMed, and Embase.

Data extraction

Two observers independently extracted data and used a validated scale to judge the methodological quality of the reviews.

Results

175 of 1596 Cochrane reviews had a meta-analysis that compared two drugs. Twenty four meta-analyses that matched the Cochrane reviews were found: eight were industry supported, nine had undeclared support, and seven had no support or were supported by non-industry sources. On a 0-7 scale, the median quality score was 7 for Cochrane reviews and 3 for other reviews (P < 0.01). Compared with industry supported reviews and reviews with undeclared support, Cochrane reviews had more often considered the potential for bias in the review--for example, by describing the method of concealment of allocation and describing excluded patients or studies. The seven industry supported reviews that had conclusions recommended the experimental drug without reservations, compared with none of the Cochrane reviews (P = 0.02), although the estimated treatment effect was similar on average (z = 0.46, P = 0.64). Reviews with undeclared support and reviews with not for profit support or no support had conclusions that were similar in cautiousness to the Cochrane reviews.

Conclusions

Industry supported reviews of drugs should be read with caution as they were less transparent, had few reservations about methodological limitations of the included trials, and had more favourable conclusions than the corresponding Cochrane reviews.

SUBMITTER: Jorgensen AW

PROVIDER: S-EPMC1602036 | biostudies-literature | 2006 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review.

Jørgensen Anders W AW Hilden Jørgen J Gøtzsche Peter C PC

BMJ (Clinical research ed.) 20061006 7572

<h4>Objective</h4>To compare the methodological quality and conclusions in Cochrane reviews with those in industry supported meta-analyses and other meta-analyses of the same drugs.<h4>Design</h4>Systematic review comparing pairs of meta-analyses that studied the same two drugs in the same disease and were published within two years of each other.<h4>Data sources</h4>Cochrane Database of Systematic Reviews (2003, issue 1), PubMed, and Embase.<h4>Data extraction</h4>Two observers independently ex ...[more]

PMID: 17028106

Similar Datasets

Project description:Meta-analyses conducted via the Cochrane Collaboration adhere to strict methodological and reporting standards aiming to minimize bias, maximize transparency/reproducibility, and improve the accuracy of summarized data. Whether this results in differences in the results reported by meta-analyses on the same topic conducted outside the Cochrane Collaboration is an open question.We conducted a matched-pair analysis with individual meta-analyses as the unit of analysis, comparing Cochrane and non-Cochrane reviews. Using meta-analyses from the cardiovascular literature, we identified pairs that matched on intervention and outcome. The pairs were contrasted in terms of how frequently results disagreed between the Cochrane and non-Cochrane reviews, whether effect sizes and statistical precision differed systematically, and how these differences related to the frequency of secondary citations of those reviews.Our search yielded 40 matched pairs of reviews. The two sets were similar in terms of which was first to publication, how many studies were included, and average sample sizes. The paired reviews included a total of 344 individual clinical trials: 111 (32.3%) studies were included only in a Cochrane review, 104 (30.2%) only in a non-Cochrane review, and 129 (37.5%) in both. Stated another way, 62.5% of studies were only included in one or the other meta-analytic literature. Overall, 37.5% of pairs had discrepant results. The most common involved shifts in the width of 95% confidence intervals that would yield a different statistical interpretation of the significance of results (7 pairs). Additionally, 20% differed in the direction of the summary effect size (5 pairs) or reported greater than a 2-fold difference in its magnitude (3 pairs). Non-Cochrane reviews reported significantly higher effect sizes (P< 0.001) and lower precision (P<0.001) than matched Cochrane reviews. Reviews reporting an effect size at least 2-fold greater than their matched pair were cited more frequently.Though results between topic-matched Cochrane and non-Cochrane reviews were quite similar, discrepant results were frequent, and the overlap of included studies was surprisingly low. Non-Cochrane reviews report larger effect sizes with lower precision than Cochrane reviews, indicating systematic differences, likely reflective of methodology, between the two types of reviews that could generate different interpretations of the interventions under question.

Project description:BACKGROUND:Information on subgroup assessments in systematic reviews (SR) of atrial fibrillation (AF) is limited. This review aims to describe subgroup analyses in AF SRs to inform the design of SRs and randomized trials as well as clinical practice. METHODS:We conducted a cross sectional meta-epidemiological study of Cochrane AF reviews by searching AF (including variants) in the title, abstract, or keyword field without date or language restrictions (Issue 9; September 2018). Two reviewers independently extracted study characteristics to summarize frequency of subgroups pre-specified and conducted and report credibility of subgroup effects claimed. RESULTS:Of 39 Cochrane reviews identified, 17 met inclusion criteria (including 168 reports of 127 randomized trials) and the majority (16; 94.1%) conducted meta-analysis of outcomes. Most (13; 76.5%) planned pre-specified subgroup analyses; 7 of which (41.2%) conducted subgroups. In these 7 reviews, 56 subgroups were planned, 17 (30.4%) conducted and 6 (10.7%) yielded subgroup effects. Variables such as co-morbid disease, stroke risk factors, prior stroke/transient ischemic attack, age, race, and sex represented 44% (24 subgroups) of all planned subgroups (8 conducted; 14.3%); however, information on covariate selection was lacking. Overall, more subgroups were planned than conducted (mean difference (95% CI) 2.3 (1.2-3.5, p?<?0.001)). Of all subgroups conducted, anticoagulant characteristics comprised a third of all subgroup effects (n?=?5, 35.7%). The credibility of subgroups identified (n?=?14) was assessed and less than half (43%) represented one of a small number of pre-specified hypothesis and rarely were effects seen within studies (7%). Of 5 reviews that reported subgroup effects, only 3 discussed subgroup effects as part of the overall conclusions; none discussed credibility of subgroup effects. CONCLUSIONS:This meta-epidemiological review of a subset of Cochrane AF reviews suggests that planning and reporting of subgroup analyses in AF reviews can be improved to better inform clinical management. Most pre-specified subgroup analyses were not performed, important variables (such as stroke, bleeding risk, and other comorbidities) were rarely examined and credibility of subgroup effects claimed was low. Future reviews should aim to identify important subgroups in their protocols and use recommended approaches to test subgroup effects in order to better support clinical decision-making.

Project description:BackgroundStudies have shown that industry-sponsored meta-analyses of drugs lack scientific rigour and have biased conclusions. However, these studies have been restricted to certain medical specialities. We compared all industry-supported meta-analyses of drug-drug comparisons with those without industry support.MethodsWe searched PubMed for all meta-analyses that compared different drugs or classes of drugs published in 2004. Two authors assessed the meta-analyses and independently extracted data. We used a validated scale for judging the methodological quality and a binary scale for judging conclusions. We divided the meta-analyses according to the type of support in 3 categories: industry-supported, non-profit support or no support, and undeclared support.ResultsWe included 39 meta-analyses. Ten had industry support, 18 non-profit or no support, and 11 undeclared support. On a 0-7 scale, the median quality score was 6 for meta-analyses with non-profit or no support and 2.5 for the industry-supported meta-analyses (P < 0.01). Compared with industry-supported meta-analyses, more meta-analyses with non-profit or no support avoided bias in the selection of studies (P = 0.01), more often stated the search methods used to find studies (P = 0.02), searched comprehensively (P < 0.01), reported criteria for assessing the validity of the studies (P = 0.02), used appropriate criteria (P = 0.04), described methods of allocation concealment (P = 0.05), described methods of blinding (P = 0.05), and described excluded patients (P = 0.08) and studies (P = 0.15). Forty percent of the industry-supported meta-analyses recommended the experimental drug without reservations, compared with 22% of the meta-analyses with non-profit or no support (P = 0.57).In a sensitivity analysis, we contacted the authors of the meta-analyses with undeclared support. Eight who replied that they had not received industry funding were added to those with non-profit or no support, and 3 who did not reply were added to those with industry support. This analysis did not change the results much.ConclusionTransparency is essential for readers to make their own judgment about medical interventions guided by the results of meta-analyses. We found that industry-supported meta-analyses are less transparent than meta-analyses with non-profit support or no support.

Project description:BACKGROUND:There is growing interest in evaluating differences in healthcare interventions across routinely collected demographic characteristics. However, individual subgroup analyses in randomized controlled trials are often not prespecified, adjusted for multiple testing, or conducted using the appropriate statistical test for interaction, and therefore frequently lack credibility. Meta-analyses can be used to examine the validity of potential subgroup differences by collating evidence across trials. Here, we characterize the conduct and clinical translation of age-treatment subgroup analyses in Cochrane reviews. METHODS:For a random sample of 928 Cochrane intervention reviews of randomized trials, we determined how often subgroup analyses of age are reported, how often these analyses have a P?<?0.05 from formal interaction testing, how frequently subgroup differences first observed in an individual trial are later corroborated by other trials in the same meta-analysis, and how often statistically significant results are included in commonly used clinical management resources (BMJ Best Practice, UpToDate, Cochrane Clinical Answers, Google Scholar, and Google search). RESULTS:Among 928 Cochrane intervention reviews, 189 (20.4%) included plans to conduct age-treatment subgroup analyses. The vast majority (162 of 189, 85.7%) of the planned analyses were not conducted, commonly because of insufficient trial data. There were 22 reviews that conducted their planned age-treatment subgroup analyses, and another 3 reviews appeared to perform unplanned age-treatment subgroup analyses. These 25 (25 of 928, 2.7%) reviews conducted a total of 97 age-treatment subgroup analyses, of which 65 analyses (in 20 reviews) had non-overlapping subgroup levels. Among the 65 age-treatment subgroup analyses, 14 (21.5%) did not report any formal interaction testing. Seven (10.8%) reported P?<?0.05 from formal age-treatment interaction testing; however, none of these seven analyses were in reviews that discussed the potential biological rationale or clinical significance of the subgroup findings or had results that were included in common clinical practice resources. CONCLUSION:Age-treatment subgroup analyses in Cochrane intervention reviews were frequently planned but rarely conducted, and implications of detected interactions were not discussed in the reviews or mentioned in common clinical resources. When subgroup analyses are performed, authors should report the findings, compare the results to previous studies, and outline any potential impact on clinical care.

Project description:BackgroundMost meta-analyses include data from one or more small studies that, individually, do not have power to detect an intervention effect. The relative influence of adequately powered and underpowered studies in published meta-analyses has not previously been explored. We examine the distribution of power available in studies within meta-analyses published in Cochrane reviews, and investigate the impact of underpowered studies on meta-analysis results.Methods and findingsFor 14,886 meta-analyses of binary outcomes from 1,991 Cochrane reviews, we calculated power per study within each meta-analysis. We defined adequate power as ?50% power to detect a 30% relative risk reduction. In a subset of 1,107 meta-analyses including 5 or more studies with at least two adequately powered and at least one underpowered, results were compared with and without underpowered studies. In 10,492 (70%) of 14,886 meta-analyses, all included studies were underpowered; only 2,588 (17%) included at least two adequately powered studies. 34% of the meta-analyses themselves were adequately powered. The median of summary relative risks was 0.75 across all meta-analyses (inter-quartile range 0.55 to 0.89). In the subset examined, odds ratios in underpowered studies were 15% lower (95% CI 11% to 18%, P<0.0001) than in adequately powered studies, in meta-analyses of controlled pharmacological trials; and 12% lower (95% CI 7% to 17%, P<0.0001) in meta-analyses of controlled non-pharmacological trials. The standard error of the intervention effect increased by a median of 11% (inter-quartile range -1% to 35%) when underpowered studies were omitted; and between-study heterogeneity tended to decrease.ConclusionsWhen at least two adequately powered studies are available in meta-analyses reported by Cochrane reviews, underpowered studies often contribute little information, and could be left out if a rapid review of the evidence is required. However, underpowered studies made up the entirety of the evidence in most Cochrane reviews.

Project description:BackgroundThere has been a growing emphasis on evidence-informed decision-making in health care. Systematic reviews, such as those produced by the Cochrane Collaboration, have been a key component of this movement. The UK National Institute for Health Research (NIHR) Systematic Review Programme currently supports 20 Cochrane Review Groups (CRGs). The aim of this study was to identify the impacts of Cochrane reviews published by NIHR-funded CRGs during the years 2007-2011.MethodsWe sent questionnaires to CRGs and review authors, interviewed guideline developers and used bibliometrics and documentary review to get an overview of CRG impact and to evaluate the impact of a sample of 60 Cochrane reviews. We used a framework with four categories (knowledge production, research targeting, informing policy development and impact on practice/services).ResultsA total of 1,502 new and updated reviews were produced by the 20 NIHR-funded CRGs between 2007 and 2011. The clearest impacts were on policy with a total of 483 systematic reviews cited in 247 sets of guidance: 62 were international, 175 national (87 from the UK) and 10 local. Review authors and CRGs provided some examples of impact on practice or services, for example, safer use of medication, the identification of new effective drugs or treatments and potential economic benefits through the reduction in the use of unproven or unnecessary procedures. However, such impacts are difficult to objectively document, and the majority of reviewers were unsure if their review had produced specific impacts. Qualitative data suggested that Cochrane reviews often play an instrumental role in informing guidance, although a poor fit with guideline scope or methods, reviews being out of date and a lack of communication between CRGs and guideline developers were barriers to their use.ConclusionsHealth and economic impacts of research are generally difficult to measure. We found that to be the case with this evaluation. Impacts on knowledge production and clinical guidance were easier to identify and substantiate than those on clinical practice. Questions remain about how we define and measure impact, and more work is needed to develop suitable methods for impact analysis.

Project description:BackgroundThe QUOROM and PRISMA statements were published in 1999 and 2009, respectively, to improve the consistency of reporting systematic reviews (SRs)/meta-analyses (MAs) of clinical trials. However, not all SRs/MAs adhere completely to these important standards. In particular, it is not clear how well SRs/MAs of acupuncture studies adhere to reporting standards and which reporting criteria are generally ignored in these analyses.ObjectivesTo evaluate reporting quality in SRs/MAs of acupuncture studies.MethodsWe performed a literature search for studies published prior to 2014 using the following public archives: PubMed, EMBASE, Web of Science, the Cochrane Database of Systematic Reviews (CDSR), the Chinese Biomedical Literature Database (CBM), the Traditional Chinese Medicine (TCM) database, the Chinese Journal Full-text Database (CJFD), the Chinese Scientific Journal Full-text Database (CSJD), and the Wanfang database. Data were extracted into pre-prepared Excel data-extraction forms. Reporting quality was assessed based on the PRISMA checklist (27 items).ResultsOf 476 appropriate SRs/MAs identified in our search, 203, 227, and 46 were published in Chinese journals, international journals, and the Cochrane Database, respectively. In 476 SRs/MAs, only 3 reported the information completely. By contrast, approximately 4.93% (1/203), 8.81% (2/227) and 0.00% (0/46) SRs/Mas reported less than 10 items in Chinese journals, international journals and CDSR, respectively. In general, the least frequently reported items (reported≤50%) in SRs/MAs were "protocol and registration", "risk of bias across studies", and "additional analyses" in both methods and results sections.ConclusionsSRs/MAs of acupuncture studies have not comprehensively reported information recommended in the PRISMA statement. Our study underscores that, in addition to focusing on careful study design and performance, attention should be paid to comprehensive reporting standards in SRs/MAs on acupuncture studies.

Dataset Information

Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review.

Objective

Design

Data sources

Data extraction

Results

Conclusions

Publications

Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets