Dataset Information

The Impact of Excluding Trials from Network Meta-Analyses - An Empirical Study.

ABSTRACT: Network meta-analysis (NMA) expands the scope of a conventional pairwise meta-analysis to simultaneously compare multiple treatments, which has an inherent appeal for clinicians, patients, and policy decision makers. Two recent reports have shown that the impact of excluding a treatment on NMAs can be substantial. However, no one has assessed the impact of excluding a trial from NMAs, which is important because many NMAs selectively include trials in the analysis. This article empirically examines the impact of trial exclusion using both the arm-based (AB) and contrast-based (CB) approaches, by reanalyzing 20 published NMAs involving 725 randomized controlled trials and 449,325 patients. For the population-averaged absolute risk estimates using the AB approach, the average fold changes across all networks ranged from 1.004 (with standard deviation 0.004) to 1.072 (with standard deviation 0.184); while the maximal fold changes ranged from 1.032 to 2.349. In 12 out of 20 NMAs, a 1.20-fold or larger change is observed in at least one of the population-averaged absolute risk estimates. In addition, while excluding a trial can substantially change the estimated relative effects (e.g., log odds ratios), there is no systematic difference in terms of changes between the two approaches. Changes in treatment rankings are observed in 7 networks and changes in inconsistency are observed in 3 networks. We do not observe correlations between changes in treatment effects, treatment rankings and inconsistency. Finally, we recommend rigorous inclusion and exclusion criteria, logical study selection process, and reasonable network geometry to ensure robustness and generalizability of the results of NMAs.

SUBMITTER: Zhang J

PROVIDER: S-EPMC5142775 | biostudies-other | 2016

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

The Impact of Excluding Trials from Network Meta-Analyses - An Empirical Study.

Zhang Jing J Yuan Yiping Y Chu Haitao H

PloS one 20161207 12

Network meta-analysis (NMA) expands the scope of a conventional pairwise meta-analysis to simultaneously compare multiple treatments, which has an inherent appeal for clinicians, patients, and policy decision makers. Two recent reports have shown that the impact of excluding a treatment on NMAs can be substantial. However, no one has assessed the impact of excluding a trial from NMAs, which is important because many NMAs selectively include trials in the analysis. This article empirically examin ...[more]

PMID: 27926924

Similar Datasets

Project description:OBJECTIVE: To evaluate the frequency, validity, and relevance of statistically significant (P<0.05) sex-treatment interactions in randomized controlled trials in Cochrane meta-analyses. DESIGN: Meta-epidemiological study. DATA SOURCES: Cochrane Database of Systematic Reviews (CDSR) and PubMed. ELIGIBILITY CRITERIA FOR STUDY SELECTION: Reviews published in the CDSR with sex-treatment subgroup analyses in the forest plots, using data from randomized controlled trials. DATA EXTRACTION: Information on the study design and sex subgroup data were extracted from reviews and forest plots that met inclusion criteria. For each statistically significant sex-treatment interaction, the potential for biological plausibility and clinical significance was considered. RESULTS: Among the 41 reviews with relevant data, there were 109 separate treatment-outcome analyses ("topics"). Among the 109 topics, eight (7%) had a statistically significant sex-treatment interaction. The 109 topics included 311 randomized controlled trials (162 with both sexes, 46 with males only, 103 with females only). Of the 162 individual randomized controlled trials that included both sexes, 15 (9%) had a statistically significant sex-treatment interaction. Of four topics where the first published randomized controlled trial had a statistically significant sex-treatment interaction, no meta-analyses that included other randomized controlled trials retained the statistical significance and no meta-analyses showed statistical significance when data from the first published randomized controlled trial were excluded. Of the eight statistically significant sex-treatment interactions from the overall analyses, only three were discussed by the CDSR reviewers for a potential impact on different clinical management for males compared with females. None of these topics had a sex-treatment interaction that influenced treatment recommendations in recent guidelines. UpToDate, an online physician-authored clinical decision support resource, suggested differential management of men and women for one of these sex-treatment interactions. CONCLUSION: Statistically significant sex-treatment interactions are only slightly more frequent than what would be expected by chance and there is little evidence of subsequent corroboration or clinical relevance of sex-treatment interactions.

Project description:IntroductionAcute, non-low back-related musculoskeletal pain is common and associated with significant socioeconomic costs. No review has evaluated all interventional studies for acute musculoskeletal pain, which limits attempts to make inferences regarding the relative effectiveness of treatments.Methods and analysisWe will conduct a systematic review of all randomised controlled trials evaluating therapies for acute musculoskeletal pain (excluding low back pain). We will identify eligible, English-language, trials by a systematic search of the Cumulative Index to Nursing and Allied Health Literature (CINAHL), Embase, Medline, Physiotherapy Evidence Database (PEDro) and the Cochrane Central Register of Controlled Trials (CENTRAL) from inception to February 2018. Eligible trials will: (1) enrol patients presenting with acute, non-low back-related musculoskeletal pain (duration of pain ?4 weeks), and (2) randomise patients to alternative interventions or an intervention and a placebo/sham arm. Fractures will be considered ineligible, unless they are non-surgical and therapy is directed at pain relief. Pairs of reviewers will, independently and in duplicate, screen titles and abstracts of identified citations, review the full texts of potentially eligible trials and extract information from eligible trials. We will use a modified Cochrane instrument to evaluate risk of bias. Disagreements will be resolved through discussion to achieve consensus. We will use the Grading of Recommendations Assessment, Development and Evaluation approach to evaluate the quality of evidence supporting treatment effects. When possible, we will conduct: (1) in direct comparisons, a random-effect meta-analysis to establish the effectiveness of therapeutic interventions on patient-important outcomes; and (2) multiple treatment comparison meta-analysis to assess the relative effects of treatments. We will use a priori hypotheses to explain heterogeneity between studies. We will use STATA V.14.2 for all analyses.Ethics and disseminationNo research ethics approval is required for this systematic review, as no confidential patient data will be used. The results of this systematic review will be disseminated through publication in a peer-reviewed journal, conference presentations and will inform a clinical practice guideline.Prospero registration numberCRD42018094412.

Project description:ObjectiveTo empirically explore the level of agreement of the treatment hierarchies from different ranking metrics in network meta-analysis (NMA) and to investigate how network characteristics influence the agreement.DesignEmpirical evaluation from re-analysis of NMA.Data232 networks of four or more interventions from randomised controlled trials, published between 1999 and 2015.MethodsWe calculated treatment hierarchies from several ranking metrics: relative treatment effects, probability of producing the best value [Formula: see text] and the surface under the cumulative ranking curve (SUCRA). We estimated the level of agreement between the treatment hierarchies using different measures: Kendall's τ and Spearman's ρ correlation; and the Yilmaz [Formula: see text] and Average Overlap, to give more weight to the top of the rankings. Finally, we assessed how the amount of the information present in a network affects the agreement between treatment hierarchies, using the average variance, the relative range of variance and the total sample size over the number of interventions of a network.ResultsOverall, the pairwise agreement was high for all treatment hierarchies obtained by the different ranking metrics. The highest agreement was observed between SUCRA and the relative treatment effect for both correlation and top-weighted measures whose medians were all equal to 1. The agreement between rankings decreased for networks with less precise estimates and the hierarchies obtained from [Formula: see text] appeared to be the most sensitive to large differences in the variance estimates. However, such large differences were rare.ConclusionsDifferent ranking metrics address different treatment hierarchy problems, however they produced similar rankings in the published networks. Researchers reporting NMA results can use the ranking metric they prefer, unless there are imprecise estimates or large imbalances in the variance estimates. In this case treatment hierarchies based on both probabilistic and non-probabilistic ranking metrics should be presented.

Project description:BackgroundResearch collaboration contributes to the advancement of knowledge by exploiting the results of scientific efforts more efficiently, but the global patterns of collaboration on meta-analysis are unknown. The purpose of this research was to describe and characterize the global collaborative patterns in meta-analyses of randomized trials published in high impact factor medical journals over the past three decades.MethodsThis was a cross-sectional, social network analysis. We searched PubMed for relevant meta-analyses of randomized trials published up to December 2012. We selected meta-analyses (including at least randomized trials as primary evidence source) published in the top seven high impact factor general medical journals (according to Journal Citation Reports 2011): The New England Journal of Medicine, The Lancet, the BMJ, JAMA, Annals of Internal Medicine, Archives of Internal Medicine (now renamed JAMA Internal Medicine), and PLoS Medicine. Opinion articles, conceptual papers, narrative reviews, reviews without meta-analysis, reviews of reviews, and other study designs were excluded.ResultsOverall, we included 736 meta-analyses, in which 3,178 authors, 891 institutions, and 51 countries participated. The BMJ was the journal that published the greatest number of articles (39%), followed by The Lancet (18%), JAMA (15%) and the Archives of Internal Medicine (15%). The USA, the UK, and Canada headed the absolute global productivity ranking in number of papers. The 64 authors and the 39 institutions with the highest publication rates were identified. We also found 82 clusters of authors (one group with 55 members and one group with 54 members) and 19 clusters of institutions (one major group with 76 members). The most prolific authors were mainly affiliated with the University of Oxford (UK), McMaster University (Canada), and the University of Bern (Switzerland).ConclusionsOur analysis identified networks of authors, institutions and countries publishing meta-analyses of randomized trials in high impact medical journals. This valuable information may be used to strengthen scientific capacity for collaboration and to help to promote a global agenda for future research of excellence.

Project description:To examine whether the continuous updating of networks of prospectively planned randomised controlled trials (RCTs) ("living" network meta-analysis) provides strong evidence against the null hypothesis in comparative effectiveness of medical interventions earlier than the updating of conventional, pairwise meta-analysis.Empirical study of the accumulating evidence about the comparative effectiveness of clinical interventions.Database of network meta-analyses of RCTs identified through searches of Medline, Embase, and the Cochrane Database of Systematic Reviews until 14 April 2015.Network meta-analyses published after January 2012 that compared at least five treatments and included at least 20 RCTs. Clinical experts were asked to identify in each network the treatment comparison of greatest clinical interest. Comparisons were excluded for which direct and indirect evidence disagreed, based on side, or node, splitting test (P<0.10).Cumulative pairwise and network meta-analyses were performed for each selected comparison. Monitoring boundaries of statistical significance were constructed and the evidence against the null hypothesis was considered to be strong when the monitoring boundaries were crossed. A significance level was defined as ?=5%, power of 90% (?=10%), and an anticipated treatment effect to detect equal to the final estimate from the network meta-analysis. The frequency and time to strong evidence was compared against the null hypothesis between pairwise and network meta-analyses.49 comparisons of interest from 44 networks were included; most (n=39, 80%) were between active drugs, mainly from the specialties of cardiology, endocrinology, psychiatry, and rheumatology. 29 comparisons were informed by both direct and indirect evidence (59%), 13 by indirect evidence (27%), and 7 by direct evidence (14%). Both network and pairwise meta-analysis provided strong evidence against the null hypothesis for seven comparisons, but for an additional 10 comparisons only network meta-analysis provided strong evidence against the null hypothesis (P=0.002). The median time to strong evidence against the null hypothesis was 19 years with living network meta-analysis and 23 years with living pairwise meta-analysis (hazard ratio 2.78, 95% confidence interval 1.00 to 7.72, P=0.05). Studies directly comparing the treatments of interest continued to be published for eight comparisons after strong evidence had become evident in network meta-analysis.In comparative effectiveness research, prospectively planned living network meta-analyses produced strong evidence against the null hypothesis more often and earlier than conventional, pairwise meta-analyses.

Project description:To explore whether systematic reviewers selectively include trial effect estimates in meta-analyses when multiple are available, and what impact this may have on meta-analytic effects.Cross-sectional study.We randomly selected systematic reviews of interventions from 2 clinical specialties published between January 2010 and 2012. The first presented meta-analysis of a continuous outcome in each review was selected (index meta-analysis), and all trial effect estimates that were eligible for inclusion in the meta-analysis (eg, from multiple scales or time points) were extracted from trial reports.We calculated a statistic (the Potential Bias Index (PBI)) to quantify and test for evidence of selective inclusion. The PBI ranges from 0 to 1; values above or below 0.5 are suggestive of selective inclusion of effect estimates more or less favourable to the intervention, respectively. The impact of any potential selective inclusion was investigated by comparing the index meta-analytic standardised mean difference (SMD) to the median of a randomly constructed distribution of meta-analytic SMDs (representing the meta-analytic SMD expected when there is no selective inclusion).31 reviews (250 trials) were included. The estimated PBI was 0.57 (95% CI 0.50 to 0.63), suggesting that trial effect estimates that were more favourable to the intervention were included in meta-analyses slightly more often than expected under a process consistent with random selection; however, the 95% CI included the null hypothesis of no selective inclusion. Any potential selective inclusion did not have an important impact on the meta-analytic effects.There was no clear evidence that selective inclusion of trial effect estimates occurred in this sample of meta-analyses. Further research on selective inclusion in other clinical specialties is needed. To enable readers to assess the risk of selective inclusion bias, we recommend that systematic reviewers report the methods used to select effect estimates to include in meta-analyses.

Dataset Information

The Impact of Excluding Trials from Network Meta-Analyses - An Empirical Study.

Publications

The Impact of Excluding Trials from Network Meta-Analyses - An Empirical Study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets