Dataset Information

Agreement between ranking metrics in network meta-analysis: an empirical study.

ABSTRACT: OBJECTIVE:To empirically explore the level of agreement of the treatment hierarchies from different ranking metrics in network meta-analysis (NMA) and to investigate how network characteristics influence the agreement. DESIGN:Empirical evaluation from re-analysis of NMA. DATA:232 networks of four or more interventions from randomised controlled trials, published between 1999 and 2015. METHODS:We calculated treatment hierarchies from several ranking metrics: relative treatment effects, probability of producing the best value [Formula: see text] and the surface under the cumulative ranking curve (SUCRA). We estimated the level of agreement between the treatment hierarchies using different measures: Kendall's ? and Spearman's ? correlation; and the Yilmaz [Formula: see text] and Average Overlap, to give more weight to the top of the rankings. Finally, we assessed how the amount of the information present in a network affects the agreement between treatment hierarchies, using the average variance, the relative range of variance and the total sample size over the number of interventions of a network. RESULTS:Overall, the pairwise agreement was high for all treatment hierarchies obtained by the different ranking metrics. The highest agreement was observed between SUCRA and the relative treatment effect for both correlation and top-weighted measures whose medians were all equal to 1. The agreement between rankings decreased for networks with less precise estimates and the hierarchies obtained from [Formula: see text] appeared to be the most sensitive to large differences in the variance estimates. However, such large differences were rare. CONCLUSIONS:Different ranking metrics address different treatment hierarchy problems, however they produced similar rankings in the published networks. Researchers reporting NMA results can use the ranking metric they prefer, unless there are imprecise estimates or large imbalances in the variance estimates. In this case treatment hierarchies based on both probabilistic and non-probabilistic ranking metrics should be presented.

SUBMITTER: Chiocchia V

PROVIDER: S-EPMC7440831 | biostudies-literature | 2020 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Agreement between ranking metrics in network meta-analysis: an empirical study.

Chiocchia Virginia V Nikolakopoulou Adriani A Papakonstantinou Theodoros T Egger Matthias M Salanti Georgia G

BMJ open 20200820 8

<h4>Objective</h4>To empirically explore the level of agreement of the treatment hierarchies from different ranking metrics in network meta-analysis (NMA) and to investigate how network characteristics influence the agreement.<h4>Design</h4>Empirical evaluation from re-analysis of NMA.<h4>Data</h4>232 networks of four or more interventions from randomised controlled trials, published between 1999 and 2015.<h4>Methods</h4>We calculated treatment hierarchies from several ranking metrics: relative ...[more]

PMID: 32819946

Similar Datasets

Project description:To examine whether the continuous updating of networks of prospectively planned randomised controlled trials (RCTs) ("living" network meta-analysis) provides strong evidence against the null hypothesis in comparative effectiveness of medical interventions earlier than the updating of conventional, pairwise meta-analysis.Empirical study of the accumulating evidence about the comparative effectiveness of clinical interventions.Database of network meta-analyses of RCTs identified through searches of Medline, Embase, and the Cochrane Database of Systematic Reviews until 14 April 2015.Network meta-analyses published after January 2012 that compared at least five treatments and included at least 20 RCTs. Clinical experts were asked to identify in each network the treatment comparison of greatest clinical interest. Comparisons were excluded for which direct and indirect evidence disagreed, based on side, or node, splitting test (P<0.10).Cumulative pairwise and network meta-analyses were performed for each selected comparison. Monitoring boundaries of statistical significance were constructed and the evidence against the null hypothesis was considered to be strong when the monitoring boundaries were crossed. A significance level was defined as ?=5%, power of 90% (?=10%), and an anticipated treatment effect to detect equal to the final estimate from the network meta-analysis. The frequency and time to strong evidence was compared against the null hypothesis between pairwise and network meta-analyses.49 comparisons of interest from 44 networks were included; most (n=39, 80%) were between active drugs, mainly from the specialties of cardiology, endocrinology, psychiatry, and rheumatology. 29 comparisons were informed by both direct and indirect evidence (59%), 13 by indirect evidence (27%), and 7 by direct evidence (14%). Both network and pairwise meta-analysis provided strong evidence against the null hypothesis for seven comparisons, but for an additional 10 comparisons only network meta-analysis provided strong evidence against the null hypothesis (P=0.002). The median time to strong evidence against the null hypothesis was 19 years with living network meta-analysis and 23 years with living pairwise meta-analysis (hazard ratio 2.78, 95% confidence interval 1.00 to 7.72, P=0.05). Studies directly comparing the treatments of interest continued to be published for eight comparisons after strong evidence had become evident in network meta-analysis.In comparative effectiveness research, prospectively planned living network meta-analyses produced strong evidence against the null hypothesis more often and earlier than conventional, pairwise meta-analyses.

Project description:Rationale and objectivesMany patients with chronic obstructive pulmonary disease (COPD) have marked discordance between forced expiratory volume in 1 second (FEV1) and degree of emphysema on computed tomography (CT). Biomechanical differences between these patients have not been studied. We aimed to identify reasons for the discordance between CT and spirometry in some patients with COPD.Materials and methodsSubjects with Global initiative for chronic Obstructive Lung Disease stages I-IV from a large multicenter study (The Genetic Epidemiology of COPD) were arranged by percentiles of %predicted FEV1 and emphysema on CT. Three categories were created using differences in percentiles: Catspir with predominant airflow obstruction/minimal emphysema, CatCT with predominant emphysema/minimal airflow obstruction, and Catmatched with matched FEV1 and emphysema. Image registration was used to derive Jacobian determinants, a measure of lung elasticity, anisotropy, and strain tensors, to assess biomechanical differences between groups. Regression models were created with the previously mentioned categories as outcome variable, adjusting for demographics, scanner type, quantitative CT-derived emphysema, gas trapping, and airway thickness (model 1), and after adding biomechanical CT metrics (model 2).ResultsJacobian determinants, anisotropy, and strain tensors were strongly associated with FEV1. With Catmatched as control, model 2 predicted Catspir and CatCT better than model 1 (Akaike information criterion 255.8 vs. 320.8). In addition to demographics, the strongest independent predictors of FEV1 were Jacobian mean (β = 1.60,95%confidence intervals [CI] = 1.16 to 1.98; P < 0.001), coefficient of variation (CV) of Jacobian (β = 1.45,95%CI = 0.86 to 2.03; P < 0.001), and CV of strain (β = 1.82,95%CI = 0.68 to 2.95; P = 0.001). CVs of Jacobian and strain are both potential markers of biomechanical lung heterogeneity.ConclusionsCT-derived measures of lung mechanics improve the link between quantitative CT and spirometry, offering the potential for new insights into the linkage between regional parenchymal destruction and global decrement in lung function in patients with COPD.

Project description:BackgroundThere exist many methods for describing the complex relation between changes of gene expression in molecular pathways or gene ontologies under different experimental conditions. Among them, Gene Set Enrichment Analysis seems to be one of the most commonly used (over 10,000 citations). An important parameter, which could affect the final result, is the choice of a metric for the ranking of genes. Applying a default ranking metric may lead to poor results.Methods and resultsIn this work 28 benchmark data sets were used to evaluate the sensitivity and false positive rate of gene set analysis for 16 different ranking metrics including new proposals. Furthermore, the robustness of the chosen methods to sample size was tested. Using k-means clustering algorithm a group of four metrics with the highest performance in terms of overall sensitivity, overall false positive rate and computational load was established i.e. absolute value of Moderated Welch Test statistic, Minimum Significant Difference, absolute value of Signal-To-Noise ratio and Baumgartner-Weiss-Schindler test statistic. In case of false positive rate estimation, all selected ranking metrics were robust with respect to sample size. In case of sensitivity, the absolute value of Moderated Welch Test statistic and absolute value of Signal-To-Noise ratio gave stable results, while Baumgartner-Weiss-Schindler and Minimum Significant Difference showed better results for larger sample size. Finally, the Gene Set Enrichment Analysis method with all tested ranking metrics was parallelised and implemented in MATLAB, and is available at https://github.com/ZAEDPolSl/MrGSEA .ConclusionsChoosing a ranking metric in Gene Set Enrichment Analysis has critical impact on results of pathway enrichment analysis. The absolute value of Moderated Welch Test has the best overall sensitivity and Minimum Significant Difference has the best overall specificity of gene set analysis. When the number of non-normally distributed genes is high, using Baumgartner-Weiss-Schindler test statistic gives better outcomes. Also, it finds more enriched pathways than other tested metrics, which may induce new biological discoveries.

Dataset Information

Agreement between ranking metrics in network meta-analysis: an empirical study.

Publications

Agreement between ranking metrics in network meta-analysis: an empirical study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets