Rank reversal in indirect comparisons.
Ontology highlight
ABSTRACT: Objective
To describe rank reversal as a source of inconsistent interpretation intrinsic to indirect comparison (Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epi 1997;50:683-91) of treatments and to propose best practice.Methods
We prove our main points with intuition, examples, graphs, and mathematical proofs. We also provide software and discuss implications for research and policy.Results
When comparing treatments by indirect means and sorting them by effect size, three common measures of comparison (risk ratio, risk difference, and odds ratio) may lead to vastly different rankings.Conclusions
The choice of risk measure matters when making indirect comparisons of treatments. The choice should depend primarily on the study design and the conceptual framework for that study.
SUBMITTER: Norton EC
PROVIDER: S-EPMC3527821 | biostudies-literature | 2012 Dec
REPOSITORIES: biostudies-literature
Publications
Rank reversal in indirect comparisons.
Norton Edward C EC Miller Morgen M MM Wang Jason J JJ Coyne Kasey K Kleinman Lawrence C LC
Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research 20120907 8
<h4>Objective</h4>To describe rank reversal as a source of inconsistent interpretation intrinsic to indirect comparison (Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epi 1997;50:683-91) of treatments and to propose best practice.<h4>Methods</h4>We prove our main points with intuition, examples, graphs, and mathematical proofs. We also provide software and discuss implications for re ...[more]
Similar Datasets
Project description: Not available
| S-EPMC11344805 | biostudies-literature
Project description:BackgroundThe indirect comparison of two interventions can be valuable in many situations. However, the quality of an indirect comparison will depend on several factors including the chosen methodology and validity of underlying assumptions. Published indirect comparisons are increasingly more common in the medical literature, but as yet, there are no published recommendations of how they should be reported. Our aim is to systematically review the quality of published indirect comparisons to add to existing empirical data suggesting that improvements can be made when reporting and applying indirect comparisons.Methodology/findingsReviews applying statistical methods to indirectly compare the clinical effectiveness of two interventions using randomised controlled trials were eligible. We searched (1966-2008) Database of Abstracts and Reviews of Effects, The Cochrane library, and Medline. Full review publications were assessed for eligibility. Specific criteria to assess quality were developed and applied. Forty-three reviews were included. Adequate methodology was used to calculate the indirect comparison in 41 reviews. Nineteen reviews assessed the similarity assumption using sensitivity analysis, subgroup analysis, or meta-regression. Eleven reviews compared trial-level characteristics. Twenty-four reviews assessed statistical homogeneity. Twelve reviews investigated causes of heterogeneity. Seventeen reviews included direct and indirect evidence for the same comparison; six reviews assessed consistency. One review combined both evidence types. Twenty-five reviews urged caution in interpretation of results, and 24 reviews indicated when results were from indirect evidence by stating this term with the result.ConclusionsThis review shows that the underlying assumptions are not routinely explored or reported when undertaking indirect comparisons. We recommend, therefore, that the quality of indirect comparisons should be improved, in particular, by assessing assumptions and reporting the assessment methods applied. We propose that the quality criteria applied in this article may provide a basis to help review authors carry out indirect comparisons and to aid appropriate interpretation.
| S-EPMC2978085 | biostudies-literature
Project description:Standard methods for indirect comparisons and network meta-analysis are based on aggregate data, with the key assumption that there is no difference between the trials in the distribution of effect-modifying variables. Methods which relax this assumption are becoming increasingly common for submissions to reimbursement agencies, such as the National Institute for Health and Care Excellence (NICE). These methods use individual patient data from a subset of trials to form population-adjusted indirect comparisons between treatments, in a specific target population. Recently proposed population adjustment methods include the Matching-Adjusted Indirect Comparison (MAIC) and the Simulated Treatment Comparison (STC). Despite increasing popularity, MAIC and STC remain largely untested. Furthermore, there is a lack of clarity about exactly how and when they should be applied in practice, and even whether the results are relevant to the decision problem. There is therefore a real and present risk that the assumptions being made in one submission to a reimbursement agency are fundamentally different to-or even incompatible with-the assumptions being made in another for the same indication. We describe the assumptions required for population-adjusted indirect comparisons, and demonstrate how these may be used to generate comparisons in any given target population. We distinguish between anchored and unanchored comparisons according to whether a common comparator arm is used or not. Unanchored comparisons make much stronger assumptions, which are widely regarded as infeasible. We provide recommendations on how and when population adjustment methods should be used, and the supporting analyses that are required to provide statistically valid, clinically meaningful, transparent and consistent results for the purposes of health technology appraisal. Simulation studies are needed to examine the properties of population adjustment methods and their robustness to breakdown of assumptions.
| S-EPMC5774635 | biostudies-literature
Project description:Whereas the theory of confirmatory adaptive designs is well understood for uncensored data, implementation of adaptive designs in the context of survival trials remains challenging. Commonly used adaptive survival tests are based on the independent increments structure of the log-rank statistic. This implies some relevant limitations: On the one hand, essentially only the interim log-rank statistic may be used for design modifications (such as data-dependent sample size recalculation). Furthermore, the treatment arm allocation ratio in these classical methods is assumed to be constant throughout the trial period. Here, we propose an extension of the independent increments approach to adaptive survival tests that addresses some of these limitations. We present a confirmatory adaptive two-sample log-rank test that allows rejection regions and sample size recalculation rules to be based not only on the interim log-rank statistic, but also on point-wise survival rate estimates, simultaneously. In addition, the possibility is opened to adapt the treatment arm allocation ratio after each interim analysis in a data-dependent way. The ability to include point-wise survival rate estimators in the rejection region of a test for comparing survival curves might be attractive, e.g., for seamless phase II/III designs. Data-dependent adaptation of the allocation ratio could be helpful in multi-arm trials in order to successively steer recruitment into the study arms with the greatest chances of success. The methodology is motivated by the LOGGIC Europe Trial from pediatric oncology. Distributional properties are derived using martingale techniques in the large sample limit. Small sample properties are studied by simulation.
| S-EPMC8649467 | biostudies-literature
Project description:Why tropical forests harbour an exceptional number of species with striking differences in abundances remains an open question. We propose a theoretical framework to address this question in which rare species may have different extirpation risks depending on species ranks in tree growth and sensitivities to neighbourhood interactions. To evaluate the framework, we studied tree growth and its responses to neighbourhood dissimilarity (ND) in traits and phylogeny for 146 species in a neotropical forest. We found that tree growth was positively related to ND, and common species were more strongly affected by ND than rare species, which may help delay dominance of common species. Rare species grew more slowly at the community-wide average ND than common species. But rare species grew faster when common species tended to dominate locally, which may help reduce extirpation risk of rare species. Our study highlights that tree growth rank among species depends on their responses to neighbourhood interactions, which can be important in fostering diversity maintenance in tropical forests.
| S-EPMC5904314 | biostudies-literature
Project description:ObjectiveTo investigate the agreement between direct and indirect comparisons of competing healthcare interventions.DesignMeta-epidemiological study based on sample of meta-analyses of randomised controlled trials. Data sources Cochrane Database of Systematic Reviews and PubMed. Inclusion criteria Systematic reviews that provided sufficient data for both direct comparison and independent indirect comparisons of two interventions on the basis of a common comparator and in which the odds ratio could be used as the outcome statistic.Main outcome measureInconsistency measured by the difference in the log odds ratio between the direct and indirect methods.ResultsThe study included 112 independent trial networks (including 1552 trials with 478,775 patients in total) that allowed both direct and indirect comparison of two interventions. Indirect comparison had already been explicitly done in only 13 of the 85 Cochrane reviews included. The inconsistency between the direct and indirect comparison was statistically significant in 16 cases (14%, 95% confidence interval 9% to 22%). The statistically significant inconsistency was associated with fewer trials, subjectively assessed outcomes, and statistically significant effects of treatment in either direct or indirect comparisons. Owing to considerable inconsistency, many (14/39) of the statistically significant effects by direct comparison became non-significant when the direct and indirect estimates were combined.ConclusionsSignificant inconsistency between direct and indirect comparisons may be more prevalent than previously observed. Direct and indirect estimates should be combined in mixed treatment comparisons only after adequate assessment of the consistency of the evidence.
| S-EPMC3156578 | biostudies-literature
Project description:PURPOSE:Prior studies have found that women in academic medicine do not advance or remain in their careers in parity with men. The authors examined a cohort of faculty from the 1995 National Faculty Survey to identify predictors of advancement, retention, and leadership for women faculty. METHOD:The authors followed 1,273 faculty at 24 medical schools in the continental United States for 17 years to identify predictors of advancement, retention, and leadership for women faculty. Schools were balanced for public or private status and the four Association of American Medical Colleges geographic regions. The authors used regression models to adjust for covariates: seniority, department, academic setting, and race/ethnicity. RESULTS:After adjusting for significant covariates, women were less likely than men to achieve the rank of professor (OR = 0.57; 95% CI, 0.43-0.78) or to remain in academic careers (OR = 0.68; 95% CI, 0.49-0.94). When number of refereed publications was added to the model, differences by gender in retention and attainment of senior rank were no longer significant. Male faculty were more likely to hold senior leadership positions after adjusting for publications (OR = 0.49; 95% CI, 0.35-0.69). CONCLUSIONS:Gender disparities in rank, retention, and leadership remain across the career trajectories of the faculty cohort in this study. Women were less likely to attain senior-level positions than men, even after adjusting for publication-related productivity. Institutions must examine the climate for women to ensure their academic capital is fully utilized and equal opportunity exists for leadership.
| S-EPMC6066448 | biostudies-literature
Project description:Population adjustment methods such as matching-adjusted indirect comparison (MAIC) are increasingly used to compare marginal treatment effects when there are cross-trial differences in effect modifiers and limited patient-level data. MAIC is based on propensity score weighting, which is sensitive to poor covariate overlap and cannot extrapolate beyond the observed covariate space. Current outcome regression-based alternatives can extrapolate but target a conditional treatment effect that is incompatible in the indirect comparison. When adjusting for covariates, one must integrate or average the conditional estimate over the relevant population to recover a compatible marginal treatment effect. We propose a marginalization method based on parametric G-computation that can be easily applied where the outcome regression is a generalized linear model or a Cox model. The approach views the covariate adjustment regression as a nuisance model and separates its estimation from the evaluation of the marginal treatment effect of interest. The method can accommodate a Bayesian statistical framework, which naturally integrates the analysis into a probabilistic framework. A simulation study provides proof-of-principle and benchmarks the method's performance against MAIC and the conventional outcome regression. Parametric G-computation achieves more precise and more accurate estimates than MAIC, particularly when covariate overlap is poor, and yields unbiased marginal treatment effect estimates under no failures of assumptions. Furthermore, the marginalized regression-adjusted estimates provide greater precision and accuracy than the conditional estimates produced by the conventional outcome regression, which are systematically biased because the measure of effect is non-collapsible.
| S-EPMC9790405 | biostudies-literature
Project description:Standard network meta-analysis and indirect comparisons combine aggregate data from multiple studies on treatments of interest, assuming that any factors that interact with treatment effects (effect modifiers) are balanced across populations. Population adjustment methods such as multilevel network meta-regression (ML-NMR), matching-adjusted indirect comparison (MAIC), and simulated treatment comparison (STC) relax this assumption using individual patient data from one or more studies, and are becoming increasingly prevalent in health technology appraisals and the applied literature. Motivated by an applied example and two recent reviews of applications, we undertook an extensive simulation study to assess the performance of these methods in a range of scenarios under various failures of assumptions. We investigated the impact of varying sample size, missing effect modifiers, strength of effect modification and validity of the shared effect modifier assumption, validity of extrapolation and varying between-study overlap, and different covariate distributions and correlations. ML-NMR and STC performed similarly, eliminating bias when the requisite assumptions were met. Serious concerns are raised for MAIC, which performed poorly in nearly all simulation scenarios and may even increase bias compared with standard indirect comparisons. All methods incur bias when an effect modifier is missing, highlighting the necessity of careful selection of potential effect modifiers prior to analysis. When all effect modifiers are included, ML-NMR and STC are robust techniques for population adjustment. ML-NMR offers additional advantages over MAIC and STC, including extending to larger treatment networks and producing estimates in any target population, making this an attractive choice in a variety of scenarios.
| S-EPMC8690023 | biostudies-literature
Project description:IntroductionIt can be challenging for dermatologists to keep abreast of the growing evidence from published indirect comparisons (ICs) of treatments for psoriasis. The objective of this analysis was to summarise comparative clinical efficacy and safety findings from ICs of systemic biologics for the treatment of moderate-to-severe psoriasis and to identify factors potentially affecting efficacy outcomes and their possible implications for clinical decision making.MethodsAn umbrella review of short- and long-term efficacy and safety findings from 26 ICs visually compared 90% improvement in Psoriasis Area Severity Index (PASI90) treatment rankings and three safety outcome rankings side by side. Pearson's correlation coefficients measured the strength of the association between each pair of ICs on the basis of identified factors that could potentially affect efficacy findings.ResultsSome consistency in short-term PASI90 efficacy rankings was observed for certain drugs, although rankings for most drugs varied by IC. Factors potentially affecting efficacy outcomes included the use of different methodologies for head-to-head comparison and statistical analyses, and variation in drugs and classes included treatment dosing and duration, outcome definitions and effect measures reported between ICs. Considerable variation in these factors was found across all 26 ICs. Comparative safety information of value to physicians was limited.ConclusionsSubstantial differences were identified between ICs in factors that could potentially affect efficacy outcomes. Treatment rankings must be interpreted alongside actual differences in IC outcomes to allow conclusions on clinical relevance. Drugs within a class cannot be considered of equal efficacy: therapies should be considered individually rather than by class.
| S-EPMC9357597 | biostudies-literature