Dataset Information

A Bias in the Evaluation of Bias Comparing Randomized Trials with Nonexperimental Studies.

ABSTRACT: In a recent BMJ article, the authors conducted a meta-analysis to compare estimated treatment effects from randomized trials with those derived from observational studies based on routinely collected data (RCD). They calculated a pooled relative odds ratio (ROR) of 1.31 (95% confidence interval [CI]: 1.03-1.65) and concluded that RCD studies systematically over-estimated protective effects. However, their meta-analysis inverted results for some clinical questions to force all estimates from RCD to be below 1. We evaluated the statistical properties of this pooled ROR, and found that the selective inversion rule employed in the original meta-analysis can positively bias the estimate of the ROR. We then repeated the random effects meta-analysis using a different inversion rule and found an estimated ROR of 0.98 (0.78-1.23), indicating the ROR is highly dependent on the direction of comparisons. As an alternative to the ROR, we calculated the observed proportion of clinical questions where the RCD and trial CIs overlap, as well as the expected proportion assuming no systematic difference between the studies. Out of 16 clinical questions, 50% CIs overlapped for 8 (50%; 25 to 75%) compared with an expected overlap of 60% assuming no systematic difference between RCD studies and trials. Thus, there was little evidence of a systematic difference in effect estimates between RCD and RCTs. Estimates of pooled RORs across distinct clinical questions are generally not interpretable and may be misleading.

SUBMITTER: Franklin JM

PROVIDER: S-EPMC5710821 | biostudies-literature | 2017 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A Bias in the Evaluation of Bias Comparing Randomized Trials with Nonexperimental Studies.

Franklin Jessica M JM Dejene Sara S Huybrechts Krista F KF Wang Shirley V SV Kulldorff Martin M Rothman Kenneth J KJ

Epidemiologic methods 20170422 1

In a recent <i>BMJ</i> article, the authors conducted a meta-analysis to compare estimated treatment effects from randomized trials with those derived from observational studies based on routinely collected data (RCD). They calculated a pooled relative odds ratio (ROR) of 1.31 (95% confidence interval [CI]: 1.03-1.65) and concluded that RCD studies systematically over-estimated protective effects. However, their meta-analysis inverted results for some clinical questions to force all estimates fr ...[more]

PMID: 29204330

Similar Datasets

Project description:BackgroundEvidence based medicine aims to integrate scientific evidence, clinical experience, and patient values and preferences. Individual health care professionals need to appraise the evidence from randomized trials and observational studies when guidelines are not yet available. To date, tools for assessment of bias and terminologies for bias are specific for each study design. Moreover, most tools appeal only to methodological knowledge to detect bias, not to subject matter knowledge, i.e. in-depth medical knowledge about a topic. We propose a unified framework that enables the coherent assessment of bias across designs.MethodsEpidemiologists traditionally distinguish between three types of bias in observational studies: confounding, information bias, and selection bias. These biases result from a common cause, systematic error in the measurement or common effect of the intervention and outcome respectively. We applied this conceptual framework to randomized trials and show how it can be used to identify bias. The three sources of bias were illustrated with graphs that visually represent researchers' assumptions about the relationships between the investigated variables (causal diagrams).ResultsCritical appraisal of evidence started with the definition of the research question in terms of the population of interest, the compared interventions and the main outcome. Next, we used causal diagrams to illustrate how each source of bias can lead to over- or underestimated treatment effects. Then, we discussed how randomization, blinded outcome measurement and intention-to-treat analysis minimize bias in trials. Finally, we identified study aspects that can only be appraised with subject matter knowledge, irrespective of study design.ConclusionsThe unified framework encompassed the three main sources of bias for the effect of an assigned intervention on an outcome. It facilitated the integration of methodological and subject matter knowledge in the assessment of bias. We hope that graphical diagrams will help clarify debate among professionals by reducing misunderstandings based on different terminology for bias.

Project description:BackgroundAlthough oral opioid agonist therapies (OATs), buprenorphine and methadone, are effective first-line treatments, OAT remains largely underutilized due to low retention rates and wide variation across programs. This rapid review therefore sought to summarize the retention rates reported by randomized controlled trials (RCTs) and controlled observational study designs that compared methadone to buprenorphine (or buprenorphine-naloxone).MethodsWe searched four electronic databases (EMBASE, MEDLINE, Cochrane Central Register of Controlled Trials, CINAHL, up to April 2018) for RCTs and controlled observational studies that compared oral fixed-dose methadone to buprenorphine versus methadone (or buprenorphine-naloxone). Data were extracted separately for two different definitions of retention in treatment: (1) length of time retained in the study and (2) presence on the final day of a study. Separate random effects meta-analyses were performed for RCTs and controlled observational studies. Data from controlled observational studies where retention was measured as the length of time retained in the study were not amenable to meta-analysis.ResultsAmong 7603 studies reviewed, 10 RCTs and 3 observational studies met inclusion criteria (n = 5065) and compared fixed-dose oral buprenorphine with methadone. Across studies, the average retention rate was highly variable (RCTs: buprenorphine 20.0-82.5% and methadone 30.7-83.8%; observational studies: buprenorphine 20.2-78.3% and methadone 48.3-74.8%). For time period retained in the study, we observed no significant difference in treatment retention for buprenorphine versus methadone in RCTs (standardized mean difference [SMD] = - 0.07; 95% CI - 0.35-0.21, p = 0.63; quality of evidence: low). For presence on the final study day, we observed no significant difference between buprenorphine and methadone treatment retention in RCTs (risk ratio [RR] = 0.89; 95% CI 0.73-1.08, p = 0.24; quality of evidence: low) and controlled observational studies (RR = 0.75; 95% CI 0.36-1.58, p = 0.45).ConclusionMeta-analysis of existing RCTs suggests retention in oral fixed-dose opioid agonist therapy with methadone appears to be generally equal to buprenorphine (or buprenorphine-naloxone), with wide variation across studies. Similarly, a meta-analysis of three controlled observational studies indicated no difference in treatment retention although there was significant heterogeneity among the included studies. The length of follow-up did not appear to affect the retention rate.Systematic review registrationPROSPERO CRD42018104452 .

Project description:BackgroundPartial factorial trials compare two or more pairs of treatments on overlapping patient groups, randomising some (but not all) patients to more than one comparison. The aims of this research were to compare different methods for conducting and analysing economic evaluations on partial factorial trials and assess the implications of considering factors simultaneously rather than drawing independent conclusions about each comparison.MethodsWe estimated total costs and quality-adjusted life years (QALYs) within 10 years of surgery for 2252 patients in the Knee Arthroplasty Trial who were randomised to one or more comparisons of different surgical types. We compared three analytical methods: an "at-the-margins" analysis including all patients randomised to each comparison (assuming no interaction); an "inside-the-table" analysis that included interactions but focused on those patients randomised to two comparisons; and a Bayesian vetted bootstrap, which used results from patients randomised to one comparison as priors when estimating outcomes for patients randomised to two comparisons. Outcomes comprised incremental costs, QALYs and net benefits.ResultsQualitative interactions were observed for costs, QALYs and net benefits. Bayesian bootstrapping generally produced smaller standard errors than inside-the-table analysis and gave conclusions that were consistent with at-the-margins analysis, while allowing for these interactions. By contrast, inside-the-table gave different conclusions about which intervention had the highest net benefits compared with other analyses.ConclusionsAll analyses of partial factorial trials should explore interactions and assess whether results are sensitive to assumptions about interactions, either as a primary analysis or as a sensitivity analysis. For partial factorial trials closely mirroring routine clinical practice, at-the-margins analysis may provide a reasonable estimate of average costs and benefits for the whole trial population, even in the presence of interactions. However, such conclusions will be misleading if there are large interactions or if the proportion of patients allocated to different treatments differs markedly from what occurs in clinical practice. The Bayesian bootstrap provides an alternative to at-the-margins analysis for analysing clinical or economic endpoints from partial factorial trials, which allows for interactions while making use of the whole sample. The same techniques could be applied to analyses of clinical endpoints.Trial registrationISRCTN, ISRCTN45837371 . Registered on 25 April 2003.

Dataset Information

A Bias in the Evaluation of Bias Comparing Randomized Trials with Nonexperimental Studies.

Publications

A Bias in the Evaluation of Bias Comparing Randomized Trials with Nonexperimental Studies.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets