Dataset Information

Optimism bias leads to inconclusive results-an empirical study.

ABSTRACT: Optimism bias refers to unwarranted belief in the efficacy of new therapies. We assessed the impact of optimism bias on a proportion of trials that did not answer their research question successfully and explored whether poor accrual or optimism bias is responsible for inconclusive results.Systematic review.Retrospective analysis of a consecutive-series phase III randomized controlled trials (RCTs) performed under the aegis of National Cancer Institute Cooperative groups.Three hundred fifty-nine trials (374 comparisons) enrolling 150,232 patients were analyzed. Seventy percent (262 of 374) of the trials generated conclusive results according to the statistical criteria. Investigators made definitive statements related to the treatment preference in 73% (273 of 374) of studies. Investigators' judgments and statistical inferences were concordant in 75% (279 of 374) of trials. Investigators consistently overestimated their expected treatment effects but to a significantly larger extent for inconclusive trials. The median ratio of expected and observed hazard ratio or odds ratio was 1.34 (range: 0.19-15.40) in conclusive trials compared with 1.86 (range: 1.09-12.00) in inconclusive studies (P<0.0001). Only 17% of the trials had treatment effects that matched original researchers' expectations.Formal statistical inference is sufficient to answer the research question in 75% of RCTs. The answers to the other 25% depend mostly on subjective judgments, which at times are in conflict with statistical inference. Optimism bias significantly contributes to inconclusive results.

SUBMITTER: Djulbegovic B

PROVIDER: S-EPMC3079810 | biostudies-other | 2011 Jun

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

Optimism bias leads to inconclusive results-an empirical study.

Djulbegovic Benjamin B Kumar Ambuj A Magazin Anja A Schroen Anneke T AT Soares Heloisa H Hozo Iztok I Clarke Mike M Sargent Daniel D Schell Michael J MJ

Journal of clinical epidemiology 20101216 6

<h4>Objective</h4>Optimism bias refers to unwarranted belief in the efficacy of new therapies. We assessed the impact of optimism bias on a proportion of trials that did not answer their research question successfully and explored whether poor accrual or optimism bias is responsible for inconclusive results.<h4>Study design</h4>Systematic review.<h4>Setting</h4>Retrospective analysis of a consecutive-series phase III randomized controlled trials (RCTs) performed under the aegis of National Cance ...[more]

PMID: 21163620

Similar Datasets

Project description:BACKGROUND:The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias and outcome reporting bias have been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. METHODOLOGY/PRINCIPAL FINDINGS:In this update, we review and summarise the evidence from cohort studies that have assessed study publication bias or outcome reporting bias in randomised controlled trials. Twenty studies were eligible of which four were newly identified in this update. Only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Fifteen of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40-62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies. CONCLUSIONS:This update does not change the conclusions of the review in which 16 studies were included. Direct empirical evidence for the existence of study publication bias and outcome reporting bias is shown. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.

Project description:BackgroundClinical trials on Traditional Chinese Medicine (TCM) should be registered in a publicly accessible international trial register and report on all outcomes. We systematically assessed and evaluated TCM trials in registries with their subsequent publications.ObjectiveTo describe the characteristics of TCM trials, estimate bias risk and outcome-reporting bias in clinical trials.Data sources and study selectionFifteen trial registries were searched from their inception to July 2012 to identify randomised trials on TCM including Chinese herbs, acupuncture and/or moxibustion, cupping, tuina, qigong, etc.Data extractionWe extracted data including TCM specialty and treated disease/conditions from the registries and searched for subsequent publications in PubMed and Chinese databases. We compared information in the registries of completed trials with any publications focusing on study design, sample size, randomisation, bias risk including reporting bias from the register protocol.Results1096 registered randomised trials were identified evaluating TCM, of which 505 were completed studies (46.1%). The most frequent conditions were pain (13.3%), musculoskeletal (11.7%), nervous (8.7%), digestive (7.1%), circulatory (6.5%), respiratory (6.3%), mental and behavioural disorders (6.2%) and cancer (6.0%). The trial register data identified parallel, phase II/III randomised trials with sample size estimations and blinding, but limited information about randomisation (sequence generation and allocation concealment). Comparing trial registration data of 115 completed trials (22.8%) with their subsequent 136 publications, inconsistencies were identified in one or more of the following: sample size (11%), outcome assessor blinding (37.5%), primary outcomes (29%) and safety (28%) reporting.ConclusionsIncreasing numbers of clinical trials investigating a variety of TCM interventions have been registered in international trial registries. The study design of registered TCM trials has improved in estimating sample size, use of blinding and placebos. However, selective outcome reporting is widespread and similar to conventional medicine and therefore study conclusions should be interpreted with caution.

Project description:To explore whether systematic reviewers selectively include trial effect estimates in meta-analyses when multiple are available, and what impact this may have on meta-analytic effects.Cross-sectional study.We randomly selected systematic reviews of interventions from 2 clinical specialties published between January 2010 and 2012. The first presented meta-analysis of a continuous outcome in each review was selected (index meta-analysis), and all trial effect estimates that were eligible for inclusion in the meta-analysis (eg, from multiple scales or time points) were extracted from trial reports.We calculated a statistic (the Potential Bias Index (PBI)) to quantify and test for evidence of selective inclusion. The PBI ranges from 0 to 1; values above or below 0.5 are suggestive of selective inclusion of effect estimates more or less favourable to the intervention, respectively. The impact of any potential selective inclusion was investigated by comparing the index meta-analytic standardised mean difference (SMD) to the median of a randomly constructed distribution of meta-analytic SMDs (representing the meta-analytic SMD expected when there is no selective inclusion).31 reviews (250 trials) were included. The estimated PBI was 0.57 (95% CI 0.50 to 0.63), suggesting that trial effect estimates that were more favourable to the intervention were included in meta-analyses slightly more often than expected under a process consistent with random selection; however, the 95% CI included the null hypothesis of no selective inclusion. Any potential selective inclusion did not have an important impact on the meta-analytic effects.There was no clear evidence that selective inclusion of trial effect estimates occurred in this sample of meta-analyses. Further research on selective inclusion in other clinical specialties is needed. To enable readers to assess the risk of selective inclusion bias, we recommend that systematic reviewers report the methods used to select effect estimates to include in meta-analyses.

Project description:BackgroundEstimations of causal effects from observational data are subject to various sources of bias. One method for adjusting for the residual biases in the estimation of treatment effects is through the use of negative control outcomes, which are outcomes not believed to be affected by the treatment of interest. The empirical calibration procedure is a technique that uses negative control outcomes to calibrate p-values. An extension of this technique calibrates the coverage of the 95% confidence interval of a treatment effect estimate by using negative control outcomes as well as positive control outcomes, which are outcomes for which the treatment of interest has known effects. Although empirical calibration has been used in several large observational studies, there is no systematic examination of its effect under different bias scenarios.MethodsThe effect of empirical calibration of confidence intervals was analyzed using simulated datasets with known treatment effects. The simulations consisted of binary treatment and binary outcome, with biases resulting from unmeasured confounder, model misspecification, measurement error, and lack of positivity. The performance of the empirical calibration was evaluated by determining the change in the coverage of the confidence interval and the bias in the treatment effect estimate.ResultsEmpirical calibration increased coverage of the 95% confidence interval of the treatment effect estimate under most bias scenarios but was inconsistent in adjusting the bias in the treatment effect estimate. Empirical calibration of confidence intervals was most effective when adjusting for the unmeasured confounding bias. Suitable negative controls had a large impact on the adjustment made by empirical calibration, but small improvements in the coverage of the outcome of interest were also observable when using unsuitable negative controls.ConclusionsThis work adds evidence to the efficacy of empirical calibration of the confidence intervals in observational studies. Calibration of confidence intervals is most effective where there are biases due to unmeasured confounding. Further research is needed on the selection of suitable negative controls.

Dataset Information

Optimism bias leads to inconclusive results-an empirical study.

Publications

Optimism bias leads to inconclusive results-an empirical study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets