Dataset Information

Delta inflation: a bias in the design of randomized controlled trials in critical care medicine.

ABSTRACT:

Introduction

Mortality is the most widely accepted outcome measure in randomized controlled trials of therapies for critically ill adults, but most of these trials fail to show a statistically significant mortality benefit. The reasons for this are unknown.

Methods

We searched five high impact journals (Annals of Internal Medicine, British Medical Journal, JAMA, The Lancet, New England Journal of Medicine) for randomized controlled trials comparing mortality of therapies for critically ill adults over a ten year period. We abstracted data on the statistical design and results of these trials to compare the predicted delta (delta; the effect size of the therapy compared to control expressed as an absolute mortality reduction) to the observed delta to determine if there is a systematic overestimation of predicted delta that might explain the high prevalence of negative results in these trials.

Results

We found 38 trials meeting our inclusion criteria. Only 5/38 (13.2%) of the trials provided justification for the predicted delta. The mean predicted delta among the 38 trials was 10.1% and the mean observed delta was 1.4% (P < 0.0001), resulting in a delta-gap of 8.7%. In only 2/38 (5.3%) of the trials did the observed delta exceed the predicted delta and only 7/38 (18.4%) of the trials demonstrated statistically significant results in the hypothesized direction; these trials had smaller delta-gaps than the remainder of the trials (delta-gap 0.9% versus 10.5%; P < 0.0001). For trials showing non-significant trends toward benefit greater than 3%, large increases in sample size (380% - 1100%) would be required if repeat trials use the observed delta from the index trial as the predicted delta for a follow-up study.

Conclusions

Investigators of therapies for critical illness systematically overestimate treatment effect size (delta) during the design of randomized controlled trials. This bias, which we refer to as "delta inflation", is a potential reason that these trials have a high rate of negative results."Absence of evidence is not evidence of absence."

SUBMITTER: Aberegg SK

PROVIDER: S-EPMC2887200 | biostudies-literature | 2010

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Delta inflation: a bias in the design of randomized controlled trials in critical care medicine.

Aberegg Scott K SK Richards D Roxanne DR O'Brien James M JM

Critical care (London, England) 20100429 2

<h4>Introduction</h4>Mortality is the most widely accepted outcome measure in randomized controlled trials of therapies for critically ill adults, but most of these trials fail to show a statistically significant mortality benefit. The reasons for this are unknown.<h4>Methods</h4>We searched five high impact journals (Annals of Internal Medicine, British Medical Journal, JAMA, The Lancet, New England Journal of Medicine) for randomized controlled trials comparing mortality of therapies for criti ...[more]

PMID: 20429873

Similar Datasets

Project description:IntroductionEvidence from randomized controlled trials (RCTs) is required to guide treatment of critically ill children, but the number of RCTs available is limited and the publications are often difficult to find. The objectives of this review were to systematically identify RCTs in pediatric critical care and describe their methods and reporting.MethodsWe searched MEDLINE, EMBASE, LILACS and CENTRAL (from inception to April 16, 2013) and reference lists of included RCTs and relevant systematic reviews. We included published RCTs administering any intervention to children in a pediatric ICU. We excluded trials conducted in neonatal ICUs, those enrolling exclusively preterm infants, and individual patient crossover trials. Pairs of reviewers independently screened studies for eligibility, assessed risk of bias, and abstracted data. Discrepancies were resolved by consensus.ResultsWe included 248 RCTs: 45 (18%) were multicentered and 14 (6%) were multinational. Trials most frequently enrolled both medical and surgical patients (43%) but postoperative cardiac surgery was the single largest population studied (19%). The most frequently evaluated types of intervention were medications (63%), devices (11%) and nutrition (8%). Laboratory or physiological measurements were the most frequent type of primary outcomes (18%). Half of these trials (50%) reported blinding. Of the 107 (43%) trials that reported an a priori sample size, 34 (32%) were stopped early. The median number of children randomized per trial was 49 and ranged from 6 to 4,947. The frequency of RCT publications increased at a mean rate of 0.7 RCTs per year (P<0.001) from 1 to 20 trials per year.ConclusionsThis scoping review identified the available RCTs in pediatric critical care and made them accessible to clinicians and researchers (http://epicc.mcmaster.ca). Most focused on medications and intermediate or surrogate outcomes, were single-centered and were conducted in North America and Western Europe. The results of this review underscore the need for trials with rigorous methodology, appropriate outcome measures, and improved quality of reporting to ensure that high quality evidence exists to support clinical decision-making in this vulnerable population.

Project description:Conventional randomized placebo-controlled study design assumes the absence of drug*placebo interaction. We hypothesized the presence of such an interaction and that conventionally estimated drug effect might be biased. The objectives of the study were to determine the drug*placebo interaction effect (main) and compare conventionally estimated and interaction model-estimated drug effects (secondary).We used a hybrid of balanced placebo and randomized placebo-controlled designs. Four hundred eighty healthy volunteers were randomized to three groups. The first received hydroxyzine (25 mg) described as hydroxyzine or placebo, the second received placebo described as hydroxyzine or placebo, and the third received hydroxyzine and placebo described as unknown; each in a randomized crossover design. Seven participants failed to crossover. Group assignment was concealed from participants and study coordinators. Coordinators were blinded to group and intervention assignment. Participants and coordinators were deceived as to study objectives. Main outcomes were mean area-under-the-curve of drowsiness (therapeutic outcome) and mouth-dryness (adverse outcome), self-reported on 100 mm visual analog scale over 7 h. Drug, placebo, placebo?+?interaction, and total effects were estimated using analysis of covariance by comparing received hydroxyzine/told placebo to received placebo/told placebo, received placebo/told hydroxyzine to received placebo/told placebo, received hydroxyzine/told hydroxyzine to received hydroxyzine/told placebo, and received hydroxyzine/told hydroxyzine to received placebo/told placebo, respectively. Drug effect was also conventionally estimated in the third group.Mean (SD) age was 31.4 (6.6) years, 65% were males. There was significant difference between placebo?+?interaction effect and placebo effect for both drowsiness and mouth-dryness with a mean difference (95% confidence interval) of 35.1 (5.6 to 64.6) and 23.8 (2.4 to 45.2) mm*hr, respectively. Total effect was larger than the sum of drug and placebo effects for drowsiness (139.7 (109.8 to 169.6) vs. 99.1 (68.2 to 130.0) mm*hr) and mouth-dryness (63.6 (41.1 to 86.1) vs. 34.7 (11.1 to 58.4) mm*hr). Conventionally estimated drug effect was larger than interaction model-estimated drug effect for drowsiness (69.2 (45.5 to 92.8) vs. (58.3 (31.6 to 85.0) mm*hr) and mouth-dryness (19.9 (5.3 to 34.5) vs. 9.5 (-9.2 to 28.1) mm*hr).There is significant and important drug*placebo interaction effect that may bias conventionally estimated drug effect.ClinicalTrial.gov identifier: NCT01501591 (registered December 25, 2011).

Project description:ObjectivesRecurring issues in clinical trial design may bias results toward the null, yielding findings inconclusive for treatment effects. This study evaluated for powering bias among high-impact critical care trials and the associated risk of masking clinically important treatment effects.Design, setting, and patientsSecondary analysis of multicenter randomized trials of critically ill adults in which mortality was the main endpoint. Trials were eligible for inclusion if published between 2008 and 2018 in leading journals. Analyses evaluated for accuracy of estimated control group mortality, adaptive sample size strategy, plausibility of predicted treatment effect, and results relative to the minimal clinically important difference. The main outcome was the mortality risk difference at the study-specific follow-up interval.InterventionsNone.Measurements and main resultsOf 101 included trials, 12 met statistical significance for their main endpoint, five for increased intervention-associated mortality. Most trials (77.3%) overestimated control group mortality in power calculations (observed minus predicted difference, -6.7% ± 9.8%; p < 0.01). Due to this misestimation of control group mortality, in 14 trials, the intervention would have had to prevent at least half of all deaths to achieve the hypothesized treatment effect. Seven trials prespecified adaptive sample size strategies that might have mitigated this issue. The observed risk difference for mortality fell within 5% of predicted in 20 trials, of which 16 did not reach statistical significance. Half of trials (47.0%) were powered for an absolute risk reduction greater than or equal to 10%, but this effect size was observed in only three trials with a statistically significant treatment benefit. Most trials (67.3%) could not exclude clinically important treatment benefit or harm.ConclusionsThe design of most high-impact critical care trials biased results toward the null by overestimating control group mortality and powering for unrealistic treatment effects. Clinically important treatment effects often cannot be excluded.

Dataset Information

Delta inflation: a bias in the design of randomized controlled trials in critical care medicine.

Introduction

Methods

Results

Conclusions

Publications

Delta inflation: a bias in the design of randomized controlled trials in critical care medicine.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure