Dataset Information

Powering Bias and Clinically Important Treatment Effects in Randomized Trials of Critical Illness.

ABSTRACT:

Objectives

Recurring issues in clinical trial design may bias results toward the null, yielding findings inconclusive for treatment effects. This study evaluated for powering bias among high-impact critical care trials and the associated risk of masking clinically important treatment effects.

Design, setting, and patients

Secondary analysis of multicenter randomized trials of critically ill adults in which mortality was the main endpoint. Trials were eligible for inclusion if published between 2008 and 2018 in leading journals. Analyses evaluated for accuracy of estimated control group mortality, adaptive sample size strategy, plausibility of predicted treatment effect, and results relative to the minimal clinically important difference. The main outcome was the mortality risk difference at the study-specific follow-up interval.

Interventions

None.

Measurements and main results

Of 101 included trials, 12 met statistical significance for their main endpoint, five for increased intervention-associated mortality. Most trials (77.3%) overestimated control group mortality in power calculations (observed minus predicted difference, -6.7% ± 9.8%; p < 0.01). Due to this misestimation of control group mortality, in 14 trials, the intervention would have had to prevent at least half of all deaths to achieve the hypothesized treatment effect. Seven trials prespecified adaptive sample size strategies that might have mitigated this issue. The observed risk difference for mortality fell within 5% of predicted in 20 trials, of which 16 did not reach statistical significance. Half of trials (47.0%) were powered for an absolute risk reduction greater than or equal to 10%, but this effect size was observed in only three trials with a statistically significant treatment benefit. Most trials (67.3%) could not exclude clinically important treatment benefit or harm.

Conclusions

The design of most high-impact critical care trials biased results toward the null by overestimating control group mortality and powering for unrealistic treatment effects. Clinically important treatment effects often cannot be excluded.

SUBMITTER: Abrams D

PROVIDER: S-EPMC7708428 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:IntroductionMortality is the most widely accepted outcome measure in randomized controlled trials of therapies for critically ill adults, but most of these trials fail to show a statistically significant mortality benefit. The reasons for this are unknown.MethodsWe searched five high impact journals (Annals of Internal Medicine, British Medical Journal, JAMA, The Lancet, New England Journal of Medicine) for randomized controlled trials comparing mortality of therapies for critically ill adults over a ten year period. We abstracted data on the statistical design and results of these trials to compare the predicted delta (delta; the effect size of the therapy compared to control expressed as an absolute mortality reduction) to the observed delta to determine if there is a systematic overestimation of predicted delta that might explain the high prevalence of negative results in these trials.ResultsWe found 38 trials meeting our inclusion criteria. Only 5/38 (13.2%) of the trials provided justification for the predicted delta. The mean predicted delta among the 38 trials was 10.1% and the mean observed delta was 1.4% (P < 0.0001), resulting in a delta-gap of 8.7%. In only 2/38 (5.3%) of the trials did the observed delta exceed the predicted delta and only 7/38 (18.4%) of the trials demonstrated statistically significant results in the hypothesized direction; these trials had smaller delta-gaps than the remainder of the trials (delta-gap 0.9% versus 10.5%; P < 0.0001). For trials showing non-significant trends toward benefit greater than 3%, large increases in sample size (380% - 1100%) would be required if repeat trials use the observed delta from the index trial as the predicted delta for a follow-up study.ConclusionsInvestigators of therapies for critical illness systematically overestimate treatment effect size (delta) during the design of randomized controlled trials. This bias, which we refer to as "delta inflation", is a potential reason that these trials have a high rate of negative results."Absence of evidence is not evidence of absence."

Project description:BackgroundCluster randomized trials (CRTs) are trials in which intact groups such as hemodialysis centers or shifts are randomized to treatment or control arms. Pragmatic CRTs have been promoted as a promising trial design for nephrology research yet may also pose ethical challenges. While randomization occurs at the cluster level, the intervention and data collection may vary in a CRT, challenging the identification of research participants. Moreover, when a waiver of patient consent is granted by a research ethics committee, there is an open question as to whether and to what degree patients should be notified about ongoing research or be provided with a debrief regarding the nature and results of the trial upon completion. While empirical and conceptual research exploring ethical issues in pragmatic CRTs has begun to emerge, there has been limited discussion with patients, families, or caregivers of patients undergoing hemodialysis.ObjectiveTo explore with patients and families with experience of hemodialysis research the challenges raised by different approaches to designing pragmatic CRTs in hemodialysis. Specifically, their perceptions of (1) the use of a waiver of consent, (2) notification processes and information provided to participants, and (3) any other concerns about cluster randomized designs in hemodialysis.DesignFocus group and interview discussions of hypothetical clinical trial designs.SettingFocus groups and interviews were conducted in-person or via videoconference or telephone.ParticipantsPatient partners in hemodialysis research, defined as patients with personal experience of dialysis or a family member who had experience supporting a patient receiving hemodialysis, who have been actively involved in discussions to advise a research team on the design, conduct, or implementation of a hemodialysis trial.MethodsParticipants were invited to participate in focus groups or individual discussions that were audio recorded with consent. Recorded interviews were transcribed verbatim prior to analysis. Transcripts were analyzed using a thematic analysis approach.ResultsTwo focus groups, three individual interviews, and one interview involving a patient and family member were conducted with 17 individuals between February 2019 and May 2020. Participants expressed support for approaches that emphasized patient choice. Disclosure of patient-relevant risks and information were key themes. Both consent and notification processes served to generate trust, but bypassing patient choice was perceived as undermining this trust. Participants did not dismiss the option of a waiver of consent. They were, however, more restrictive in their views about when a waiver of consent may be acceptable. Patient partners were skeptical of claims to impracticability based on costs or the time commitments for staff.LimitationsAll participants were from Canada and had been involved in the design or conduct of a trial, limiting the degree to which results may be extrapolated.ConclusionsGiven the preferences of participants to be afforded the opportunity to decide about trial participation, we argue that investigators should thoroughly investigate approaches that allow participants to make an informed choice regarding trial participation. In keeping with the preference for autonomous choice, there remains a need to further explore how consent approaches can be designed to facilitate clinical trial conduct while meeting their ethical requirements. Finally, further work is needed to define the limited circumstances in which waivers of consent are appropriate.

Project description:OBJECTIVE:Hospital-level variations in structure and process may affect clinical outcomes in ICUs. We sought to characterize the organizational structure, processes of care, use of protocols, and standardized outcomes in a large sample of U.S. ICUs. DESIGN:We surveyed 69 ICUs about organization, size, volume, staffing, processes of care, use of protocols, and annual ICU mortality. SETTING:ICUs participating in the United States Critical Illness and Injury Trials Group Critical Illness Outcomes Study. SUBJECTS:Sixty-nine intensivists completed the survey. MEASUREMENTS AND MAIN RESULTS:We characterized structure and process variables across ICUs, investigated relationships between these variables and annual ICU mortality, and adjusted for illness severity using Acute Physiology and Chronic Health Evaluation II. Ninety-four ICU directors were invited to participate in the study and 69 ICUs (73%) were enrolled, of which 25 (36%) were medical, 24 (35%) were surgical, and 20 (29%) were of mixed type, and 64 (93%) were located in teaching hospitals with a median number of five trainees per ICU. Average annual ICU mortality was 10.8%, average Acute Physiology and Chronic Health Evaluation II score was 19.3, 58% were closed units, and 41% had a 24-hour in-house intensivist. In multivariable linear regression adjusted for Acute Physiology and Chronic Health Evaluation II and multiple ICU structure and process factors, annual ICU mortality was lower in surgical ICUs than in medical ICUs (5.6% lower [95% CI, 2.4-8.8%]) or mixed ICUs (4.5% lower [95% CI, 0.4-8.7%]). We also found a lower annual ICU mortality among ICUs that had a daily plan of care review (5.8% lower [95% CI, 1.6-10.0%]) and a lower bed-to-nurse ratio (1.8% lower when the ratio decreased from 2:1 to 1.5:1 [95% CI, 0.25-3.4%]). In contrast, 24-hour intensivist coverage (p = 0.89) and closed ICU status (p = 0.16) were not associated with a lower annual ICU mortality. CONCLUSIONS:In a sample of 69 ICUs, a daily plan of care review and a lower bed-to-nurse ratio were both associated with a lower annual ICU mortality. In contrast to 24-hour intensivist staffing, improvement in team communication is a low-cost, process-targeted intervention strategy that may improve clinical outcomes in ICU patients.

Project description:Annual seasonal influenza epidemics of variable severity result in significant morbidity and mortality in the United States (U.S.) and worldwide. In temperate climate countries, including the U.S., influenza activity peaks during the winter months. Annual influenza vaccination is recommended for all persons in the U.S. aged 6 months and older, and among those at increased risk for influenza-related complications in other parts of the world (e.g. young children, elderly). Observational studies have reported effectiveness of influenza vaccination to reduce the risks of severe disease requiring hospitalization, intensive care unit admission, and death. A diagnosis of influenza should be considered in critically ill patients admitted with complications such as exacerbation of underlying chronic comorbidities, community-acquired pneumonia, and respiratory failure during influenza season. Molecular tests are recommended for influenza testing of respiratory specimens in hospitalized patients. Antigen detection assays are not recommended in critically ill patients because of lower sensitivity; negative results of these tests should not be used to make clinical decisions, and respiratory specimens should be tested for influenza by molecular assays. Because critically ill patients with lower respiratory tract disease may have cleared influenza virus in the upper respiratory tract, but have prolonged influenza viral replication in the lower respiratory tract, an endotracheal aspirate (preferentially) or bronchoalveolar lavage fluid specimen (if collected for other diagnostic purposes) should be tested by molecular assay for detection of influenza viruses.Observational studies have reported that antiviral treatment of critically ill adult influenza patients with a neuraminidase inhibitor is associated with survival benefit. Since earlier initiation of antiviral treatment is associated with the greatest clinical benefit, standard-dose oseltamivir (75 mg twice daily in adults) for enteric administration is recommended as soon as possible as it is well absorbed in critically ill patients. Based upon observational data that suggest harms, adjunctive corticosteroid treatment is currently not recommended for children or adults hospitalized with influenza, including critically ill patients, unless clinically indicated for another reason, such as treatment of asthma or COPD exacerbation, or septic shock. A number of pharmaceutical agents are in development for treatment of severe influenza.