Dataset Information

Estimands in published protocols of randomised trials: urgent improvement needed.

ABSTRACT:

Background

An estimand is a precise description of the treatment effect to be estimated from a trial (the question) and is distinct from the methods of statistical analysis (how the question is to be answered). The potential use of estimands to improve trial research and reporting has been underpinned by the recent publication of the ICH E9(R1) Addendum on the use of estimands in clinical trials in 2019. We set out to assess how well estimands are described in published trial protocols.

Methods

We reviewed 50 trial protocols published in October 2020 in Trials and BMJ Open. For each protocol, we determined whether the estimand for the primary outcome was explicitly stated, not stated but inferable (i.e. could be constructed from the information given), or not inferable.

Results

None of the 50 trials explicitly described the estimand for the primary outcome, and in 74% of trials, it was impossible to infer the estimand from the information included in the protocol. The population attribute of the estimand could not be inferred in 36% of trials, the treatment condition attribute in 20%, the population-level summary measure in 34%, and the handling of intercurrent events in 60% (the strategy for handling non-adherence was not inferable in 32% of protocols, and the strategy for handling mortality was not inferable in 80% of the protocols for which it was applicable). Conversely, the outcome attribute was stated for all trials. In 28% of trials, three or more of the five estimand attributes could not be inferred.

Conclusions

The description of estimands in published trial protocols is poor, and in most trials, it is impossible to understand exactly what treatment effect is being estimated. Given the utility of estimands to improve clinical research and reporting, this urgently needs to change.

SUBMITTER: Kahan BC

PROVIDER: S-EPMC8500821 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:Many randomized controlled trials (RCTs) are biased and difficult to reproduce due to methodological flaws and poor reporting. There is increasing attention for responsible research practices and implementation of reporting guidelines, but whether these efforts have improved the methodological quality of RCTs (e.g., lower risk of bias) is unknown. We, therefore, mapped risk-of-bias trends over time in RCT publications in relation to journal and author characteristics. Meta-information of 176,620 RCTs published between 1966 and 2018 was extracted. The risk-of-bias probability (random sequence generation, allocation concealment, blinding of patients/personnel, and blinding of outcome assessment) was assessed using a risk-of-bias machine learning tool. This tool was simultaneously validated using 63,327 human risk-of-bias assessments obtained from 17,394 RCTs evaluated in the Cochrane Database of Systematic Reviews (CDSR). Moreover, RCT registration and CONSORT Statement reporting were assessed using automated searches. Publication characteristics included the number of authors, journal impact factor (JIF), and medical discipline. The annual number of published RCTs substantially increased over 4 decades, accompanied by increases in authors (5.2 to 7.8) and institutions (2.9 to 4.8). The risk of bias remained present in most RCTs but decreased over time for allocation concealment (63% to 51%), random sequence generation (57% to 36%), and blinding of outcome assessment (58% to 52%). Trial registration (37% to 47%) and the use of the CONSORT Statement (1% to 20%) also rapidly increased. In journals with a higher impact factor (>10), the risk of bias was consistently lower with higher levels of RCT registration and the use of the CONSORT Statement. Automated risk-of-bias predictions had accuracies above 70% for allocation concealment (70.7%), random sequence generation (72.1%), and blinding of patients/personnel (79.8%), but not for blinding of outcome assessment (62.7%). In conclusion, the likelihood of bias in RCTs has generally decreased over the last decades. This optimistic trend may be driven by increased knowledge augmented by mandatory trial registration and more stringent reporting guidelines and journal requirements. Nevertheless, relatively high probabilities of bias remain, particularly in journals with lower impact factors. This emphasizes that further improvement of RCT registration, conduct, and reporting is still urgently needed.

Project description:ImportancePublishing study protocols might reduce research waste because of unclear methods or incomplete reporting; on the other hand, there might be few additional benefits of publishing protocols for registered trials that are never completed or published. No study has investigated the proportion of published protocols associated with published results.ObjectiveTo estimate the proportion of published trial protocols for which there are not associated published results.Design, setting, and participantsThis cross-sectional study used stratified random sampling to identify registered clinical trials with protocols published between January 2011 and August 2022 and indexed in PubMed Central. Ongoing studies and those within 1 year of the primary completion date on ClinicalTrials.gov were excluded. Published results were sought from August 2022 to March 2023 by searching ClinicalTrials.gov, emailing authors, and using an automated tool, as well as through incidental discovery.Main outcomes and measuresThe primary outcome was a weighted estimate of the proportion of registered trials with published protocols that also had published main results. The proportion of trials with unpublished results was estimated using a weighted mean.ResultsFrom 1500 citations that were screened, 308 clinical trial protocols were included, and it was found that 87 trials had not published their main results. Most included trials were investigator-initiated evaluations of nonregulated products. When published, results appeared a mean (SD) of 3.4 (2.0) years after protocol publications. With the use of a weighted mean, an estimated 4754 (95% CI, 4296-5226) eligible clinical trial protocols were published and indexed in PubMed Central between 2011 and 2022. In the weighted analysis, 1708 of those protocols (36%; 95% CI, 31%-41%) were not associated with publication of main results. In a sensitivity analysis excluding protocols published after 2019, an estimated 25% (95% CI, 20%-30%) of 3670 (95% CI, 3310-4032) protocol publications were not associated with publication of main results.Conclusions and relevanceThis cross-sectional study of clinical trial protocols published on PubMed Central between 2011 and 2022 suggests that many protocols were not associated with subsequent publication of results. The overall benefits of publishing study protocols might outweigh the research waste caused by unnecessary protocol publications.

Project description:ObjectivesTo evaluate how often the precise research question being addressed about an intervention (the estimand) is stated or can be determined from reported methods, and to identify what types of questions are being investigated in phase 2-4 randomised trials.DesignSystematic review of the clarity of research questions being investigated in randomised trials in 2020 in six leading general medical journals.Data sourcePubMed search in February 2021.Eligibility criteria for selecting studiesPhase 2-4 randomised trials, with no restrictions on medical conditions or interventions. Cluster randomised, crossover, non-inferiority, and equivalence trials were excluded.Main outcome measuresNumber of trials that stated the precise primary question being addressed about an intervention (ie, the primary estimand), or for which the primary estimand could be determined unambiguously from the reported methods using statistical knowledge. Strategies used to handle post-randomisation events that affect the interpretation or existence of patient outcomes, such as intervention discontinuations or uses of additional drug treatments (known as intercurrent events), and the corresponding types of questions being investigated.Results255 eligible randomised trials were identified. No trials clearly stated all the attributes of the estimand. In 117 (46%) of 255 trials, the primary estimand could be determined from the reported methods. Intercurrent events were reported in 242 (95%) of 255 trials; but the handling of these could only be determined in 125 (49%) of 255 trials. Most trials that provided this information considered the occurrence of intercurrent events as irrelevant in the calculation of the treatment effect and assessed the effect of the intervention regardless (96/125, 77%)-that is, they used a treatment policy strategy. Four (4%) of 99 trials with treatment non-adherence owing to adverse events estimated the treatment effect in a hypothetical setting (ie, the effect as if participants continued treatment despite adverse events), and 19 (79%) of 24 trials where some patients died estimated the treatment effect in a hypothetical setting (ie, the effect as if participants did not die).ConclusionsThe precise research question being investigated in most trials is unclear, mainly because of a lack of clarity on the approach to handling intercurrent events. Clear reporting of estimands is necessary in trial reports so that all stakeholders, including clinicians, patients and policy makers, can make fully informed decisions about medical interventions.Systematic review registrationPROSPERO CRD42021238053.

Project description:IntroductionPublishing protocols of randomized controlled trials (RCT) facilitates a more detailed description of study rational, design, and related ethical and safety issues, which should promote transparency. Little is known about how the practice of publishing protocols developed over time. Therefore, this study describes the worldwide trends in volume and methodological quality of published RCT protocols.MethodsA systematic search was performed in PubMed and EMBASE, identifying RCT protocols published over a decade from 1 September 2001. Data were extracted on quality characteristics of RCT protocols. The primary outcome, methodological quality, was assessed by individual methodological characteristics (adequate generation of allocation, concealment of allocation and intention-to-treat analysis). A comparison was made by publication period (First, September 2001- December 2004; Second, January 2005-May 2008; Third, June 2008-September 2011), geographical region and medical specialty.ResultsThe number of published RCT protocols increased from 69 in the first, to 390 in the third period (p<0.0001). Internal medicine and paediatrics were the most common specialty topics. Whereas most published RCT protocols in the first period originated from North America (n = 30, 44%), in the second and third period this was Europe (respectively, n = 65, 47% and n = 190, 48%, p = 0.02). Quality of RCT protocols was higher in Europe and Australasia, compared to North America (OR = 0.63, CI = 0.40-0.99, p = 0.04). Adequate generation of allocation improved with time (44%, 58%, 67%, p = 0.001), as did concealment of allocation (38%, 53%, 55%, p = 0.03). Surgical protocols had the highest quality among the three specialty topics used in this study (OR = 1.94, CI = 1.09-3.45, p = 0.02).ConclusionPublishing RCT protocols has become popular, with a five-fold increase in the past decade. The quality of published RCT protocols also improved, although variation between geographical regions and across medical specialties was seen. This emphasizes the importance of international standards of comprehensive training in RCT methodology.

Project description:ObjectiveTo investigate to what extent evidence from previous similar trials or systematic reviews was considered before conducting new trials.DesignCohort study of contemporary protocols for trials with ethical approval.MethodsAll protocols for randomised trials approved by the five ethical committees in Denmark between January 2012 and March 2013 were screened for eligibility. Included protocols were read in full to determine whether a systematic search had been conducted and references were checked to evaluate whether trial rationale and design could be challenged for not adequately considering previous evidence. To investigate whether protocols cited relevant trials, we used simple search strategies that could easily be conducted by researchers without experience with literature searches.ResultsSixty-seven protocols were included. Only two (3%) of the protocols explicitly stated to have conducted a literature search and only one (1%) provided information that allowed the search to be replicated. Eleven (16%) of the protocols described trials where we found the information insufficient to judge if the trial was ethically justified, either due to a comparator that was not supported by the presented evidence (six protocols), because they did not present a rationale for conducting the trial (two protocols), or for both reasons (three protocols). For eight (12%) of the protocols, our search identified trials that could have been relevant to cite as justification.ConclusionsWhile most protocols seem to adequately consider existing evidence, a substantial minority of trials might lack a sufficient evidence base. Very few trials seemed to have been based on a literature search which makes it impossible to know whether all relevant previous trials had been considered. Rules for ethical approval should include requirements for systematic literature searches to ensure that research participants are not exposed to sub-optimal treatments or unnecessary harms as well as to reduce research waste.

Project description:In many patients with mild asthma, the low frequency of symptoms and the episodic nature of exacerbations make adherence to regular maintenance treatment difficult. This often leads to over-reliance on short-acting ?2-agonist (SABA) reliever medication and under-treatment of the underlying inflammation, with poor control of asthma symptoms and increased risk of exacerbations. The use of budesonide/formoterol 'as needed' in response to symptoms may represent an alternative treatment option for patients with mild asthma.The SYmbicort Given as needed in Mild Asthma (SYGMA) programme consists of two 52-week, double-blind, randomised, multicentre, parallel-group, phase 3 trials of patients aged 12 years and older with a clinical diagnosis of asthma for at least 6 months, who would qualify for treatment with regular inhaled corticosteroids (ICS). SYGMA1 aims to recruit 3750 patients who will be randomised to placebo twice daily (bid) plus as-needed budesonide/formoterol 160/4.5 ?g, placebo bid plus as-needed terbutaline 0.4 mg, or budesonide 200 ?g bid plus as-needed terbutaline 0.4 mg. The primary objective is to demonstrate the superiority of as-needed budesonide/formoterol over as-needed terbutaline for asthma control, as measured by well-controlled asthma weeks; a secondary objective is to establish the noninferiority of as-needed budesonide/formoterol versus maintenance budesonide plus as-needed terbutaline using the same outcome measure. SYGMA2 aims to recruit 4114 patients who will be randomised to placebo bid plus as-needed budesonide/formoterol 160/4.5 ?g, or budesonide 200 ?g bid plus as-needed terbutaline 0.4 mg. The primary objective is to demonstrate the noninferiority of as-needed budesonide/formoterol over budesonide bid plus as-needed terbutaline as measured by the annualised severe exacerbation rate. In both studies, use of all blinded study inhalers will be recorded electronically using Turbuhaler® Usage Monitors.Given the known risks of mild asthma, and known poor adherence with regular inhaled corticosteroids, the results of the SYGMA programme will help to determine the efficacy and safety of as-needed budesonide/formoterol therapy in mild asthma. Patient recruitment is complete, and completion of the phase 3 studies is planned in 2017.ClinicalTrials.gov identifiers: NCT02149199 SYGMA1 and NCT02224157 SYGMA2. Registered on 16 May 2014 and 19 August 2014, respectively.