Interpretation of CIs in clinical trials with non-significant results: systematic review and recommendations.
Ontology highlight
ABSTRACT: Objectives
Interpretation of CIs in randomised clinical trials (RCTs) with treatment effects that are not statistically significant can distinguish between results that are 'negative' (the data are not consistent with a clinically meaningful treatment effect) or 'inconclusive' (the data remain consistent with the possibility of a clinically meaningful treatment effect). This interpretation is important to ensure that potentially beneficial treatments are not prematurely abandoned in future research or clinical practice based on invalid conclusions.Design
Systematic review of RCT reports published in 2014 in Annals of Internal Medicine, New England Journal of Medicine, JAMA, JAMA Internal Medicine and The Lancet (n=247).Results
85 of 99 articles with statistically non-significant results reported CIs for the treatment effect. Only 17 of those 99 articles interpreted the CI. Of the 22 articles in which CIs indicated an inconclusive result, only four acknowledged that the study could not rule out a clinically meaningful treatment effect.Conclusions
Interpretation of CIs is important but occurs infrequently in study reports of trials with treatment effects that are not statistically significant. Increased author interpretation of CIs could improve application of RCT results. Reporting recommendations are provided.
SUBMITTER: Gewandter JS
PROVIDER: S-EPMC5726092 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:ObjectivesNon-inferiority (NI) randomized clinical trials (RCTs) commonly evaluate efficacy of new antiretroviral (ARV) drugs in human immunodeficiency virus (HIV) patients. Their reporting and interpretation have not been systematically evaluated. We evaluated the reporting of NI RCTs in HIV patients according to the CONSORT statement and assessed the degree of misinterpretation of RCTs when NI was inconclusive or not established.DesignSystematic review.MethodsPubMed, Web of Science, and Scopus were reviewed until December 2011. Selection and extraction was performed independently by three reviewers.ResultsOf the 42 RCTs (n = 21,919; range 41-3,316) selected, 23 were in ARV-naïve and 19 in ARV-experienced patients. Twenty-seven (64%) RCTs provided information about prior RCTs of the active comparator, and 37 (88%) used 2-sided CIs. Two thirds of trials used a NI margin between 10 and 12%, although only 12 explained the method to determine it. Blinding was used in 9 studies only. The main conclusion was based on both intention-to-treat (ITT) and per protocol (PP) analyses in 5 trials, on PP analysis only in 4 studies, and on ITT only in 31 studies. Eleven of 16 studies with NI inconclusive or not established highlighted NI or equivalence, and distracted readers with positive secondary results.ConclusionsThere is poor reporting and interpretation of NI RCTs performed in HIV patients. Maximizing the reporting of the method of NI margin determination, use of blinding and both ITT and PP analyses, and interpreting negative NI according to actual primary findings will improve the understanding of results and their translation into clinical practice.
| S-EPMC3643946 | biostudies-literature
Project description:ImportanceSpin, the inaccurate reporting of randomized clinical trials (RCTs) with results that are not statistically significant for the primary end point, distorts interpretation of results and leads to misinterpretation. However, the prevalence of spin and related factors in noninferiority cancer RCTs remains unclear.ObjectiveTo examine misleading reporting, or spin, and the associated factors in noninferiority cancer RCTs through a systematic review.Data sourcesA systematic search of the PubMed database was performed for articles published between January 1, 2010, and December 31, 2019, using the Cochrane Highly Sensitive Search Strategy.Study selectionTwo investigators independently selected studies using the inclusion criteria of noninferiority parallel-group RCTs aiming to confirm effects to cancer treatments published between January 1, 2010, and December 31, 2019, reporting results that were not statistically significant for the primary end points.Data extraction and synthesisStandardized data abstraction was used to extract information concerning the trial characteristics and spin based on a prespecified definition. The main investigator extracted the trial characteristics while both readers independently evaluated the spin. The Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guideline was followed.Main outcomes and measuresThe main outcome was spin prevalence in any section of the report. Spin was defined as use of specific reporting strategies, from whatever motive, to highlight that the experimental treatment is beneficial, despite no statistically significant difference for the primary outcome, or to distract the reader from results that are not statistically significant. The associations (prevalence difference and odds ratios [ORs]) between spin and trial characteristics were also evaluated.ResultsThe analysis included 52 of 2752 reports identified in the PubMed search. Spin was identified in 39 reports (75.0%; 95% CI, 61.6%-84.9%), including the abstract (34 reports [65.4%; 95% CI, 51.1%-76.9%]) and the main text (38 reports [73.1%; 95% CI, 59.7%-83.3%]). Univariate analysis found that the spin prevalence was higher in reports with data managers (prevalence difference, 27%; 95% CI, 1.1%-50.3%), reports without funding from for-profit sources (prevalence difference, 31.2%; 95% CI, 4.8%-53.8%), and reports of novel experimental treatments (prevalence difference, 37.5%; 95% CI, 5.8%-64.7%). Multivariable analysis found that novel experimental treatment (OR, 4.64; 95% CI, 0.98-22.02) and funding only from nonprofit sources only (OR, 5.20; 95% CI, 1.21-22.29) were associated with spin.Conclusions and relevanceIn this systematic review, most noninferiority RCTs reporting results that were not statistically significant for the primary end points showed distorted interpretation and inaccurate reporting. The novelty of an experimental treatment and funding only from nonprofit sources were associated with spin.
| S-EPMC8652604 | biostudies-literature
Project description:ObjectiveTo study how composite outcomes, which have combined several components into a single measure, are defined, reported, and interpreted.DesignSystematic review of parallel group randomised clinical trials published in 2008 reporting a binary composite outcome. Two independent observers extracted the data using a standardised data sheet, and two other observers, blinded to the results, selected the most important component.ResultsOf 40 included trials, 29 (73%) were about cardiovascular topics and 24 (60%) were entirely or partly industry funded. Composite outcomes had a median of three components (range 2-9). Death or cardiovascular death was the most important component in 33 trials (83%). Only one trial provided a good rationale for the choice of components. We judged that the components were not of similar importance in 28 trials (70%); in 20 of these, death was combined with hospital admission. Other major problems were change in the definition of the composite outcome between the abstract, methods, and results sections (13 trials); missing, ambiguous, or uninterpretable data (9 trials); and post hoc construction of composite outcomes (4 trials). Only 24 trials (60%) provided reliable estimates for both the composite and its components, and only six trials (15%) had components of similar, or possibly similar, clinical importance and provided reliable estimates. In 11 of 16 trials with a statistically significant composite, the abstract conclusion falsely implied that the effect applied also to the most important component.ConclusionsThe use of composite outcomes in trials is problematic. Components are often unreasonably combined, inconsistently defined, and inadequately reported. These problems will leave many readers confused, often with an exaggerated perception of how well interventions work.
| S-EPMC2923692 | biostudies-literature
Project description:Spin in the reporting of randomized controlled trials, where authors report research in a way that potentially misrepresents results and mislead readers, has been demonstrated in the broader medical literature. We investigated spin in wound care trials with (a) no statistically significant result for the primary outcome and (b) no clearly specified primary outcome.We searched the Cochrane Wounds Group Specialised Register of Trials for randomized controlled trials (RCTs). Eligible studies were: Parallel-group RCTs of interventions for foot, leg or pressure ulcers published in 2004 to 2009 (inclusive) with either a clearly identified primary outcome for which there was a statistically non-significant result (Cohort A) or studies that had no clear primary outcome (Cohort B).We extracted general study details. For both Cohorts A and B we then assessed for the presence of spin. For Cohort A we used a pre-defined process to assess reports for spin. For Cohort B we aimed to assess spin by recording the number of positive treatment effect claims made. We also compared the number of statistically significant and non-significant results reported in the main text and the abstract looking specifically for spin in the form of selective outcome reporting.Of the 71 eligible studies, 28 were eligible for Cohort A; of these, 71% (20/28) contained spin. Cohort B contained 43 studies; of these, 86% (37/43) had abstracts that claimed a favorable treatment claim. Whilst 74% (32/43) of main text results in Cohort B included at least one statistically non-significant result, this was not reflected in the abstract where only 28% contained (12/43) at least one statistically non-significant result.Spin is a frequent phenomenon in reports of RCTs of wound treatments. Studies without statistically significant results for the primary outcome used spin in 71% of cases. Furthermore, 33% (43/132) of reports of wound RCTs did not specify a primary outcome and there was evidence of spin and selective outcome reporting in the abstracts of these. Readers should be wary of only reading the abstracts of reports of RCTs of wound treatments since they are frequently misleading regarding treatment effects.
| S-EPMC3832286 | biostudies-other
Project description:Invasive fungal infections cause excessive morbidity and mortality in premature neonates and severely ill infants.Safety and efficacy outcomes of micafungin were compared between prematurely and non-prematurely born infants <2 years of age. Data were obtained from all completed phase I-III clinical trials with micafungin that had enrolled infants (<2 years of age) that were listed in the Astellas Clinical Study Database. Demographics, adverse events, hepatic function tests and treatment success data were extracted and validated by the Astellas biostatistical group for all micafungin-treated patients, <2 years of age, using the unique patient identifier.One-hundred and sixteen patients included in 9 clinical trials, 48% premature [birth weight (BW) <2500 g and/or gestational age <37 weeks], 52% non-premature, received ? 1 dose of micafungin. Among premature patients, 14.5% were low BW (1500-2499 g), 36.4% very low BW (1000-1499 g) and 49.1% extremely low BW (<1000 g). Ninety patients (78%) completed the studies; 13 [11% (4 premature)] died. Significantly more non-premature than premature patients discontinued treatment (P = 0.003). Treatment-related adverse events were recorded in 23% of patients with no difference between groups. More extremely low BW (n = 4, 15%) and very low BW (n = 8, 40%) infants experienced treatment-related adverse events than low BW (n = 0) and there was no relation to micafungin dose or duration. For a subgroup of 30 patients with invasive candidiasis, treatment success was achieved in 73% in both premature and non-premature groups. Prophylaxis was successful in 4/5 non-premature hematopoietic stem cell transplant patients.Micafungin has a safe profile in premature and non-premature infants with substantial efficacy.
| S-EPMC4196786 | biostudies-other
Project description:The SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) 2013 Statement was developed to provide guidance for inclusion of key methodological components in clinical trial protocols. However, these standards do not include guidance specific to pathology input in clinical trials. This systematic review aims to synthesise existing recommendations specific to pathology practice in clinical trials for implementation in trial protocol design. Articles were identified from database searches and deemed eligible for inclusion if they contained: (1) guidance and/or a checklist, which was (2) pathology-related, with (3) content relevant to clinical trial protocols or could influence a clinical trial protocol design from a pathology perspective and (4) were published in 1996 or later. The quality of individual papers was assessed using the AGREE-GRS (Appraisal of Guidelines for REsearch & Evaluation - Global Rating Scale) tool, and the confidence in cumulative evidence was evaluated using the GRADE-CERQual (Grading of Recommendations Assessment, Development and Evaluation-Confidence in Evidence from Reviews of Qualitative research) approach. Extracted recommendations were synthesised using the best fit framework method, which includes thematic analysis followed by a meta-aggregative approach to synthesis within the framework. Of the 10 184 records screened and 199 full-text articles reviewed, only 40 guidance resources met the eligibility criteria for inclusion. Recommendations extracted from 22 guidance documents were generalisable enough for data synthesis. Seven recommendation statements were synthesised as follows: (1) multidisciplinary collaboration in trial design with early involvement of pathologists, particularly with respect to the use of biospecimens and associated biomarker/analytical assays and in the evaluation of pathology-related parameters; (2) funding and training for personnel undertaking trial work; (3) selection of an accredited laboratory with suitable facilities to undertake scheduled work; (4) quality assurance of pathology-related parameters; (5) transparent reporting of pathology-related parameters; (6) policies regarding informatics and tracking biospecimens across trial sites; and (7) informed consent for specimen collection and retention for future research.
| S-EPMC8073003 | biostudies-literature
Project description:BackgroundIt is unclear how to disseminate the results of randomised controlled trials effectively to health professionals and policymakers to improve treatment, care or prevention through changing policy and practice. This systematic review examined the effectiveness of different methods of dissemination of clinical research results to professional audiences.MethodsWe systematically reviewed the published and grey literature from 2000 to 2022 for studies assessing different approaches for disseminating clinical study results to professional audiences (health professionals, policymakers and guideline developers). Two reviewers assessed potentially relevant full texts for inclusion. We grouped studies by intervention type, synthesising findings using effect direction plots. Outcomes were grouped into out-takes (e.g. awareness, knowledge, understanding), outcomes (e.g. attitude changes) and impact (changes in policy/practice). The quality of evidence was assessed using GRADE.ResultsOur search identified 13,264 unique records, of which 416 full texts were assessed for eligibility. Of 60 studies that were identified as eligible for inclusion, 20 evaluated the effectiveness of interventions to disseminate clinical research results (13 RCTs, 2 observational studies, 3 pre- and post-intervention surveys and 2 cross-sectional surveys). Studies were grouped by intervention: 7 studies that involved face-to-face meetings between the target audience and trained educators were classified as 'outreach interventions'; 5 studies that provided a summary format for systematic review findings (e.g. summary of findings tables) were grouped together. There was high certainty evidence of a small beneficial impact of outreach interventions on health and moderate certainty evidence of impact on practice (mostly prescribing). There was no evidence of impact on policy and very low certainty around benefits on outcomes and out-takes. We found no consistent benefits of summary formats for systematic review results on outcomes or out-takes (moderate quality evidence). Other interventions with less evidence are reported in the Additional Materials.ConclusionsOutreach interventions to disseminate clinical research results can lead to changes in practice and improvements in health. However, these interventions can be resource-intensive. Investment is vital to identify and implement effective and cost-effective ways to disseminate results, so that the potential benefits of trials to patients can be realised.Trial registrationInternational Prospective Register of Systematic Reviews (PROSPERO), CRD42019137364.
| S-EPMC10835915 | biostudies-literature
Project description:BackgroundNAFLD clinical trials have shown suboptimal results, particularly for liver fibrosis, despite the robust preclinical drug development. We aimed to assess the histological response after the experimental treatment versus placebo by carrying out a meta-analysis of NAFLD clinical trials.MethodsAfter a systematic review of NAFLD clinical trials to May 2021, applying strict selection criteria, the following primary outcomes were observed: (a) NASH resolution, with no worsening of fibrosis when available; (b) fibrosis improvement ≥ 1 stage, with no worsening of NAS when available; (c) worsening of NAS; (d) worsening of liver fibrosis ≥ 1 stage, including the progression to cirrhosis on histopathology. Other histological, clinical, and biochemical outcomes were considered secondary endpoints. Heterogeneity was explored by subgroup and sensitivity analyses, and univariable meta-regression.ResultsTwenty-seven randomized clinical trials were included. The pooled efficacy for NASH resolution receiving experimental therapy was 19% (95%CI 15-23; I2 96.2%) compared with placebo 10% (95%CI 7-12; I2 85.8%) (OR 1.66 (95%CI 1.24-2.21); I2 57.8%), while it was 26% (95%CI 22-29); I2 90%)) versus 18% (95%CI 15-21; I2 59%)) for fibrosis improvement (OR 1.34 (95%CI 1.13-1.58); I2 25.4%). For these outcomes, the therapy showed higher efficacy in trials longer than 48 weeks, with < 60% of diabetic population, and when it targeted FXR, PPAR, and antidiabetic mechanisms, and with a NAS < 5 for NASH resolution. Also, NASH (OR 0.57 (95%CI 0.39-0.84); I2 67%) and fibrosis worsening (OR 0.65 (95%CI 0.46-0.92); I2 61.9%) were prevented with the therapy.ConclusionThis meta-analysis provides information about the efficacy of the therapy versus placebo by comparing different and combined trial outcomes such as NASH resolution, fibrosis improvement, and NAS and fibrosis worsening. Changes in the experimental design and selection criteria of the clinical trials might be suitable to increase the efficacy.
| S-EPMC9016009 | biostudies-literature
Project description:As the nontherapeutic use of prescription medications escalates, serious associated consequences have also increased. This makes it essential to estimate misuse, abuse, and related events (MAREs) in the development and postmarketing adverse event surveillance and monitoring of prescription drugs accurately. However, classifications and definitions to describe prescription drug MAREs differ depending on the purpose of the classification system, may apply to single events or ongoing patterns of inappropriate use, and are not standardized or systematically employed, thereby complicating the ability to assess MARE occurrence adequately. In a systematic review of existing prescription drug MARE terminology and definitions from consensus efforts, review articles, and major institutions and agencies, MARE terms were often defined inconsistently or idiosyncratically, or had definitions that overlapped with other MARE terms. The Analgesic, Anesthetic, and Addiction Clinical Trials, Translations, Innovations, Opportunities, and Networks (ACTTION) public-private partnership convened an expert panel to develop mutually exclusive and exhaustive consensus classifications and definitions of MAREs occurring in clinical trials of analgesic medications to increase accuracy and consistency in characterizing their occurrence and prevalence in clinical trials. The proposed ACTTION classifications and definitions are designed as a first step in a system to adjudicate MAREs that occur in analgesic clinical trials and postmarketing adverse event surveillance and monitoring, which can be used in conjunction with other methods of assessing a treatment's abuse potential.
| S-EPMC5460151 | biostudies-literature
Project description:ObjectivesThe concept of non-inferiority is widely adopted in randomized trials comparing transcatheter aortic valve replacement (TAVR) and surgical aortic valve replacement (SAVR). However, uncertainty exists regarding the long-term outcomes of TAVR, and non-inferiority may be difficult to assess. We performed a systematic review and meta-analysis of randomized trials comparing TAVR and SAVR, with a specific emphasis on the non-inferiority margin for 5-year all-cause mortality.MethodsA systematic search was applied to 3 electronic databases. Randomized trials comparing TAVR and SAVR were included. Bayesian methods were implemented to evaluate the posterior probability of non-inferiority at different trial non-inferiority margins under either a vague, Cauchy, or a literature-based prior. Primary outcomes were 5-year actuarial all-cause mortality, and the probability of non-inferiority at various transformed trial non-inferiority margins. Secondary outcomes were long-term survival and 1- and 2-year actuarial survival.ResultsEight trials (n = 8698 patients) were included. Kaplan-Meier-derived 5-year survival was 61.6% (95% CI 59.8-63.5%) for TAVR, and 63.7% (95% CI 61.9-65.6%) for SAVR. Six trials (n = 6370 patients) reported all-cause mortality at 5-year follow-up. Under a vague prior, the posterior median relative risk for all-cause mortality of TAVR was 1.14, compared to SAVR (95% credible interval 1.06-1.22, probability of relative risk <1.00 = 0.01%, I2 = 0%). Similar results in terms of point estimate and uncertainty measures were obtained using frequentist methods. Based on the various trial non-inferiority margins, the results of the analysis suggest that non-inferiority at 5 years is no longer likely.ConclusionsIt is unlikely that TAVR is still non-inferior to SAVR at 5 years in terms of all-cause mortality.
| S-EPMC10684360 | biostudies-literature