Dataset Information

Application of the Reverse Fragility Index to Statistically Nonsignificant Randomized Clinical Trial Results.

ABSTRACT: Importance:Interpreting randomized clinical trials (RCTs) and their clinical relevance is challenging when P values are either marginally above or below the P?=?.05 threshold. Objective:To use the concept of reverse fragility index (RFI) to provide a measure of confidence in the neutrality of RCT results when assessed from the clinical perspective. Design, Setting, and Participants:In this cross-sectional study, a MEDLINE search was conducted for RCTs published from January 1, 2013, to December 31, 2018, in JAMA, the New England Journal of Medicine (NEJM), and The Lancet. Eligible studies were phase 3 and 4 trials with 1:1 randomization and statistically nonsignificant binary primary end points. Data analysis was performed from August 1, 2019, to August 31, 2019. Exposures:Single vs multicenter enrollment, total number of events, private vs government funding, placebo vs active control, and time to event vs frequency data. Main Outcomes and Measures:The primary outcome was the median RFI with interquartile range (IQR) at the P?=?.05 threshold. Secondary outcomes were the number of RCTs in which the number of participants lost to follow-up was greater than the RFI; the median RFI with IQR at different P value thresholds; the median reverse fragility quotient with IQR; and the correlation between sample sizes, number of events, and P values of the RCT and RFI. Results:Of the 167 RCTs included, 76 (46%) were published in the NEJM, 50 (30%) in JAMA, and 41 (24%) in The Lancet. The median (IQR) sample size was 970 (470-3427) participants, and the median (IQR) number of events was 251 (105-570). The median (IQR) RFI at the P?=?.05 threshold was 8 (5-13). Fifty-seven RCTs (34%) had an RFI of 5 or lower, and in 68 RCTs (41%) the number of participants lost to follow-up was greater than the RFI. Trials with P values ranging from P?=?.06 to P?=?.10 had a median (IQR) RFI of 3 (2-4). When compared, median (IQR) RFIs were not statistically significant for single-center vs multicenter enrollment (5 [4-13] vs 8 [5-13]; P?=?.41), private vs government-funded studies (9 [5-13] vs 8 [5-13]; P?=?.34), and time-to-event primary end points vs frequency data (9 [5-14] vs 7 [4-13]; P?=?.43). The median (IQR) RFI at the P?=?.01 threshold was 12 (7-19) and at the P?=?.005 threshold was 14 (9-21). Conclusions and Relevance:This cross-sectional study found that a relatively small number of events (median of 8) had to change to move the primary end point of an RCT from nonsignificant to statistically significant. These findings emphasize the nuance required when interpreting trial results that did not meet prespecified significance thresholds.

SUBMITTER: Khan MS

PROVIDER: S-EPMC7407075 | biostudies-literature | 2020 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Application of the Reverse Fragility Index to Statistically Nonsignificant Randomized Clinical Trial Results.

Khan Muhammad Shahzeb MS Fonarow Gregg C GC Friede Tim T Lateef Noman N Khan Safi U SU Anker Stefan D SD Harrell Frank E FE Butler Javed J

JAMA network open 20200803 8

<h4>Importance</h4>Interpreting randomized clinical trials (RCTs) and their clinical relevance is challenging when P values are either marginally above or below the P = .05 threshold.<h4>Objective</h4>To use the concept of reverse fragility index (RFI) to provide a measure of confidence in the neutrality of RCT results when assessed from the clinical perspective.<h4>Design, setting, and participants</h4>In this cross-sectional study, a MEDLINE search was conducted for RCTs published from January ...[more]

PMID: 32756927

Similar Datasets

Project description:ImportanceClinical researchers are obligated to present results objectively and accurately to ensure readers are not misled. In studies in which primary end points are not statistically significant, placing a spin, defined as the manipulation of language to potentially mislead readers from the likely truth of the results, can distract the reader and lead to misinterpretation and misapplication of the findings.ObjectiveTo determine the level and prevalence of spin in published reports of cardiovascular randomized clinical trial (RCT) reports.Data sourceMEDLINE was searched from January 1, 2015, to December 31, 2017, using the Cochrane highly sensitive search strategy.Study selectionInclusion criteria were parallel-group RCTs published from January 1, 2015, to December 31, 2017 in 1 of 6 high-impact journals (New England Journal of Medicine, The Lancet, JAMA, European Heart Journal, Circulation, and Journal of the American College of Cardiology) with primary outcomes that were not statistically significant were included in the analysis.Data extraction and synthesisAnalysis began in August 2018. Data were extracted and verified by 2 independent investigators using a standard collection form. In cases of disagreement between the 2 investigators, a third investigators served as arbitrator.Main outcomes and measuresThe classifications of spin type, severity, and extent were determined according to predefined criteria. Primary clinical outcomes were divided into safety of treatment, efficacy of treatment, and both.ResultsOf 587 studies identified, 93 RCT reports (15.8%) met inclusion criteria. Spin was identified in 53 abstracts (57%; 95% CI, 47%-67%) and 62 main texts of published articles (67%; 95% CI, 57%-75%). Ten reports (11%; 95% CI, 6%-19%) had spin in the title, 35 reports (38%; 95% CI, 28%-48%) had spin in the results section, and 50 reports (54%; 95% CI, 44%-64%) had spin in the conclusions. Among the abstracts, spin was observed in 38 results sections (41%; 95% CI, 31%-51%) and 45 conclusions sections (48%; 95% CI, 38%-58%).Conclusions and relevanceThis study suggests that in reports of cardiovascular RCTs with statistically nonsignificant primary outcomes, investigators often manipulate the language of the report to detract from the neutral primary outcomes. To best apply evidence to patient care, consumers of cardiovascular research should be aware that peer review does not always preclude the use of misleading language in scientific articles.

Project description:BackgroundEvidence-based care relies on robust research. The fragility index (FI) is used to assess the robustness of statistically significant findings in randomized controlled trials (RCTs). While the traditional FI is limited to dichotomous outcomes, a novel tool, the continuous fragility index (CFI), allows for the assessment of the robustness of continuous outcomes.PurposeTo calculate the CFI of statistically significant continuous outcomes in RCTs evaluating interventions for managing anterior shoulder instability (ASI).Study designMeta-analysis; Level of evidence, 2.MethodsA search was conducted across the MEDLINE, Embase, and CENTRAL databases for RCTs assessing management strategies for ASI from inception to October 6, 2022. Studies that reported a statistically significant difference between study groups in ≥1 continuous outcome were included. The CFI was calculated and applied to all available RCTs reporting interventions for ASI. Multivariable linear regression was performed between the CFI and various study characteristics as predictors.ResultsThere were 27 RCTs, with a total of 1846 shoulders, included. The median sample size was 61 shoulders (IQR, 43). The median CFI across 27 RCTs was 8.2 (IQR, 17.2; 95% CI, 3.6-15.4). The median CFI was 7.9 (IQR, 21; 95% CI, 1-22) for 11 studies comparing surgical methods, 22.6 (IQR, 16; 95% CI, 8.2-30.4) for 6 studies comparing nonsurgical reduction interventions, 2.8 for 3 studies comparing immobilization methods, and 2.4 for 3 studies comparing surgical versus nonsurgical interventions. Significantly, 22 of 57 included outcomes (38.6%) from studies with completed follow-up data had a loss to follow-up exceeding their CFI. Multivariable regression demonstrated that there was a statistically significant positive correlation between a trial's sample size and the CFI of its outcomes (r = 0.23 [95% CI, 0.13-0.33]; P < .001).ConclusionMore than a third of continuous outcomes in ASI trials had a CFI less than the reported loss to follow-up. This carries the significant risk of reversing trial findings and should be considered when evaluating available RCT data. We recommend including the FI, CFI, and loss to follow-up in the abstracts of future RCTs.

Project description:ImportanceSpin, the inaccurate reporting of randomized clinical trials (RCTs) with results that are not statistically significant for the primary end point, distorts interpretation of results and leads to misinterpretation. However, the prevalence of spin and related factors in noninferiority cancer RCTs remains unclear.ObjectiveTo examine misleading reporting, or spin, and the associated factors in noninferiority cancer RCTs through a systematic review.Data sourcesA systematic search of the PubMed database was performed for articles published between January 1, 2010, and December 31, 2019, using the Cochrane Highly Sensitive Search Strategy.Study selectionTwo investigators independently selected studies using the inclusion criteria of noninferiority parallel-group RCTs aiming to confirm effects to cancer treatments published between January 1, 2010, and December 31, 2019, reporting results that were not statistically significant for the primary end points.Data extraction and synthesisStandardized data abstraction was used to extract information concerning the trial characteristics and spin based on a prespecified definition. The main investigator extracted the trial characteristics while both readers independently evaluated the spin. The Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guideline was followed.Main outcomes and measuresThe main outcome was spin prevalence in any section of the report. Spin was defined as use of specific reporting strategies, from whatever motive, to highlight that the experimental treatment is beneficial, despite no statistically significant difference for the primary outcome, or to distract the reader from results that are not statistically significant. The associations (prevalence difference and odds ratios [ORs]) between spin and trial characteristics were also evaluated.ResultsThe analysis included 52 of 2752 reports identified in the PubMed search. Spin was identified in 39 reports (75.0%; 95% CI, 61.6%-84.9%), including the abstract (34 reports [65.4%; 95% CI, 51.1%-76.9%]) and the main text (38 reports [73.1%; 95% CI, 59.7%-83.3%]). Univariate analysis found that the spin prevalence was higher in reports with data managers (prevalence difference, 27%; 95% CI, 1.1%-50.3%), reports without funding from for-profit sources (prevalence difference, 31.2%; 95% CI, 4.8%-53.8%), and reports of novel experimental treatments (prevalence difference, 37.5%; 95% CI, 5.8%-64.7%). Multivariable analysis found that novel experimental treatment (OR, 4.64; 95% CI, 0.98-22.02) and funding only from nonprofit sources only (OR, 5.20; 95% CI, 1.21-22.29) were associated with spin.Conclusions and relevanceIn this systematic review, most noninferiority RCTs reporting results that were not statistically significant for the primary end points showed distorted interpretation and inaccurate reporting. The novelty of an experimental treatment and funding only from nonprofit sources were associated with spin.

Dataset Information

Application of the Reverse Fragility Index to Statistically Nonsignificant Randomized Clinical Trial Results.

Publications

Application of the Reverse Fragility Index to Statistically Nonsignificant Randomized Clinical Trial Results.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets