Dataset Information

Empirical Evidence of Study Design Biases in Randomized Trials: Systematic Review of Meta-Epidemiological Studies.

ABSTRACT:

Objective

To synthesise evidence on the average bias and heterogeneity associated with reported methodological features of randomized trials.

Design

Systematic review of meta-epidemiological studies.

Methods

We retrieved eligible studies included in a recent AHRQ-EPC review on this topic (latest search September 2012), and searched Ovid MEDLINE and Ovid EMBASE for studies indexed from Jan 2012-May 2015. Data were extracted by one author and verified by another. We combined estimates of average bias (e.g. ratio of odds ratios (ROR) or difference in standardised mean differences (dSMD)) in meta-analyses using the random-effects model. Analyses were stratified by type of outcome ("mortality" versus "other objective" versus "subjective"). Direction of effect was standardised so that ROR < 1 and dSMD < 0 denotes a larger intervention effect estimate in trials with an inadequate or unclear (versus adequate) characteristic.

Results

We included 24 studies. The available evidence suggests that intervention effect estimates may be exaggerated in trials with inadequate/unclear (versus adequate) sequence generation (ROR 0.93, 95% CI 0.86 to 0.99; 7 studies) and allocation concealment (ROR 0.90, 95% CI 0.84 to 0.97; 7 studies). For these characteristics, the average bias appeared to be larger in trials of subjective outcomes compared with other objective outcomes. Also, intervention effects for subjective outcomes appear to be exaggerated in trials with lack of/unclear blinding of participants (versus blinding) (dSMD -0.37, 95% CI -0.77 to 0.04; 2 studies), lack of/unclear blinding of outcome assessors (ROR 0.64, 95% CI 0.43 to 0.96; 1 study) and lack of/unclear double blinding (ROR 0.77, 95% CI 0.61 to 0.93; 1 study). The influence of other characteristics (e.g. unblinded trial personnel, attrition) is unclear.

Conclusions

Certain characteristics of randomized trials may exaggerate intervention effect estimates. The average bias appears to be greatest in trials of subjective outcomes. More research on several characteristics, particularly attrition and selective reporting, is needed.

SUBMITTER: Page MJ

PROVIDER: S-EPMC4939945 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:Objective To evaluate the frequency, validity, and relevance of statistically significant (P<0.05) sex-treatment interactions in randomized controlled trials in Cochrane meta-analyses.Design Meta-epidemiological study.Data sources Cochrane Database of Systematic Reviews (CDSR) and PubMed.Eligibility criteria for study selection Reviews published in the CDSR with sex-treatment subgroup analyses in the forest plots, using data from randomized controlled trials.Data extraction Information on the study design and sex subgroup data were extracted from reviews and forest plots that met inclusion criteria. For each statistically significant sex-treatment interaction, the potential for biological plausibility and clinical significance was considered.Results Among the 41 reviews with relevant data, there were 109 separate treatment-outcome analyses ("topics"). Among the 109 topics, eight (7%) had a statistically significant sex-treatment interaction. The 109 topics included 311 randomized controlled trials (162 with both sexes, 46 with males only, 103 with females only). Of the 162 individual randomized controlled trials that included both sexes, 15 (9%) had a statistically significant sex-treatment interaction. Of four topics where the first published randomized controlled trial had a statistically significant sex-treatment interaction, no meta-analyses that included other randomized controlled trials retained the statistical significance and no meta-analyses showed statistical significance when data from the first published randomized controlled trial were excluded. Of the eight statistically significant sex-treatment interactions from the overall analyses, only three were discussed by the CDSR reviewers for a potential impact on different clinical management for males compared with females. None of these topics had a sex-treatment interaction that influenced treatment recommendations in recent guidelines. UpToDate, an online physician-authored clinical decision support resource, suggested differential management of men and women for one of these sex-treatment interactions.Conclusion Statistically significant sex-treatment interactions are only slightly more frequent than what would be expected by chance and there is little evidence of subsequent corroboration or clinical relevance of sex-treatment interactions.

Project description:BackgroundRandomized controlled trials (RCTs) and cohort studies are the most common study design types used to assess the treatment effects of medical interventions. To evaluate the agreement of effect estimates between bodies of evidence (BoE) from randomized controlled trials (RCTs) and cohort studies and to identify factors associated with disagreement.MethodsSystematic reviews were published in the 13 medical journals with the highest impact factor identified through a MEDLINE search. BoE-pairs from RCTs and cohort studies with the same medical research question were included. We rated the similarity of PI/ECO (Population, Intervention/Exposure, Comparison, Outcome) between BoE from RCTs and cohort studies. The agreement of effect estimates across BoE was analyzed by pooling ratio of ratios (RoR) for binary outcomes and difference of mean differences for continuous outcomes. We performed subgroup analyses to explore factors associated with disagreements.ResultsOne hundred twenty-nine BoE pairs from 64 systematic reviews were included. PI/ECO-similarity degree was moderate: two BoE pairs were rated as "more or less identical"; 90 were rated as "similar but not identical" and 37 as only "broadly similar". For binary outcomes, the pooled RoR was 1.04 (95% CI 0.97-1.11) with considerable statistical heterogeneity. For continuous outcomes, differences were small. In subgroup analyses, degree of PI/ECO-similarity, type of intervention, and type of outcome, the pooled RoR indicated that on average, differences between both BoE were small. Subgroup analysis by degree of PI/ECO-similarity revealed high statistical heterogeneity and wide prediction intervals across PI/ECO-dissimilar BoE pairs.ConclusionsOn average, the pooled effect estimates between RCTs and cohort studies did not differ. Statistical heterogeneity and wide prediction intervals were mainly driven by PI/ECO-dissimilarities (i.e., clinical heterogeneity) and cohort studies. The potential influence of risk of bias and certainty of the evidence on differences of effect estimates between RCTs and cohort studies needs to be explored in upcoming meta-epidemiological studies.

Project description:BACKGROUND:Publication and related biases (including publication bias, time-lag bias, outcome reporting bias and p-hacking) have been well documented in clinical research, but relatively little is known about their presence and extent in health services research (HSR). This paper aims to systematically review evidence concerning publication and related bias in quantitative HSR. METHODS:Databases including MEDLINE, EMBASE, HMIC, CINAHL, Web of Science, Health Systems Evidence, Cochrane EPOC Review Group and several websites were searched to July 2018. Information was obtained from: (1) Methodological studies that set out to investigate publication and related biases in HSR; (2) Systematic reviews of HSR topics which examined such biases as part of the review process. Relevant information was extracted from included studies by one reviewer and checked by another. Studies were appraised according to commonly accepted scientific principles due to lack of suitable checklists. Data were synthesised narratively. RESULTS:After screening 6155 citations, four methodological studies investigating publication bias in HSR and 184 systematic reviews of HSR topics (including three comparing published with unpublished evidence) were examined. Evidence suggestive of publication bias was reported in some of the methodological studies, but evidence presented was very weak, limited in both quality and scope. Reliable data on outcome reporting bias and p-hacking were scant. HSR systematic reviews in which published literature was compared with unpublished evidence found significant differences in the estimated intervention effects or association in some but not all cases. CONCLUSIONS:Methodological research on publication and related biases in HSR is sparse. Evidence from available literature suggests that such biases may exist in HSR but their scale and impact are difficult to estimate for various reasons discussed in this paper. SYSTEMATIC REVIEW REGISTRATION:PROSPERO 2016 CRD42016052333.

Project description:ObjectivesAdaptive platforms allow for the evaluation of multiple interventions at a lower cost and have been growing in popularity, especially during the COVID-19 pandemic. The objective of this review is to summarize published platform trials, examine specific methodological design features among these studies, and hopefully aid readers in the evaluation and interpretation of platform trial results.MethodsWe performed a systematic review of EMBASE, MEDLINE, Cochrane Central Register of Controlled Trials (CENTRAL), and clinicaltrials.gov from January 2015 to January 2022 for protocols or results of platform trials. Pairs of reviewers, working independently and in duplicate, collected data on trial characteristics of trial registrations, protocols, and publications of platform trials. We reported our results using total numbers and percentages, as well as medians with interquartile range (IQR) when appropriate.ResultsWe identified 15,277 unique search records and screened 14,403 titles and abstracts after duplicates were removed. We identified 98 unique randomized platform trials. Sixteen platform trials were sourced from a systematic review completed in 2019, which included platform trials reported prior to 2015. Most platform trials (n = 67, 68.3%) were registered between 2020 and 2022, coinciding with the COVID-19 pandemic. The included platform trials primarily recruited or plan to recruit patients from North America or Europe, with most subjects being recruited from the United States (n = 39, 39.7%) and the United Kingdom (n = 31, 31.6%). Bayesian methods were used in 28.6% (n = 28) of platform RCTs and frequentist methods in 66.3% (n = 65) of trials, including 1 (1%) that used methods from both paradigms. Out of the twenty-five trials with peer-reviewed publication of results, seven trials used Bayesian methods (28%), and of those, two (8%) used a predefined sample size calculation while the remainder used pre-specified probabilities of futility, harm, or benefit calculated at (pre-specified) intervals to inform decisions about stopping interventions or the entire trial. Seventeen (68%) peer-reviewed publications used frequentist methods. Out of the seven published Bayesian trials, seven (100%) reported thresholds for benefit. The threshold for benefit ranged from 80% to >99%.ConclusionWe identified and summarized key components of platform trials, including the basics of the methodological and statistical considerations. Ultimately, improving standardization and reporting in platform trials require an understanding of the current landscape. We provide the most updated and rigorous review of platform trials to date.

Project description:We aimed to identify and compare empirical data to determine the concordance of diet-disease effect estimates of bodies of evidence (BoE) from randomized controlled trials (RCTs), dietary intake, and biomarkers of dietary intake in cohort studies (CSs). The Cochrane Database of Systematic Reviews and MEDLINE were searched for systematic reviews (SRs) of RCTs and SRs of CSs that investigated both dietary intake and biomarkers of intake published between 1 January 2010 and 31 December 2019. For matched diet-disease associations, the concordance between results from the 3 different BoE was analyzed using 2 definitions: qualitative (e.g., 95% CI within a predefined range) and quantitative (test hypothesis on the z score). Moreover, the differences in the results coming from BoERCTs, BoECSs dietary intake, and BoECSs biomarkers were synthesized to get a pooled ratio of risk ratio (RRR) across all eligible diet-disease associations, so as to compare the 3 BoE. Overall, 49 diet-disease associations derived from 41 SRs were identified and included in the analysis. Twenty-four percent, 10%, and 39% of the diet-disease associations were qualitatively concordant comparing BoERCTs with BoECSs dietary intake, BoERCTs with BoECSs biomarkers, and comparing both BoE from CSs, respectively; 88%, 69%, and 90% of the diet-disease associations were quantitatively concordant comparing BoERCTs with BoECSs dietary intake, BoERCTs with BoECSs biomarkers, and comparing both BoE from CSs, respectively. The pooled RRRs comparing effects from BoERCTs with effects from BoECSs dietary intake were 1.09 (95% CI: 1.06, 1.13) and 1.18 (95% CI: 1.10, 1.25) compared with BoECSs biomarkers. Comparing both BoE from CSs, the difference in the results was also small (RRR: 0.92; 95% CI: 0.88, 0.96). Our findings suggest that BoE from RCTs and CSs are often quantitatively concordant. Prospective SRs in nutrition research should include, whenever possible, BoE from RCTs and CSs on dietary intake and biomarkers of intake to provide the whole picture for an investigated diet-disease association.

Project description:ObjectiveTo evaluate the agreement between diet-disease effect estimates of bodies of evidence from randomised controlled trials and those from cohort studies in nutrition research, and to investigate potential factors for disagreement.DesignMeta-epidemiological study.Data sourcesCochrane Database of Systematic Reviews, and Medline.Review methodsPopulation, intervention or exposure, comparator, outcome (PI/ECO) elements from a body of evidence from cohort studies (BoE(CS)) were matched with corresponding elements of a body of evidence from randomised controlled trials (BoE(RCT)). Pooled ratio of risk ratios or difference of mean differences across all diet-disease outcome pairs were calculated. Subgroup analyses were conducted to explore factors for disagreement. Heterogeneity was assessed through I2 and τ2. Prediction intervals were calculated to assess the range of possible values for the difference in the results between evidence from randomised controlled trials and evidence from cohort studies in future comparisons.Results97 diet-disease outcome pairs (that is, matched BoE(RCT) and BoE(CS)) were identified overall. For binary outcomes, the pooled ratio of risk ratios comparing estimates from BoE(RCT) with BoE(CS) was 1.09 (95% confidence interval 1.04 to 1.14; I2=68%; τ2=0.021; 95% prediction interval 0.81 to 1.46). The prediction interval indicated that the difference could be much more substantial, in either direction. We further explored heterogeneity and found that PI/ECO dissimilarities, especially for the comparisons of dietary supplements in randomised controlled trials and nutrient status in cohort studies, explained most of the differences. When the type of intake or exposure between both types of evidence was identical, the estimates were similar. For continuous outcomes, small differences were observed between randomised controlled trials and cohort studies.ConclusionOn average, the difference in pooled results between estimates from BoE(RCT) and BoE(CS) was small. But wide prediction intervals and some substantial statistical heterogeneity in cohort studies indicate that important differences or potential bias in individual comparisons or studies cannot be excluded. Observed differences were mainly driven by dissimilarities in population, intervention or exposure, comparator, and outcome. These findings could help researchers further understand the integration of such evidence into prospective nutrition evidence syntheses and improve evidence based dietary guidelines.