Dataset Information

Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study.

ABSTRACT:

Objectives

To study the impact of blinding on estimated treatment effects, and their variation between trials; differentiating between blinding of patients, healthcare providers, and observers; detection bias and performance bias; and types of outcome (the MetaBLIND study).

Design

Meta-epidemiological study.

Data source

Cochrane Database of Systematic Reviews (2013-14).

Eligibility criteria for selecting studies

Meta-analyses with both blinded and non-blinded trials on any topic.

Review methods

Blinding status was retrieved from trial publications and authors, and results retrieved automatically from the Cochrane Database of Systematic Reviews. Bayesian hierarchical models estimated the average ratio of odds ratios (ROR), and estimated the increases in heterogeneity between trials, for non-blinded trials (or of unclear status) versus blinded trials. Secondary analyses adjusted for adequacy of concealment of allocation, attrition, and trial size, and explored the association between outcome subjectivity (high, moderate, low) and average bias. An ROR lower than 1 indicated exaggerated effect estimates in trials without blinding.

Results

The study included 142 meta-analyses (1153 trials). The ROR for lack of blinding of patients was 0.91 (95% credible interval 0.61 to 1.34) in 18 meta-analyses with patient reported outcomes, and 0.98 (0.69 to 1.39) in 14 meta-analyses with outcomes reported by blinded observers. The ROR for lack of blinding of healthcare providers was 1.01 (0.84 to 1.19) in 29 meta-analyses with healthcare provider decision outcomes (eg, readmissions), and 0.97 (0.64 to 1.45) in 13 meta-analyses with outcomes reported by blinded patients or observers. The ROR for lack of blinding of observers was 1.01 (0.86 to 1.18) in 46 meta-analyses with subjective observer reported outcomes, with no clear impact of degree of subjectivity. Information was insufficient to determine whether lack of blinding was associated with increased heterogeneity between trials. The ROR for trials not reported as double blind versus those that were double blind was 1.02 (0.90 to 1.13) in 74 meta-analyses.

Conclusion

No evidence was found for an average difference in estimated treatment effect between trials with and without blinded patients, healthcare providers, or outcome assessors. These results could reflect that blinding is less important than often believed or meta-epidemiological study limitations, such as residual confounding or imprecision. At this stage, replication of this study is suggested and blinding should remain a methodological safeguard in trials.

SUBMITTER: Moustgaard H

PROVIDER: S-EPMC7190062 | biostudies-literature | 2020 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study.

Moustgaard Helene H Clayton Gemma L GL Jones Hayley E HE Boutron Isabelle I Jørgensen Lars L Laursen David R T DRT Olsen Mette F MF Paludan-Müller Asger A Ravaud Philippe P Savović Jelena J Sterne Jonathan A C JAC Higgins Julian P T JPT Hróbjartsson Asbjørn A

BMJ (Clinical research ed.) 20200121

<h4>Objectives</h4>To study the impact of blinding on estimated treatment effects, and their variation between trials; differentiating between blinding of patients, healthcare providers, and observers; detection bias and performance bias; and types of outcome (the MetaBLIND study).<h4>Design</h4>Meta-epidemiological study.<h4>Data source</h4>Cochrane Database of Systematic Reviews (2013-14).<h4>Eligibility criteria for selecting studies</h4>Meta-analyses with both blinded and non-blinded trials ...[more]

PMID: 31964641

Similar Datasets

Project description:Objective To examine whether a very large effect (VLE; defined as a relative risk of ?0.2 or ?5) in a randomised trial could be an empirical marker that subsequent trials are unnecessary.Design Meta-epidemiological assessment of existing published data on randomised trials.Data sources Cochrane Database of Systematic Reviews (2010, issue 7) with data on subsequent large trials updated to 2015, issue 12.Eligibility criteria All binary outcome forest plots were selected, which contained an index randomised trial with a VLE that was nominally statistically significant (P<0.05), included a subsequent large randomised trial (?200 events and ?200 non-events) for validation of the effect, assessed a primary outcome of the review, and was not a subgroup or sensitivity analysis.Results Of 3082 reviews yielding 85?002 forest plots, only 44 (0.05%) satisfied the inclusion criteria. Index trials were generally small, with a median sample of 99 (median 14 events). Few index trials were rated at low risk of bias (9 of 44; 20%). The relative risk was closer to the null in the subsequent large trials in 43 of 44 cases. Subsequent large trial data failed to find a statistically significant (P<0.05) effect in the same direction in 19 cases (43%, 95% confidence interval 29% to 58%). Even when the subsequent large trials did find a significant effect in the same direction, the additional primary outcomes in most of these trials would have to be considered before deciding in favour of using the intervention. Subsequent large trial data found a statistically significant effect in the same direction in 19 of 21 cases when the index trial also had a value of P<0.001.Conclusions The frequency of VLEs followed by a large trial is vanishingly small, and where they occur they do not appear to be a reliable marker for a benefit that is reproducible and directly actionable. An empirical rule using a VLE in a randomised controlled trial as a marker that further trials are unnecessary would be neither practical nor useful. Caution should be taken when interpreting small studies with very large treatment effects.

Project description:ObjectiveTo identify factors and assess to what extent they impact the magnitude of the treatment effect of acupuncture therapies across therapeutic areas.Data sourceMedline, Embase, Cochrane Central Register of Controlled Trials, China National Knowledge Infrastructure, Wanfang Database, VIP Database, and China Biology Medicine disc, between 2015 and 2019.Study selectionThe inclusion criteria were trials with a total number of randomised patients larger than 100, at least one patient-important outcome and one of two sets of comparisons.Data analysisThe potential independent variables were identified by reviewing relevant literature and consulting with experts. We conducted meta-regression analyses with standardised mean difference (SMD) as effect estimate for the dependent variable. The analyses included univariable meta-regression and multivariable meta-regression using a three-level robust mixed model.Results1304 effect estimates from 584 acupuncture randomised controlled trials (RCTs) were analysed. The multivariable analyses contained 15 independent variables . In the multivariable analysis, the following produced larger treatment effects of large magnitude (>0.4): quality of life (difference of adjusted SMDs 0.51, 95% CI 0.24 to 0.77), or pain (0.48, 95% CI 0.27 to 0.69), or function (0.41, 95% CI 0.21 to 0.61) vs major events. The following produced larger treatment effects of moderate magnitude (0.2-0.4): single-centred vs multicentred RCTs (0.38, 95% CI 0.10 to 0.66); penetration acupuncture vs non-penetration types of acupuncture (0.34, 95% CI 0.15 to 0.53); non-pain symptoms vs major events (0.32, 95% CI 0.12 to 0.52). The following produced larger treatment effects of small magnitude (<0.2): high vs low frequency treatment sessions (0.19, 95% CI 0.03 to 0.35); pain vs non-pain symptoms (0.16, 95% CI 0.04 to 0.27); unreported vs reported funding (0.12, 95% CI 0 to 0.25).ConclusionPatients, clinicians and policy-makers should consider penetrating over non-penetrating acupuncture and more frequent treatment sessions when feasible and acceptable. When designing future acupuncture RCTs, trialists should consider factors that impact acupuncture treatment effects.

Project description:ObjectiveTo examine whether deviation from the standard intention to treat analysis has an influence on treatment effect estimates of randomised trials.DesignMeta-epidemiological study.Data sourcesMedline, via PubMed, searched between 2006 and 2010; 43 systematic reviews of interventions and 310 randomised trials were included.Eligibility criteria for selecting studiesFrom each year searched, random selection of 5% of intervention reviews with a meta-analysis that included at least one trial that deviated from the standard intention to treat approach. Basic characteristics of the systematic reviews and randomised trials were extracted. Information on the reporting of intention to treat analysis, outcome data, risk of bias items, post-randomisation exclusions, and funding were extracted from each trial. Trials were classified as: ITT (reporting the standard intention to treat approach), mITT (reporting a deviation from the standard approach), and no ITT (reporting no approach). Within each meta-analysis, treatment effects were compared between mITT and ITT trials, and between mITT and no ITT trials. The ratio of odds ratios was calculated (value <1 indicated larger treatment effects in mITT trials than in other trial categories).Results50 meta-analyses and 322 comparisons of randomised trials (from 84 ITT trials, 118 mITT trials, and 108 no ITT trials; 12 trials contributed twice to the analysis) were examined. Compared with ITT trials, mITT trials showed a larger intervention effect (pooled ratio of odds ratios 0.83 (95% confidence interval 0.71 to 0.96), P=0.01; between meta-analyses variance τ(2)=0.13). Adjustments for sample size, type of centre, funding, items of risk of bias, post-randomisation exclusions, and variance of log odds ratio yielded consistent results (0.80 (0.69 to 0.94), P=0.005; τ(2)=0.08). After exclusion of five influential studies, results remained consistent (0.85 (0.75 to 0.98); τ(2)=0.08). The comparison between mITT trials and no ITT trials showed no statistical difference between the two groups (adjusted ratio of odds ratios 0.92 (0.70 to 1.23); τ(2)=0.57).ConclusionsTrials that deviated from the intention to treat analysis showed larger intervention effects than trials that reported the standard approach. Where an intention to treat analysis is impossible to perform, authors should clearly report who is included in the analysis and attempt to perform multiple imputations.

Project description:IntroductionAlthough interest in including non-randomised studies of interventions (NRSIs) in meta-analysis of randomised controlled trials (RCTs) is growing, estimates of effectiveness obtained from NRSIs are vulnerable to greater bias than RCTs. The objectives of this study are to: (1) explore how NRSIs can be integrated into a meta-analysis of RCTs; (2) assess concordance of the evidence from non-randomised and randomised trials and explore factors associated with agreement; and (3) investigate the impact on estimates of pooled bodies of evidence when NRSIs are included.Methods and analysisWe will conduct a systematic survey of 210 systematic reviews that include both RCTs and NRSIs, published from 2017 to 2022. We will randomly select reviews, stratified in a 1:1 ratio by Core vs non-Core clinical journals, as defined by the National Library of Medicine. Teams of paired reviewers will independently determine eligibility and abstract data using standardised, pilot-tested forms. The concordance of the evidence will be assessed by exploring agreement in the relative effect reported by NRSIs and RCT addressing the same clinical question, defined as similarity of the population, intervention/exposure, control and outcomes. We will conduct univariable and multivariable logistic regression analyses to examine the association of prespecified study characteristics with agreement in the estimates between NRSIs and RCTs. We will calculate the ratio of the relative effect estimate from NRSIs over that from RCTs, along with the corresponding 95% CI. We will use a bias-corrected meta-analysis model to investigate the influence on pooled estimates when NRSIs are included in the evidence synthesis.Ethics and disseminationEthics approval is not required. The findings of this study will be disseminated through peer-reviewed publications, conference presentations and condensed summaries for clinicians, health policymakers and guideline developers regarding the design, conduct, analysis, and interpretation of meta-analysis that integrate RCTs and NRSIs.

Project description:BackgroundBlinding aims to minimize biases from what participants and investigators know or believe. Randomized controlled trials, despite being the gold standard to evaluate treatment effect, do not generally assess the success of blinding. We investigated the extent of blinding in back pain trials and the associations between participant guesses and treatment effects.MethodsWe did a review with PubMed/OvidMedline, 2000-2019. Eligibility criteria were back pain trials with data available on treatment effect and participants' guess of treatment. For blinding, blinding index was used as chance-corrected measure of excessive correct guess (0 for random guess). For treatment effects, within- or between-arm effect sizes were used. Analyses of investigators' guess/blinding or by treatment modality were performed exploratorily.ResultsForty trials (3899 participants) were included. Active and sham treatment groups had mean blinding index of 0.26 (95% confidence interval: 0.12, 0.41) and 0.01 (-0.11, 0.14), respectively, meaning 26% of participants in active treatment believed they received active treatment, whereas only 1% in sham believed they received sham treatment, beyond chance, that is, random guess. A greater belief of receiving active treatment was associated with a larger within-arm effect size in both arms, and ideal blinding (namely, "random guess," and "wishful thinking" that signifies both groups believing they received active treatment) showed smaller effect sizes, with correlation of effect size and summary blinding indexes of 0.35 (p = 0.028) for between-arm comparison. We observed uniformly large sham treatment effects for all modalities, and larger correlation for investigator's (un)blinding, 0.53 (p = 0.046).ConclusionParticipants in active treatments in back pain trials guessed treatment identity more correctly, while those in sham treatments tended to display successful blinding. Excessive correct guesses (that could reflect weaker blinding and/or noticeable effects) by participants and investigators demonstrated larger effect sizes. Blinding and sham treatment effects on back pain need due consideration in individual trials and meta-analyses.

Project description:IntroductionEvidence exists that unblinded randomised clinical trials (RCTs) overestimate intervention effects compared with blinded RCTs. It has been suggested that this is less pronounced for objective (ie, not subject to interpretation) outcome measures, including mortality. This may not apply in the intensive care unit (ICU), as most deaths are preceded by decisions to withhold or withdraw treatments. Lack of blinding of physicians in RCTs of ICU interventions may potentially influence the decision towards a higher threshold for discontinuing treatment in patients who receive the investigational treatment and/or a lower threshold for discontinuing treatment in patients who receive the comparator (control). This may have important implications for patients, caregivers, researchers and society. Accordingly, we aim to assess whether lack of blinding affects mortality effect estimates in RCTs of ICU interventions.Methods and analysisWe will conduct a systematic review with meta-analyses and assess the effect of blinding versus no blinding on mortality effect estimates in RCTs of interventions used in adult ICU patients.We will systematically search the Cochrane Library for systematic reviews reporting mortality effect estimates of any intervention used in adult ICU patients which includes at least one RCT with 'low risk of bias' in the bias domains 'blinding of participants and personnel' and/or 'blinding of outcome assessment' and one RCT with 'unclear' or 'high risk of bias' in the same bias domain(s). For each intervention, we will compare summary mortality effect estimates in blinded versus unblinded trials.Ethics and disseminationThis research does not require ethical approval as we will use summary data from trials already approved by relevant ethical institutions. We will report the results in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) statement and submit the final paper to an international peer-reviewed journal.Trial registration numberPROSPERO, registration number: CRD42017056212.

Dataset Information

Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study.

Objectives

Design

Data source

Eligibility criteria for selecting studies

Review methods

Results

Conclusion

Publications

Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets