Dataset Information

Are more observational studies being included in Cochrane Reviews?

ABSTRACT:

Background

Increasing the scope of an evidence based approach to areas outside healthcare has renewed the importance of a long-standing discussion on randomised versus observational study designs in evaluating the effectiveness of interventions. We investigate statistically if an increasing recognition of the role of certain nonrandomised studies to support or generalize the results of randomised controlled trials has had an impact on the actual inclusion criteria applied in Cochrane reviews.

Methods

We conduct an on-line search of the Cochrane Database of Systematic Reviews (CDSR) and divide all Cochrane reviews according to their design inclusion criterion: (A) RCTs only or (B) RCTs and (some subset of) observational studies. We test statistically whether a shift in the proportion of category B reviews has occurred by comparing reviews published before 2008 with reviews published during 2008/09.

Results

We find that the proportion of Cochrane reviews choosing a broader inclusion criterion has increased, although by less than two percentage points. The shift is not statistically significant (P = 0.08).

Conclusions

There is currently not sufficient data to support a hypothesis of a significant shift in favour of including observational studies, neither at the aggregate level nor at the level of individual Review Groups within the Cochrane Collaboration.

SUBMITTER: Kongsted HC

PROVIDER: S-EPMC3503546 | biostudies-literature | 2012 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Are more observational studies being included in Cochrane Reviews?

Kongsted Hans Christian HC Konnerup Merete M

BMC research notes 20121016

<h4>Background</h4>Increasing the scope of an evidence based approach to areas outside healthcare has renewed the importance of a long-standing discussion on randomised versus observational study designs in evaluating the effectiveness of interventions. We investigate statistically if an increasing recognition of the role of certain nonrandomised studies to support or generalize the results of randomised controlled trials has had an impact on the actual inclusion criteria applied in Cochrane rev ...[more]

PMID: 23069208

Similar Datasets

Project description:IntroductionThe validity of systematic reviews and meta-analysis depends on methodological quality and unbiased dissemination of trials. Our objective is to evaluate the association of estimates of treatment effects with different bias-related study characteristics in meta-analyses of interventions used for treating pain in osteoarthritis (OA). From the findings, we hope to consolidate guidance on interpreting OA trials in systematic reviews based on empirical evidence from Cochrane reviews.Methods and analysisOnly systematic reviews that compare experimental interventions with sham, placebo or no intervention control will be considered eligible. Bias will be assessed with the risk of bias tool, used according to the Cochrane Collaboration's recommendations. Furthermore, center status, trial size and funding will be assessed. The primary outcome (pain) will be abstracted from the first appearing forest plot for overall pain in the Cochrane review. Treatment effect sizes will be expressed as standardised mean differences (SMDs), where the difference in mean values available from the forest plots is divided by the pooled SD. To empirically assess the risk of bias in treatment benefits, we will perform stratified analyses of the trials from the included meta-analyses and assess the interaction between trial characteristics and treatment effect. A relevant study-level covariate is defined as one that decreases the between-study variance (τ(2), estimated as Tau-squared) as a consequence of inclusion in the mixed effects statistical model.Ethics and disseminationMeta-analyses and randomised controlled trials provide the most reliable basis for treatment of patients with OA, but the actual impact of bias is unclear. This study will systematically examine the methodological quality in OA Cochrane reviews and explore the effect estimates behind possible bias. Since our study does not collect primary data, no formal ethical assessment and informed consent are required.Trial registration numberPROSPERO (CRD42013006924).

Project description:OBJECTIVES:Assess the frequency and reasons for disagreements in risk of bias assessments for randomised controlled trials (RCTs) included in more than one Cochrane review. DESIGN:Research on research study, using cross-sectional design. DATA SOURCES:2796 Cochrane reviews published between March 2011 and September 2014. DATA SELECTION:RCTs included in more than one review. DATA EXTRACTION:Risk of bias assessment and support for judgement for five key risk of bias items. DATA SYNTHESIS:For each item, we compared risk of bias assessment made in each review and calculated proportion of agreement. Two reviewers independently analysed 50% of all disagreements by comparing support for each judgement with information from study report to evaluate whether disagreements were related to a difference in information (eg, contact the study author) or a difference in interpretation (same support for judgement but different interpretation). They also identified main reasons for different interpretation. RESULTS:1604 RCTs were included in more than one review. Proportion of agreement ranged from 57% (770/1348 trials) for incomplete outcome data to 81% for random sequence generation (1193/1466). Most common source of disagreement was difference in interpretation of the same information, ranging from 65% (88/136) for random sequence generation to 90% (56/62) for blinding of participants and personnel. Access to different information explained 32/136 (24%) disagreements for random sequence generation and 38/205 (19%) for allocation concealment. Disagreements related to difference in interpretation were frequently related to incomplete or unclear reporting in the study report (83% of disagreements related to different interpretation for random sequence generation). CONCLUSIONS:Risk of bias judgements of RCTs included in more than one Cochrane review differed substantially. Most disagreements were related to a difference in interpretation of an incomplete or unclear description in the study report. A clearer guidance on common causes of incomplete information may improve agreement.

Project description:BackgroundClinical decisions are made based on Cochrane reviews, but the implementation of results of evidence syntheses such as Cochrane reviews is problematic if the evidence is not prepared consistently. All systematic reviews should assess the risk of bias (RoB) in included studies, and in Cochrane reviews, this is done by using Cochrane RoB tool. However, the tool is not necessarily applied according to the instructions. In this study, we aimed to determine the types of bias and their corresponding judgements noted in the 'other bias' domain of Cochrane RoB tool.MethodsWe analyzed Cochrane reviews that included randomized controlled trials (RCTs) and extracted data regarding 'other bias' from the RoB table and accompanying support for the judgment. We categorized different types of other bias.ResultsWe analyzed 768 Cochrane reviews that included 11,369 RCTs. There were 602 (78%) Cochrane reviews that had 'other bias' domain in the RoB tool, and they included a total of 7811 RCTs. In the RoB table of 337 Cochrane reviews for at least one of the included trials it was indicated that no other bias was found and supporting explanations were inconsistently judged as low, unclear or high RoB. In the 524 Cochrane reviews that described various sources of other bias, there were 5762 individual types of explanations which we categorized into 31 groups. The judgments of the same supporting explanations were highly inconsistent. We found numerous other inconsistencies in reporting of sources of other bias in Cochrane reviews.ConclusionCochrane authors mention a wide range of sources of other bias in the RoB tool and they inconsistently judge the same supporting explanations. Inconsistency in appraising risk of other bias hinders reliability and comparability of Cochrane systematic reviews. Discrepant and erroneous judgments of bias in evidence synthesis may hinder implementation of evidence in routine clinical practice and reduce confidence in otherwise trustworthy sources of information. These results can help authors of Cochrane and non-Cochrane reviews to gain insight into various sources of other bias that can be found in trials, and also to help them avoid mistakes that were recognized in published Cochrane reviews.

Project description:BackgroundMost meta-analyses include data from one or more small studies that, individually, do not have power to detect an intervention effect. The relative influence of adequately powered and underpowered studies in published meta-analyses has not previously been explored. We examine the distribution of power available in studies within meta-analyses published in Cochrane reviews, and investigate the impact of underpowered studies on meta-analysis results.Methods and findingsFor 14,886 meta-analyses of binary outcomes from 1,991 Cochrane reviews, we calculated power per study within each meta-analysis. We defined adequate power as ?50% power to detect a 30% relative risk reduction. In a subset of 1,107 meta-analyses including 5 or more studies with at least two adequately powered and at least one underpowered, results were compared with and without underpowered studies. In 10,492 (70%) of 14,886 meta-analyses, all included studies were underpowered; only 2,588 (17%) included at least two adequately powered studies. 34% of the meta-analyses themselves were adequately powered. The median of summary relative risks was 0.75 across all meta-analyses (inter-quartile range 0.55 to 0.89). In the subset examined, odds ratios in underpowered studies were 15% lower (95% CI 11% to 18%, P<0.0001) than in adequately powered studies, in meta-analyses of controlled pharmacological trials; and 12% lower (95% CI 7% to 17%, P<0.0001) in meta-analyses of controlled non-pharmacological trials. The standard error of the intervention effect increased by a median of 11% (inter-quartile range -1% to 35%) when underpowered studies were omitted; and between-study heterogeneity tended to decrease.ConclusionsWhen at least two adequately powered studies are available in meta-analyses reported by Cochrane reviews, underpowered studies often contribute little information, and could be left out if a rapid review of the evidence is required. However, underpowered studies made up the entirety of the evidence in most Cochrane reviews.

Project description:BACKGROUND:Nudge interventions are those that seek to modify the social and physical environment to enhance capacity for subconscious behaviours that align with the intrinsic values of an individual, without actively restricting options. This study sought to describe the application and effects of nudge strategies on clinician implementation of health-related guidelines, policies and practices within studies included in relevant Cochrane systematic reviews. METHODS:As there is varied terminology used to describe nudge, this study examined studies within relevant systematic reviews. A two-stage screening process was undertaken where, firstly, all systematic reviews published in the Cochrane Library between 2016 and 2018 were screened to identify reviews that included quantitative studies to improve implementation of guidelines among healthcare providers. Secondly, individual studies within relevant systematic reviews were included if they were (i) randomised controlled trials (RCTs), (ii) included a nudge strategy in at least one intervention arm, and (iii) explicitly aimed to improve clinician implementation behaviour. We categorised nudge strategies into priming, salience and affect, default, incentives, commitment and ego, and norms and messenger based on the Mindspace framework. SYNTHESIS:The number and percentage of trials using each nudge strategy was calculated. Due to substantial heterogeneity, we did not undertake a meta-analysis. Instead, we calculated within-study point estimates and 95% confidence intervals, and used a vote-counting approach to explore effects. RESULTS:Seven reviews including 42 trials reporting on 57 outcomes were included. The most common nudge strategy was priming (69%), then norms and messenger (40%). Of the 57 outcomes, 86% had an effect on clinician behaviour in the hypothesised direction, and 53% of those were statistically significant. For continuous outcomes, the median effect size was 0.39 (0.22, 0.45), while for dichotomous outcomes the median Odds Ratio was 1.62 (1.13, 2.76). CONCLUSIONS:This review of 42 RCTs included in Cochrane systematic reviews found that the impact of nudge strategies on clinician behaviour was at least comparable to other interventions targeting implementation of evidence-based guidelines. While uncertainty remains, the review provides justification for ongoing investigation of the evaluation and application of nudge interventions to support provider behaviour change. TRIAL REGISTRATION:This review was not prospectively registered.

Dataset Information

Are more observational studies being included in Cochrane Reviews?

Background

Methods

Results

Conclusions

Publications

Are more observational studies being included in Cochrane Reviews?

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets