Dataset Information

Comparison of pooled risk estimates for adverse effects from different observational study designs: methodological overview.

ABSTRACT:

Background

A diverse range of study designs (e.g. case-control or cohort) are used in the evaluation of adverse effects. We aimed to ascertain whether the risk estimates from meta-analyses of case-control studies differ from that of other study designs.

Methods

Searches were carried out in 10 databases in addition to reference checking, contacting experts, and handsearching key journals and conference proceedings. Studies were included where a pooled relative measure of an adverse effect (odds ratio or risk ratio) from case-control studies could be directly compared with the pooled estimate for the same adverse effect arising from other types of observational studies.

Results

We included 82 meta-analyses. Pooled estimates of harm from the different study designs had 95% confidence intervals that overlapped in 78/82 instances (95%). Of the 23 cases of discrepant findings (significant harm identified in meta-analysis of one type of study design, but not with the other study design), 16 (70%) stemmed from significantly elevated pooled estimates from case-control studies. There was associated evidence of funnel plot asymmetry consistent with higher risk estimates from case-control studies. On average, cohort or cross-sectional studies yielded pooled odds ratios 0.94 (95% CI 0.88-1.00) times lower than that from case-control studies.

Interpretation

Empirical evidence from this overview indicates that meta-analysis of case-control studies tend to give slightly higher estimates of harm as compared to meta-analyses of other observational studies. However it is impossible to rule out potential confounding from differences in drug dose, duration and populations when comparing between study designs.

SUBMITTER: Golder S

PROVIDER: S-EPMC3748094 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundThere is considerable debate as to the relative merits of using randomised controlled trial (RCT) data as opposed to observational data in systematic reviews of adverse effects. This meta-analysis of meta-analyses aimed to assess the level of agreement or disagreement in the estimates of harm derived from meta-analysis of RCTs as compared to meta-analysis of observational studies.Methods and findingsSearches were carried out in ten databases in addition to reference checking, contacting experts, citation searches, and hand-searching key journals, conference proceedings, and Web sites. Studies were included where a pooled relative measure of an adverse effect (odds ratio or risk ratio) from RCTs could be directly compared, using the ratio of odds ratios, with the pooled estimate for the same adverse effect arising from observational studies. Nineteen studies, yielding 58 meta-analyses, were identified for inclusion. The pooled ratio of odds ratios of RCTs compared to observational studies was estimated to be 1.03 (95% confidence interval 0.93-1.15). There was less discrepancy with larger studies. The symmetric funnel plot suggests that there is no consistent difference between risk estimates from meta-analysis of RCT data and those from meta-analysis of observational studies. In almost all instances, the estimates of harm from meta-analyses of the different study designs had 95% confidence intervals that overlapped (54/58, 93%). In terms of statistical significance, in nearly two-thirds (37/58, 64%), the results agreed (both studies showing a significant increase or significant decrease or both showing no significant difference). In only one meta-analysis about one adverse effect was there opposing statistical significance.ConclusionsEmpirical evidence from this overview indicates that there is no difference on average in the risk estimate of adverse effects of an intervention derived from meta-analyses of RCTs and meta-analyses of observational studies. This suggests that systematic reviews of adverse effects should not be restricted to specific study types. Please see later in the article for the Editors' Summary.

Project description:BackgroundThere is no consensus on how authors conducting meta-analysis should deal with trial participants with missing outcome data. The objectives of this study are to assess in Cochrane and non-Cochrane systematic reviews: (1) which categories of trial participants the systematic review authors consider as having missing participant data (MPD), (2) how trialists reported on participants with missing outcome data in trials, (3) whether systematic reviewer authors actually dealt with MPD in their meta-analyses of dichotomous outcomes consistently with their reported methods, and (4) the impact of different methods of dealing with MPD on pooled effect estimates in meta-analyses of dichotomous outcomes.Methods/designWe will conduct a methodological study of Cochrane and non-Cochrane systematic reviews. Eligible systematic reviews will include a group-level meta-analysis of a patient-important dichotomous efficacy outcome, with a statistically significant effect estimate. Teams of two reviewers will determine eligibility and subsequently extract information from each eligible systematic review in duplicate and independently, using standardized, pre-piloted forms. The teams will then use a similar process to extract information from the trials included in the meta-analyses of interest. We will assess first which categories of trial participants the systematic reviewers consider as having MPD. Second, we will assess how trialists reported on participants with missing outcome data in trials. Third, we will compare what systematic reviewers report having done, and what they actually did, in dealing with MPD in their meta-analysis. Fourth, we will conduct imputation studies to assess the effects of different methods of dealing with MPD on the pooled effect estimates of meta-analyses. We will specifically calculate for each method (1) the percentage of systematic reviews that lose statistical significance and (2) the mean change of effect estimates across systematic reviews.DiscussionThe impact of different methods of dealing with MPD on pooled effect estimates will help judge the associated risk of bias in systematic reviews. Our findings will inform recommendations regarding what assumptions for MPD should be used to test the robustness of meta-analytical results.