Dataset Information

Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study.

ABSTRACT:

Background

In meta-analysis, the presence of funnel plot asymmetry is attributed to publication or other small-study effects, which causes larger effects to be observed in the smaller studies. This issue potentially mean inappropriate conclusions are drawn from a meta-analysis. If meta-analysis is to be used to inform decision-making, a reliable way to adjust pooled estimates for potential funnel plot asymmetry is required.

Methods

A comprehensive simulation study is presented to assess the performance of different adjustment methods including the novel application of several regression-based methods (which are commonly applied to detect publication bias rather than adjust for it) and the popular Trim & Fill algorithm. Meta-analyses with binary outcomes, analysed on the log odds ratio scale, were simulated by considering scenarios with and without i) publication bias and; ii) heterogeneity. Publication bias was induced through two underlying mechanisms assuming the probability of publication depends on i) the study effect size; or ii) the p-value.

Results

The performance of all methods tended to worsen as unexplained heterogeneity increased and the number of studies in the meta-analysis decreased. Applying the methods conditional on an initial test for the presence of funnel plot asymmetry generally provided poorer performance than the unconditional use of the adjustment method. Several of the regression based methods consistently outperformed the Trim & Fill estimators.

Conclusion

Regression-based adjustments for publication bias and other small study effects are easy to conduct and outperformed more established methods over a wide range of simulation scenarios.

SUBMITTER: Moreno SG

PROVIDER: S-EPMC2649158 | biostudies-literature | 2009 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study.

Moreno Santiago G SG Sutton Alex J AJ Ades A E AE Stanley Tom D TD Abrams Keith R KR Peters Jaime L JL Cooper Nicola J NJ

BMC medical research methodology 20090112

<h4>Background</h4>In meta-analysis, the presence of funnel plot asymmetry is attributed to publication or other small-study effects, which causes larger effects to be observed in the smaller studies. This issue potentially mean inappropriate conclusions are drawn from a meta-analysis. If meta-analysis is to be used to inform decision-making, a reliable way to adjust pooled estimates for potential funnel plot asymmetry is required.<h4>Methods</h4>A comprehensive simulation study is presented to ...[more]

PMID: 19138428

Similar Datasets

Project description:Background:Publication bias is a form of scientific misconduct. It threatens the validity of research results and the credibility of science. Although several tests on publication bias exist, no in-depth evaluations are available that examine which test performs best for different research settings. Methods:Four tests on publication bias, Egger's test (FAT), p-uniform, the test of excess significance (TES), as well as the caliper test, were evaluated in a Monte Carlo simulation. Two different types of publication bias and its degree (0%, 50%, 100%) were simulated. The type of publication bias was defined either as file-drawer, meaning the repeated analysis of new datasets, or p-hacking, meaning the inclusion of covariates in order to obtain a significant result. In addition, the underlying effect (β = 0, 0.5, 1, 1.5), effect heterogeneity, the number of observations in the simulated primary studies (N = 100, 500), and the number of observations for the publication bias tests (K = 100, 1,000) were varied. Results:All tests evaluated were able to identify publication bias both in the file-drawer and p-hacking condition. The false positive rates were, with the exception of the 15%- and 20%-caliper test, unbiased. The FAT had the largest statistical power in the file-drawer conditions, whereas under p-hacking the TES was, except under effect heterogeneity, slightly better. The CTs were, however, inferior to the other tests under effect homogeneity and had a decent statistical power only in conditions with 1,000 primary studies. Discussion:The FAT is recommended as a test for publication bias in standard meta-analyses with no or only small effect heterogeneity. If two-sided publication bias is suspected as well as under p-hacking the TES is the first alternative to the FAT. The 5%-caliper test is recommended under conditions of effect heterogeneity and a large number of primary studies, which may be found if publication bias is examined in a discipline-wide setting when primary studies cover different research problems.

Project description:Publication bias is prevalent within the scientific literature. Whilst there are multiple ideas on how to reduce publication bias, only a minority of journals have made substantive changes to address the problem. We aimed to explore the perceived feasibility of strategies to reduce publication bias by gauging opinions of journal editors (n = 73) and other academics/researchers (n = 160) regarding nine methods of publishing and peer-reviewing research: mandatory publication, negative results journals/articles, open reviewing, peer-review training and accreditation, post-publication review, pre-study publication of methodology, published rejection lists, research registration, and two-stage review. Participants completed a questionnaire asking both quantitative (multiple choice or Likert scales) and qualitative (open-ended) questions regarding the barriers to implementing each suggestion, and their strengths and limitations. Participants were asked to rate the nine suggestions, then choose the method they felt was most effective. Mandatory publication was most popularly selected as the 'most effective' method of reducing publication bias for editors (25%), and was the third most popular choice for academics/researchers (14%). The most common selection for academics/researchers was two-stage review (26%), but fewer editors prioritised this (11%). Negative results journals/articles were the second and third most common choices for academics/researchers (21%) and editors (16%), respectively. Editors more commonly chose research registration as 'most effective' (21%), which was favoured by only 6% of academics/researchers. Whilst mandatory publication was generally favoured by respondents, it is infeasible to trial at a journal level. Where suggestions have already been implemented (e.g. negative results journals/articles, trial registration), efforts should be made to objectively assess their efficacy. Two-stage review should be further trialled as its popularity amongst academics/researchers suggests it may be well received, though editors may be less receptive. Several underlying barriers to change also emerged, including scientific culture, impact factors, and researcher training; these should be further explored to reduce publication bias.

Project description:BackgroundSuccessful implementation of medical technologies applied in life-threatening conditions, including extracorporeal membrane oxygenation (ECMO) requires appropriate preparation and training of medical personnel. The pandemic has accelerated the creation of new ECMO centers and has highlighted continuous training in adapting to new pandemic standards. To reach high standards of patients' care, we created the first of its kind, National Education Centre for Artificial Life Support (NEC-ALS) in 40 million inhabitants' country in the Central and Eastern Europe (CEE). The role of the Center is to test and promote the novel or commonly used procedures as well as to develop staff skills on management of patients needing ECMO.MethodIn 2020, nine approved and endorsed by ELSO courses of "Artificial Life Support with ECMO" were organized. Physicians participated in the three-day high-fidelity simulation-based training that was adapted to abide by the social distancing norms of the COVID-19 pandemic. Knowledge as well as crucial cognitive, behavioral and technical aspects (on a 5-point Likert scale) of management on ECMO were assessed before and after course completion. Moreover, the results of training in mechanical chest compression were also evaluated.ResultsThere were 115 participants (60% men) predominantly in the age of 30-40 years. Majority of them (63%) were anesthesiologists or intensivists with more than 5-year clinical experience, but 54% had no previous ECMO experience. There was significant improvement after the course in all cognitive, behavioral, and technical self-assessments. Among aspects of management with ECMO that all increased significantly following the course, the most pronounced was related to the technical one (from approximately 1.0 to more 4.0 points). Knowledge scores significantly increased post-course from 11.4 ± SD to 13 ± SD (out of 15 points). The quality of manual chest compression relatively poor before course improved significantly after training.ConclusionsOur course confirmed that simulation as an educational approach is invaluable not only in training and testing of novel or commonly used procedures, skills upgrading, but also in practicing very rare cases. The implementation of the education program during COVID-19 pandemic may be helpful in founding specialized Advanced Life Support centers and teams including mobile ones. The dedicated R&D Innovation Ecosystem established in the "ECMO for Greater Poland" program, with developed National Education Center can play a crucial role in the knowledge and know-how transfer but future research is needed.

Dataset Information

Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study.

Background

Methods

Results

Conclusion

Publications

Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets