The case of the misleading funnel plot.
Ontology highlight
ABSTRACT:
SUBMITTER: Lau J
PROVIDER: S-EPMC1570006 | biostudies-literature | 2006 Sep
REPOSITORIES: biostudies-literature
Publications
The case of the misleading funnel plot.
Lau Joseph J Ioannidis John P A JP Terrin Norma N Schmid Christopher H CH Olkin Ingram I
BMJ (Clinical research ed.) 20060901 7568
Similar Datasets
Project description:Meta-analyses are increasingly used for synthesis of evidence from biomedical research, and often include an assessment of publication bias based on visual or analytical detection of asymmetry in funnel plots. We studied the influence of different normalisation approaches, sample size and intervention effects on funnel plot asymmetry, using empirical datasets and illustrative simulations. We found that funnel plots of the Standardized Mean Difference (SMD) plotted against the standard error (SE) are susceptible to distortion, leading to overestimation of the existence and extent of publication bias. Distortion was more severe when the primary studies had a small sample size and when an intervention effect was present. We show that using the Normalised Mean Difference measure as effect size (when possible), or plotting the SMD against a sample size-based precision estimate, are more reliable alternatives. We conclude that funnel plots using the SMD in combination with the SE are unsuitable for publication bias assessments and can lead to false-positive results.
| S-EPMC5621838 | biostudies-literature
Project description:Assessing publication bias is a critical procedure in meta-analyses for rating the synthesized overall evidence. Because statistical tests for publication bias are usually not powerful and only give P values that inform either the presence or absence of the bias, examining the asymmetry of funnel plots has been popular to investigate potentially missing studies and the direction of the bias. Most funnel plots present treatment effects against their standard errors, and the contours depicting studies' significance levels have been used in the plots to distinguish publication bias from other factors (such as heterogeneity and subgroup effects) that may cause the plots' asymmetry. However, treatment effects and their standard errors are frequently associated even if no publication bias exists (eg, both variables depend on the four data cells in a 2 × 2 table for the odds ratio), so standard-error-based funnel plots may lead to false positive conclusions when such association may not be negligible. In addition, the missingness of studies may relate to their sample sizes besides P values (which are partly determined by standard errors); studies with more samples are more likely published. Therefore, funnel plots based on sample sizes can be an alternative tool. However, the contours for standard-error-based funnel plots cannot be directly applied to sample-size-based ones. This article introduces contours for sample-size-based funnel plots of various effect sizes, which may help meta-analysts properly interpret such plots' asymmetry. We provide five examples to illustrate the use of the proposed contours.
| S-EPMC6642847 | biostudies-literature
Project description:BackgroundDespite efforts to assure high methodological standards, systematic reviews may be affected by publication bias. The objective of this study was to evaluate the occurrence of publication bias in a collection of high quality systematic reviews on reproductive health.MethodsSystematic reviews included in the Reproductive Health Library (RHL), issue No 9, were assessed. Funnel plot was used to assess meta-analyses containing 10 or more trials reporting a binary outcome. A funnel plot, the estimated number of missing studies and the adjusted combined effect size were obtained using the "trim and fill method". Meta-analyses results that were not considered to be robust due to a possible publication bias were submitted to a more detailed assessment.ResultsA total of 21 systematic reviews were assessed. The number of trials comprising each one ranged from 10 to 83 (median = 13), totaling 379 trials, whose results have been summarized. None of the reviews had reported any evaluation of publication bias or funnel plot asymmetry. Some degree of asymmetry in funnel plots was observed in 18 of the 21 meta-analyses evaluated (85.7%), with the estimated number of missing studies ranging from 1 to 18 (median = 3). Only for three meta-analyses, the conclusion could not be considered robust due to a possible publication bias.ConclusionAsymmetry is a frequent finding in funnel plots of meta-analyses in reproductive health, but according to the present evaluation, less than 15% of meta-analyses report conclusions that would not be considered robust. Publication bias and other sources of asymmetry in funnel plots should be systematically addressed by reproductive health meta-analysts. Next amendments in Cochrane systematic reviews should include this type of evaluation. Further studies regarding the evolution of effect size and publication bias over time in systematic reviews in reproductive health are needed.
| S-EPMC1855315 | biostudies-literature
Project description:Publication bias refers to a systematic deviation from the truth in the results of a meta-analysis due to the higher likelihood for published studies to be included in meta-analyses than unpublished studies. Publication bias can lead to misleading recommendations for decision and policy making. In this education review, we introduce, explain, and provide solutions to the pervasive misuses and misinterpretations of publication bias that afflict evidence syntheses in sport and exercise medicine, with a focus on the commonly used funnel-plot based methods. Publication bias is more routinely assessed by visually inspecting funnel plot asymmetry, although it has been consistently deemed unreliable, leading to the development of statistical tests to assess publication bias. However, most statistical tests of publication bias (i) cannot rule out alternative explanations for funnel plot asymmetry (e.g., between-study heterogeneity, choice of metric, chance) and (ii) are grossly underpowered, even when using an arbitrary minimum threshold of ten or more studies. We performed a cross-sectional meta-research investigation of how publication bias was assessed in systematic reviews with meta-analyses published in the top two sport and exercise medicine journals throughout 2021. This analysis highlights that publication bias is frequently misused and misinterpreted, even in top tier journals. Because of conceptual and methodological problems when assessing and interpreting publication bias, preventive strategies (e.g., pre-registration, registered reports, disclosing protocol deviations, and reporting all study findings regardless of direction or magnitude) offer the best and most efficient solution to mitigate the misuse and misinterpretation of publication bias. Because true publication bias is very difficult to determine, we recommend that future publications use the term "risk of publication bias".
| S-EPMC10933152 | biostudies-literature
Project description:Small-study effects are a common threat in systematic reviews and may indicate publication bias. Their existence is often verified by visual inspection of the funnel plot. Formal tests to assess the presence of funnel plot asymmetry typically estimate the association between the reported effect size and their standard error, the total sample size, or the inverse of the total sample size. In this paper, we demonstrate that the application of these tests may be less appropriate in meta-analysis of survival data, where censoring influences statistical significance of the hazard ratio. We subsequently propose 2 new tests that are based on the total number of observed events and adopt a multiplicative variance component. We compare the performance of the various funnel plot asymmetry tests in an extensive simulation study where we varied the true hazard ratio (0.5 to 1), the number of published trials (N=10 to 100), the degree of censoring within trials (0% to 90%), and the mechanism leading to participant dropout (noninformative versus informative). Results demonstrate that previous well-known tests for detecting funnel plot asymmetry suffer from low power or excessive type-I error rates in meta-analysis of survival data, particularly when trials are affected by participant dropout. Because our novel test (adopting estimates of the asymptotic precision as study weights) yields reasonable power and maintains appropriate type-I error rates, we recommend its use to evaluate funnel plot asymmetry in meta-analysis of survival data. The use of funnel plot asymmetry tests should, however, be avoided when there are few trials available for any meta-analysis.
| S-EPMC5873397 | biostudies-literature
Project description:Peripheral Primitive NeuroEctodermal Tumors (pPNETs) are rare highly malignant tumors; originating from the neuroectoderm. Although PNETs may arise in various locations (most commonly in the extremities), very few cases have been reported in the pelvis. There is still poor evidence about the management of these tumors in the literature. We present a rare case of pelvic PNET in a 20-year-old male. The patient presented with symptoms mimicking a lumbar disk hernia, which delayed the diagnosis. He was managed with a combination of a debulking procedure, adjuvant chemotherapy, radiotherapy; and has been in remission for 2 years upon follow-up. This case highlights the importance of diagnosing such aggressive tumors as early as possible (as prognosis may vary significantly), and the challenge in the management of PNETs due to poor evidence.
| S-EPMC7251491 | biostudies-literature
Project description:Misleading graphs are a source of misinformation that worry many experts. Especially people with a low graph literacy are thought to be persuaded by graphs that misrepresent the underlying data. But we know little about how people interpret misleading graphs and how these graphs influence their opinions. In this study we focus on the effect of truncating the y-axis for a line chart which exaggerates an upgoing trend. In a randomized controlled trial, we showed participants either a normal or a misleading chart, and we did so in two different contexts. After they had seen the graphs, we asked participants their opinion on the trend and to give an estimation of the increase. Finally we measured their graph literacy. Our results show that context is the only significant factor in opinion-forming; the misleading graph and graph literacy had no effect. None of these factors had a significant impact on estimations for the increase. These results show that people might be less susceptible to misleading graphs than we thought and that context has more impact than a misleading y-axis.
| S-EPMC9200168 | biostudies-literature
Project description:Different ways of calculating mortality ratios during epidemics can yield widely different results, particularly during the COVID-19 pandemic. We formulate both a survival probability model and an associated infection duration-dependent SIR model to define individual- and population-based estimates of dynamic mortality ratios. The key parameters that affect the dynamics of the different mortality estimates are the incubation period and the length of time individuals were infected before confirmation of infection. We stress that none of these ratios are accurately represented by the often misinterpreted case fatality ratio (CFR), the number of deaths to date divided by the total number of infected cases to date. Using available data on the recent SARS-CoV-2 outbreaks and simple assumptions, we estimate and compare the different dynamic mortality ratios and highlight their differences. Informed by our modeling, we propose a more systematic method to determine mortality ratios during epidemic outbreaks and discuss sensitivity to confounding effects and errors in the data.
| S-EPMC7276002 | biostudies-literature
Project description:The pfam04002 annotation describes RadC as a bacterial DNA repair protein. Although the radC gene is expressed specifically during competence for genetic transformation in Streptococcus pneumoniae, we report that radC mutants exhibit normal uptake and processing of transforming DNA. They also display normal sensitivity to DNA-damaging agents, providing no support for the rad epithet.
| S-EPMC2519389 | biostudies-literature
Project description:Online videos have become a prevalent means for people to acquire information. Videos, however, are often polarized, misleading, or contain topics on which people have different, contradictory views. In this work, we introduce natural language explanations to stimulate more deliberate reasoning about videos and raise users' awareness of potentially deceiving or biased information. With these explanations, we aim to support users in actively deciding and reflecting on the usefulness of the videos. We generate the explanations through an end-to-end pipeline that extracts reflection triggers so users receive additional information to the video based on its source, covered topics, communicated emotions, and sentiment. In a between-subjects user study, we examine the effect of showing the explanations for videos on three controversial topics. Besides, we assess the users' alignment with the video's message and how strong their belief is about the topic. Our results indicate that respondents' alignment with the video's message is critical to evaluate the video's usefulness. Overall, the explanations were found to be useful and of high quality. While the explanations do not influence the perceived usefulness of the videos compared to only seeing the video, people with an extreme negative alignment with a video's message perceived it as less useful (with or without explanations) and felt more confident in their assessment. We relate our findings to cognitive dissonance since users seem to be less receptive to explanations when the video's message strongly challenges their beliefs. Given these findings, we provide a set of design implications for explanations grounded in theories on reducing cognitive dissonance in light of raising awareness about online deception.
| S-EPMC8507585 | biostudies-literature