Controversy and debate on credibility ceilings. Paper 1: Fundamental problems with the "credibility ceiling" method for meta-analyses.
Ontology highlight
ABSTRACT: Background and objectives
The "credibility ceiling" method was proposed to conduct sensitivity analysis for unmeasured confounding and other forms of bias in meta-analyses and has been used in umbrella reviews to grade evidence strength. However, we explain that the method has fundamental statistical flaws.Methods
We use statistical reasoning to assess the method's validity, providing intuition for our findings by presenting simple applied examples in which the method yields clearly incorrect conclusions.Results
The credibility ceiling is not a valid bias correction, as we show mathematically and illustrate using examples in which, for example, the method incorrectly "adjusts" the meta-analytic point estimate in the wrong direction. Although the originators describe the method as limiting the credibility of any given observational study to a fixed ceiling, we show why this interpretation in fact bears little relation to what the method actually does.Conclusion
Given the fundamental problems with the credibility ceiling method and its demonstrated potential for misleading conclusions, we recommend against its use.
SUBMITTER: Mathur MB
PROVIDER: S-EPMC8159013 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:In this article, we focus on the potential influence of a scientist's advocacy position on the public's perceived credibility of scientists as a whole. Further, we examine how the scientist's solution position (information only, non-controversial, and controversial) affects the public's perception of the scientist's motivation for sharing information about specific issues (flu, marijuana, climate change, severe weather). Finally, we assess how perceived motivations mediate the relationship between solution position and credibility. Using data from a quota sample of American adults obtained by Qualtrics (n = 2,453), we found that in some conditions advocating for a solution positively predicted credibility, while in one condition, it negatively predicted scientist credibility. We also found that the influence of solution position on perceived credibility was mediated by several motivation perceptions; most notably through perception that the scientist was motivated to: (a) serve the public and (b) persuade the public. Further results and implications are discussed.
| S-EPMC5685625 | biostudies-literature
Project description:[This corrects the article DOI: 10.1371/journal.pone.0187511.].
| S-EPMC6338381 | biostudies-literature
Project description:The identification of ocular tremor in a small cohort of patients with Parkinson's disease (PD) had lay somewhat dormant until the recent report of a pervasive ocular tremor as a universal finding in a large PD cohort that was, however, generally absent from a cohort of age-matched healthy subjects. The reported tremor had frequency characteristics similar to those of PD limb tremor, but the amplitude and frequency of the tremor did not correlate with clinical tremor ratings. Much controversy ensued as to the origin of such a tremor, and specifically as to whether a pervasive ocular tremor was a fundamental feature of PD, or rather a compensatory eye oscillation secondary to a transmitted head tremor, and thus a measure of a normal vestibulo-ocular reflex. In this mini review, we summarize some of the evidence for and against the case for a pervasive ocular tremor in PD and suggest future experiments that may help resolve these conflicting opinions.
| S-EPMC5401892 | biostudies-other
Project description:We quantify social media user engagement with low-credibility online news media sources using a simple and intuitive methodology, that we showcase with an empirical case study of the Twitter debate on immigration in Italy. By assigning the Twitter users an Untrustworthiness (U) score based on how frequently they engage with unreliable media outlets and cross-checking it with a qualitative political annotation of the communities, we show that such information consumption is not equally distributed across the Twitter users. Indeed, we identify clusters characterised by a very high presence of accounts that frequently share content from less reliable news sources. The users with high U are more keen to interact with bot-like accounts that tend to inject more unreliable content into the network and to retweet that content. Thus, our methodology applied to this real-world network provides evidence, in an easy and straightforward way, that there is strong interplay between accounts that display higher bot-like activity and users more focused on news from unreliable sources and that this influences the diffusion of this information across the network.
| S-EPMC9108351 | biostudies-literature
Project description:In this article, we explore a specific controversy about animal experimentation and animal models in the recent history of deep brain stimulation (DBS), and we question its ramifications. DBS development intertwines clinical practice with fundamental research and stands at the crossroads of multiple legacies. We take up the various issues and controversies embedded in this rarely addressed dispute, from a standpoint that combines socio-anthropological and legal aspects. Our starting point is a debate on the role of animal experimentation in the development of DBS between Jarrod Bailey, a researcher promoting the abolition of animal experimentation, and Alim Louis Benabid, Marwan Hariz, and Mahlon DeLong, three key figures in the area of DBS and neuroscience. By clarifying the positions of the different protagonists and retracing the issues raised in these discussions, our objective is to show how this specific debate has extended from its initial space and how it provides an object of study with heuristic scope. We first present this partially polemic discussion about the history of DBS, and its link with a more general debate on the validity and use of animal models and the need for animal experiments. Then, we raise the issue of the relations and interactions between experiments on animals and on humans in the logics of biomedical innovation. The third step is to situate the discussion within the wider framework of opposition towards animal experimentation and the promotion of animal' rights. Finally, combining these interweaved issues, possible implications emerge regarding the future of DBS. We show that behind these several controversies lie the question of translational research and the model of medicine upheld by DBS. We describe how the technology contributes to blurring the lines between research (fundamental, preclinical and clinical research) and care, as well as between humans and animals as substrates and objects of knowledge. The dynamics of DBS future development might then become a point of convergence for neuroscientists and animal rights defenders' interests.
| S-EPMC6548025 | biostudies-literature
Project description:Rockbursts are some of the most severe dynamic disasters in coal mines. In this paper, the discrimination method of the tectonic stress field is proposed by analyzing the modern stress field in China. The tectonic stress field formed by modern tectonic movement guides in situ stress measurements. According to the stress state classification, most rockbursts in coal mines in China are closely associated with tectonic stress. For tectonic stress-driven rockbursts, modern tectonic movement and modern tectonic stress fields must be considered. The stress change and energy transfer caused by tectonic movement affect the geological structure where coal mines are located. Energy accumulation under rockburst conditions is mainly formed by natural geo-dynamic movement and the mining configuration, and energy accumulation is the basis for rockbursts. The application of the geo-dynamic environmental evaluation method to determine the coalfield geo-dynamic process and the influence of modern tectonic movement is proposed. Accordingly, the classification method of rockbursts in coal mines is established. Based on the distribution characteristics of modern tectonic conditions in China, it is revealed that these dynamic disasters follow a "110" distribution. Finally, a "three condition" criteria of rockbursts is proposed: the geo-dynamic environment is a necessary condition for rockbursts, mining disturbance is a sufficient condition for rockbursts, and risk-releasing measures are a condition controlling rockburst risk mitigation.
| S-EPMC9700775 | biostudies-literature
Project description:BackgroundSubgroup analyses are widely used to evaluate the heterogeneity of treatment effects in randomized clinical trials. However, there is a limited investigation of the quality of prespecified and reported subgroup analyses in stroke trials. This study evaluated the credibility of subgroup analyses in stroke trials.Methods and analysisWe searched Medline/PubMed, Embase, the Cochrane Central Register of Controlled Trials, and the Web of Science from inception to 24 March 2021. Three reviewers screened, extracted, and analyzed the data from the publications. Primary publications of stroke trials that reported at least one subgroup effect and had published corresponding study protocols were included. The Instrument for Assessing the Credibility of Effect Modification Analyses (ICEMAN) was used to examine the quality of the subgroup effects reported, with each subgroup effect assigned a credibility rating ranging from very low to high. Subgroup effects with two or more "definitely no" responses received a low credibility rating. The risk of bias was assessed using the Cochrane Risk-of-Bias tool for randomized trials version 2.ResultsSeventy-four articles met the inclusion criteria and reported a combined total of 647 subgroup effects. The median sample size was 1264 (interquartile range (IQR): 380-3876), and the median number of subgroups prespecified in the protocol was 6 (IQR: 2-10). Sixty-one (82%) studies used the univariate test of interaction. Of the total 647 subgroup effects reported in these studies, 319 (49%) were reported in acute stroke trials, while 423 (65%) had low credibility.ConclusionThe quality of subgroup analysis reporting in stroke trials remains poor. More effort is needed to train trialists on the best methods for designing and performing subgroup analyses, and how to report the results.Trial registration numberWe prospectively registered the review with International Prospective Register for Systematic Reviews (registration number: CRD42020223133).
| S-EPMC10676048 | biostudies-literature
Project description:Podcasting is considered to be a communication medium with minimal barriers to entry, making it an attractive method for people, especially from historically underrepresented communities, to tell their own stories. Using the “Breaking Green Ceilings” podcast as a case study, we explore how podcasting serves as an ideal approach to effectively amplify the voices of environmentalists from historically underrepresented communities, specifically Disabled, Queer, Trans, Black, Indigenous, and People of Color. By sharing these stories, the podcast aims to challenge misconceptions and highlight the contributions that environmentalists from historically underrepresented communities make toward environmental sustainability at all levels—grassroots, academia, nonprofit, and government. The “Breaking Green Ceilings” podcast also helps address the lack of diversity in mainstream environmental media and organizations. We use an interdisciplinary, intersectional lens in this article to demonstrate how environmental issues are interconnected with race, religion, socio-economic status, and politics as relates to historically underrepresented communities. We explore four main themes that have emerged on the podcast: environmental justice, structural racism and conservation, traditional ecological knowledge, and access to nature and the outdoors. Finally, strategies are provided to show how the podcast goes beyond the airwaves to build an inclusive community, raising awareness on the issues discussed and moving people to action to undo and unlearn some of the harmful practices and attitudes that have divided the environmental movement for decades. Supplementary Information The online version contains supplementary material available at 10.1007/s13412-021-00723-z.
| S-EPMC8560216 | biostudies-literature
Project description:ObjectiveTo provide considerations for reporting and interpretation that can improve assessment of the credibility of exploratory analyses in aetiologic research.DesignMini-review of the literature and account of exploratory research principles.SettingThis study focuses on a particular type of causal research, namely aetiologic studies, which investigate the causal effect of one or multiple risk factors on a particular health outcome or disease. The mini review included aetiologic research articles published in four epidemiology journals in the first issue of 2021: American Journal of Epidemiology, Epidemiology, European Journal of Epidemiology, and International Journal of Epidemiology, specifically focusing on observational studies of causal risk factors of diseases.Main outcome measuresNumber of exposure-outcome associations reported, grouped by type of analysis (main, sensitivity, and additional).ResultsThe journal articles reported many exposure-outcome associations: a mean number of 33 (range 1-120) exposure-outcome associations for the primary analysis, 30 (0-336) for sensitivity analyses, and 163 (0-1467) for additional analyses. Six considerations were discussed that are important in assessing the credibility of exploratory analyses: research problem, protocol, statistical criteria, interpretation of findings, completeness of reporting, and effect of exploratory findings on future causal research.ConclusionsBased on this mini-review, exploratory analyses in aetiologic research were not always reported properly. Six considerations for reporting of exploratory analyses in aetiologic research were provided to stimulate a discussion about their preferred handling and reporting. Researchers should take responsibility for the results of exploratory analyses by clearly reporting their exploratory nature and specifying which findings should be investigated in future research and how.
| S-EPMC9062703 | biostudies-literature
Project description:Phylostratigraphy, originally designed for gene age estimation by BLAST-based protein homology searches of sequenced genomes, has been widely used for studying patterns and inferring mechanisms of gene origination and evolution. We previously showed by computer simulation that phylostratigraphy underestimates gene age for a non-negligible fraction of genes and that the underestimation is severer for genes with certain properties such as fast evolution and short protein sequences. Consequently, many previously reported age distributions of gene properties may have been methodological artifacts rather than biological realities. Domazet-Lošo and colleagues recently argued that our simulations were flawed and that phylostratigraphic bias does not impact inferences about gene emergence and evolution. Here we discuss conceptual difficulties of phylostratigraphy, identify numerous problems in Domazet-Lošo et al.'s argument, reconfirm phylostratigraphic error using simulations suggested by Domazet-Lošo and colleagues, and demonstrate that a phylostratigraphic trend claimed to be robust to error disappears when genes likely to be error-resistant are analyzed. We conclude that extreme caution is needed in interpreting phylostratigraphic results because of the inherent biases of the method and that reanalysis using genes exhibiting no error in realistic simulations may help reduce spurious findings.
| S-EPMC5501971 | biostudies-other