The use and quality of reporting of propensity score methods in multiple sclerosis literature: A review.
Ontology highlight
ABSTRACT: Background
Propensity score (PS) analyses are increasingly used in multiple sclerosis (MS) research, largely owing to the greater availability of large observational cohorts and registry databases.Objective
To evaluate the use and quality of reporting of PS methods in the recent MS literature.Methods
We searched the PubMed database for articles published between January 2013 and July 2019. We restricted the search to comparative effectiveness studies of two disease-modifying therapies.Results
Thirty-nine studies were included in the review, with most studies (62%) published within the past 3 years. All studies reported the list of covariates used for the PS model, but only 21% of studies mentioned how those covariates were selected. Most studies used PS matching (72%), followed by PS adjustment (18%), weighting (15%), and stratification (3%), with some overlap. Most studies using matching or weighting reported checking post-PS covariate imbalance (91%), although about 45% of these studies relied on p values from various statistical tests. Only 25% of studies using matching reported calculating robust standard errors for the PS analyses.Conclusions
The quality of reporting of PS methods in the MS literature is sub-optimal in general, and in some cases, inappropriate methods are used.
SUBMITTER: Karim ME
PROVIDER: S-EPMC9260477 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BackgroundWith many disease-modifying therapies currently approved for the management of multiple sclerosis, there is a growing need to evaluate the comparative effectiveness and safety of those therapies from real-world data sources. Propensity score methods have recently gained popularity in multiple sclerosis research to generate real-world evidence. Recent evidence suggests, however, that the conduct and reporting of propensity score analyses are often suboptimal in multiple sclerosis studies.ObjectivesTo provide practical guidance to clinicians and researchers on the use of propensity score methods within the context of multiple sclerosis research.MethodsWe summarize recommendations on the use of propensity score matching and weighting based on the current methodological literature, and provide examples of good practice.ResultsStep-by-step recommendations are presented, starting with covariate selection and propensity score estimation, followed by guidance on the assessment of covariate balance and implementation of propensity score matching and weighting. Finally, we focus on treatment effect estimation and sensitivity analyses.ConclusionThis comprehensive set of recommendations highlights key elements that require careful attention when using propensity score methods.
| S-EPMC9260471 | biostudies-literature
Project description:BackgroundPropensity score methods are becoming increasingly popular in infectious disease medicine to correct for confounding in observational studies. However, applying and reporting propensity score techniques correctly requires substantial knowledge of these methods. The quality of conduct and reporting of propensity score methods in studies investigating the effectiveness of antimicrobial therapy is yet undetermined.MethodsA systematic review was performed to provide an overview of studies (2005-2020) on the effectiveness of antimicrobial therapy that used propensity score methods. A quality assessment tool and a standardized quality score were developed to evaluate a subset of studies in which antibacterial therapy was investigated in detail. The scale of this standardized score ranges between 0 (lowest quality) and 100 (excellent).ResultsA total of 437 studies were included. The absolute number of studies that investigated the effectiveness of antimicrobial therapy and that used propensity score methods increased 15-fold between the periods 2005-2009 and 2015-2019. Propensity score matching was the most frequently applied technique (65%), followed by propensity score-adjusted multivariable regression (25%). A subset of 108 studies was evaluated in detail. The median standardized quality score per year ranged between 53 and 61 (overall range: 33-88) and remained constant over the years.ConclusionsThe quality of conduct and reporting of propensity score methods in research on the effectiveness of antimicrobial therapy needs substantial improvement. The quality assessment instrument that was developed in this study may serve to help investigators improve the conduct and reporting of propensity score methods.
| S-EPMC8962720 | biostudies-literature
Project description:BackgroundPropensity score matching (PSM) is a statistical technique used to reduce bias in observational studies by controlling for measured confounders. Given its complexity and popularity, it is imperative that researchers comprehensively report their methodologies to ensure accurate interpretation and reproducibility.PurposeThis systematic review sought to define how often PSM has been used in recent orthopedic research and to describe how such studies reported their methods. Secondary aims included analyzing study reproducibility, bibliometric factors associated with reproducibility, and associations between methodology and the reporting of statistically significant results.MethodsPubMed and Embase databases were queried for studies containing "propensity score" and "match*" published in 20 orthopedic journals prior to 2020. All studies meeting inclusion criteria were used for trend analysis. Articles published between 2017 and 2019 were used for analysis of reporting quality and reproducibility.ResultsIn all, 261 studies were included for trend analysis, and 162 studies underwent full-text review. The proportion of orthopedic studies using PSM significantly increased over time. Seventy-one (41%) articles did not provide justification for covariate selection. The majority of studies illustrated covariate balance through P values. We found that 19% of the studies were fully reproducible. Most studies failed to report the use of replacement (67.3%) or independent or paired statistical methods (34.0%). Studies reporting standardized mean differences to illustrate covariate balance were less likely to report statistically significant results.ConclusionDespite the increased use of PSM in orthopedic research, observational studies employing PSM have largely failed to adequately report their methodology.
| S-EPMC9527541 | biostudies-literature
Project description:BackgroundPropensity score techniques can reduce confounding and bias in observational studies. Such analyses are able to measure and balance pre-determined covariates between treated and untreated groups, leading to results that can approximate those generated by randomized prospective studies when such trials are not feasible. The most commonly used propensity score -based analytic technique is propensity score matching (PSM). Although PSM popularity has continued to increase in medical literature, improper methodology or methodological reporting may lead to biased interpretation of treatment effects or limited scientific reproducibility and generalizability. In this study, we aim to characterize and assess the quality of PSM methodology reporting in high-impact otolaryngologic literature.MethodsPubMed and Embase based systematic review of the top 20 journals in otolaryngology, as measured by impact factor from the Journal Citations Reports from 2012 to 2018, for articles using PSM analysis throughout their publication history. Eligible articles were reviewed and assessed for quality and reporting of PSM methodology.ResultsOur search yielded 101 studies, of which 92 were eligible for final analysis and review. The proportion of studies utilizing PSM increased significantly over time (p < 0.001). Nearly all studies (96.7%, n = 89) specified the covariates used to calculate propensity scores. Covariate balance was illustrated in 67.4% (n = 62) of studies, most frequently through p-values. A minority (17.4%, n = 16) of studies were found to be fully reproducible according to previously established criteria.ConclusionsWhile PSM analysis is becoming increasingly prevalent in otolaryngologic literature, the quality of PSM methodology reporting can be improved. We provide potential recommendations for authors regarding optimal reporting for analyses using PSM.
| S-EPMC7774981 | biostudies-literature
Project description:Continuous treatments propensity scoring remains understudied as the majority of methods are focused on the binary treatment setting. Current propensity score methods for continuous treatments typically rely on weighting in order to produce causal estimates. It has been shown that in some continuous treatment settings, weighting methods can result in worse covariate balance than had no adjustments been made to the data. Furthermore, weighting is not always stable, and resultant estimates may be unreliable due to extreme weights. These issues motivate the current development of novel propensity score stratification techniques to be used with continuous treatments. Specifically, the generalized propensity score cumulative distribution function (GPS-CDF) and the nonparametric GPS-CDF approaches are introduced. Empirical CDFs are used to stratify subjects based on pretreatment confounders in order to produce causal estimates. A detailed simulation study shows superiority of these new stratification methods based on the empirical CDF, when compared with standard weighting techniques. The proposed methods are applied to the "Mexican-American Tobacco use in Children" study to determine the causal relationship between continuous exposure to smoking imagery in movies, and smoking behavior among Mexican-American adolescents. These promising results provide investigators with new options for implementing continuous treatment propensity scoring.
| S-EPMC8629138 | biostudies-literature
Project description:: Propensity score (PS) analysis is increasingly being used in observational studies, especially in some cancer studies where random assignment is not feasible. This systematic review evaluates the use and reporting quality of PS analysis in oncology studies. : We searched PubMed to identify the use of PS methods in cancer studies (CS) and cancer surgical studies (CSS) in major medical, cancer, and surgical journals over time and critically evaluated 33 CS published in top medical and cancer journals in 2014 and 2015 and 306 CSS published up to November 26, 2015, without earlier date limits. The quality of reporting in PS analysis was evaluated. It was also compared over time and among journals with differing impact factors. All statistical tests were two-sided. More than 50% of the publications with PS analysis from the past decade occurred within the past two years. Of the studies critically evaluated, a considerable proportion did not clearly provide the variables used to estimate PS (CS 12.1%, CSS 8.8%), incorrectly included non baseline variables (CS 3.4%, CSS 9.3%), neglected the comparison of baseline characteristics (CS 21.9%, CSS 15.6%), or did not report the matching algorithm utilized (CS 19.0%, CSS 36.1%). In CSS, the reporting of the matching algorithm improved in 2014 and 2015 ( P = .04), and the reporting of variables used to estimate PS was better in top surgery journals ( P = .008). However, there were no statistically significant differences for the inclusion of non baseline variables and reporting of comparability of baseline characteristics. The use of PS in cancer studies has dramatically increased recently, but there is substantial room for improvement in the quality of reporting even in top journals. Herein we have proposed reporting guidelines for PS analyses that are broadly applicable to different areas of medical research that will allow better evaluation and comparison across studies applying this approach.
| S-EPMC6059208 | biostudies-literature
Project description:IntroductionSeveral published systematic reviews have examined the potential associations between e-cigarette use and cigarette smoking, but their methodological and/or reporting quality have not yet been assessed. This systematic quality review followed Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines and AMSTAR (A MeaSurement Tool to Assess systematic Reviews) 2 to evaluate the quality of systematic reviews investigating potential associations between e-cigarette use and cigarette smoking.Materials and methodsPubMed/MEDLINE, Embase, and PsycINFO were searched from 01 January 2007 to 24 June 2020. Methodological quality was assessed using AMSTAR 2, and reporting quality was assessed using PRISMA guidelines.ResultsOf 331 potentially relevant systematic reviews, 20 met predefined inclusion criteria. Most reviews (n = 15; 75%) reported on e-cigarette use and cigarette smoking cessation, while three reported on e-cigarette use and cigarette smoking initiation (15%); and two reported on cigarette smoking initiation and cessation (10%). According to AMSTAR 2 guidelines, 18 of the 20 reviews (90%) were "critically low" in overall confidence of the results, while two were ranked "low." Additionally, reporting quality varied across the reviews, with only 60% reporting at least half of the PRISMA items.DiscussionMethodological limitations were identified across reviews examining potential associations between e-cigarette use and cigarette smoking behaviors, indicating that findings from these reviews should be interpreted with caution.ConclusionsFuture systematic reviews in this field should strive to adhere to AMSTAR 2 and PRISMA guidelines, to provide high quality syntheses of the available data with transparent and complete reporting.
| S-EPMC8627036 | biostudies-literature
Project description:BackgroundObservational analysis methods can be refined by benchmarking against randomized trials. We reviewed studies systematically comparing observational analyses using propensity score methods against randomized trials to explore whether intervention or outcome characteristics predict agreement between designs.MethodsWe searched PubMed (from January 1, 2000, to April 30, 2017), the AHRQ Scientific Resource Center Methods Library, reference lists, and bibliographies to identify systematic reviews that compared estimates from observational analyses using propensity scores against randomized trials across three or more clinical topics; reported extractable relative risk (RR) data; and were published in English. One reviewer extracted data from all eligible systematic reviews; a second reviewer verified the extracted data.ResultsSix systematic reviews matching published observational studies to randomized trials, published between 2012 and 2016, met our inclusion criteria. The reviews reported on 127 comparisons overall, in cardiology (29 comparisons), surgery (49), critical care medicine and sepsis (46), nephrology (2), and oncology (1). Disagreements were large (relative RR < 0.7 or > 1.43) in 68 (54%) and statistically significant in 12 (9%) of the comparisons. The degree of agreement varied among reviews but was not strongly associated with intervention or outcome characteristics.DiscussionDisagreements between observational studies using propensity score methods and randomized trials can occur for many reasons and the available data cannot be used to discern the reasons behind specific disagreements. Better benchmarking of observational analyses using propensity scores (and other causal inference methods) is possible using observational studies that explicitly attempt to emulate target trials.
| S-EPMC7210373 | biostudies-literature
Project description:BackgroundMixed methods designs are increasingly used in health care research to enrich findings. However, little is known about the frequency of use of this methodology in chiropractic research, or the quality of reporting among chiropractic studies using mixed methods.ObjectiveTo quantify the use and quality of mixed methods in chiropractic research, and explore the association of study characteristics (e.g., authorship, expertise, journal impact factor, country and year of publication) with reporting quality.MethodsWe will conduct a systematic search of MEDLINE, EMBASE, CINAHL, and the Index to Chiropractic Literature to identify all chiropractic mixed methods studies published from inception of each database to December 31, 2020. Articles reporting the use of both qualitative and quantitative methods, or mixed qualitative methods, will be included. Pairs of reviewers will perform article screening, data extraction, risk of bias with the Mixed Methods Appraisal Tool (MMAT), and appraisal of reporting quality using the Good Reporting of A Mixed Methods Study (GRAMMS) guideline. We will explore the correlation between GRAMMS and MMAT scores, and construct generalized estimating equations to explore factors associated with reporting quality.DiscussionThis will be the first methodological review to examine the reporting quality of published mixed methods studies involving chiropractic research. The results of our review will inform opportunities to improve reporting in chiropractic mixed methods studies. Our results will be disseminated in a peer-reviewed publication and presented publicly at conferences and as part of a doctoral thesis.
| S-EPMC8442283 | biostudies-literature
Project description:The maintenance of the excitability of neurons and circuits is a fundamental process for healthy brain functions. One of the main homeostatic mechanisms responsible for such regulation is synaptic scaling. While this type of plasticity is well-characterized through a robust body of literature, there are no systematic evaluations of the methodological and reporting features from these studies. Our review yielded 168 articles directly investigating synaptic scaling mechanisms, which display relatively high impact, with a median impact factor of 7.76 for the publishing journals. Our methodological analysis identified that 86% of the articles made use of inhibitory interventions to induce synaptic scaling, while only 41% of those studies contain excitatory manipulations. To verify the effects of synaptic scaling, the most assessed outcome was miniature excitatory postsynaptic current (mEPSC) recordings, performed in 71% of the articles. We could also observe that the field is mostly focused on mechanistic studies of the synaptic scaling pathways (70%), rather than the interaction with other types of plasticity, such as Hebbian processes (4%). We found that more than half of the articles failed to describe simple features, such as regulatory compliance statements, ethics committee approval, or statements of conflict of interests. In light of these results, we discuss the strengths and pitfalls existing in synaptic scaling literature.
| S-EPMC7309364 | biostudies-literature