Dataset Information

Propensity score matching in otolaryngologic literature: A systematic review and critical appraisal.

ABSTRACT:

Background

Propensity score techniques can reduce confounding and bias in observational studies. Such analyses are able to measure and balance pre-determined covariates between treated and untreated groups, leading to results that can approximate those generated by randomized prospective studies when such trials are not feasible. The most commonly used propensity score -based analytic technique is propensity score matching (PSM). Although PSM popularity has continued to increase in medical literature, improper methodology or methodological reporting may lead to biased interpretation of treatment effects or limited scientific reproducibility and generalizability. In this study, we aim to characterize and assess the quality of PSM methodology reporting in high-impact otolaryngologic literature.

Methods

PubMed and Embase based systematic review of the top 20 journals in otolaryngology, as measured by impact factor from the Journal Citations Reports from 2012 to 2018, for articles using PSM analysis throughout their publication history. Eligible articles were reviewed and assessed for quality and reporting of PSM methodology.

Results

Our search yielded 101 studies, of which 92 were eligible for final analysis and review. The proportion of studies utilizing PSM increased significantly over time (p < 0.001). Nearly all studies (96.7%, n = 89) specified the covariates used to calculate propensity scores. Covariate balance was illustrated in 67.4% (n = 62) of studies, most frequently through p-values. A minority (17.4%, n = 16) of studies were found to be fully reproducible according to previously established criteria.

Conclusions

While PSM analysis is becoming increasingly prevalent in otolaryngologic literature, the quality of PSM methodology reporting can be improved. We provide potential recommendations for authors regarding optimal reporting for analyses using PSM.

SUBMITTER: Prasad A

PROVIDER: S-EPMC7774981 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundPropensity score matching is a statistical method that is often used to make inferences on the treatment effects in observational studies. In recent years, there has been widespread use of the technique in the cardiothoracic surgery literature to evaluate to potential benefits of new surgical therapies or procedures. However, the small sample size and the strong dependence of the treatment assignment on the baseline covariates that often characterize these studies make such an evaluation challenging from a statistical point of view. In such settings, the use of propensity score matching in combination with oversampling and replacement may provide a solution to these issues by increasing the initial sample size of the study and thus improving the statistical power that is needed to detect the effect of interest. In this study, we review the use of propensity score matching in combination with oversampling and replacement in small sample size settings.MethodsWe performed a series of Monte Carlo simulations to evaluate how the sample size, the proportion of treated, and the assignment mechanism affect the performances of the proposed approaches. We assessed the performances with overall balance, relative bias, root mean squared error and nominal coverage. Moreover, we illustrate the methods using a real case study from the cardiac surgery literature.ResultsMatching without replacement produced estimates with lower bias and better nominal coverage than matching with replacement when 1:1 matching was considered. In contrast to that, matching with replacement showed better balance, relative bias, and root mean squared error than matching without replacement for increasing levels of oversampling. The best nominal coverage was obtained by using the estimator that accounts for uncertainty in the matching procedure on sets of units obtained after matching with replacement.ConclusionsThe use of replacement provides the most reliable treatment effect estimates and that no more than 1 or 2 units from the control group should be matched to each treated observation. Moreover, the variance estimator that accounts for the uncertainty in the matching procedure should be used to estimate the treatment effect.

Project description:BackgroundConsumers of research (researchers, administrators, educators and clinicians) frequently use standard critical appraisal tools to evaluate the quality of published research reports. However, there is no consensus regarding the most appropriate critical appraisal tool for allied health research. We summarized the content, intent, construction and psychometric properties of published, currently available critical appraisal tools to identify common elements and their relevance to allied health research.MethodsA systematic review was undertaken of 121 published critical appraisal tools sourced from 108 papers located on electronic databases and the Internet. The tools were classified according to the study design for which they were intended. Their items were then classified into one of 12 criteria based on their intent. Commonly occurring items were identified. The empirical basis for construction of the tool, the method by which overall quality of the study was established, the psychometric properties of the critical appraisal tools and whether guidelines were provided for their use were also recorded.ResultsEighty-seven percent of critical appraisal tools were specific to a research design, with most tools having been developed for experimental studies. There was considerable variability in items contained in the critical appraisal tools. Twelve percent of available tools were developed using specified empirical research. Forty-nine percent of the critical appraisal tools summarized the quality appraisal into a numeric summary score. Few critical appraisal tools had documented evidence of validity of their items, or reliability of use. Guidelines regarding administration of the tools were provided in 43% of cases.ConclusionsThere was considerable variability in intent, components, construction and psychometric properties of published critical appraisal tools for research reports. There is no "gold standard' critical appraisal tool for any study design, nor is there any widely accepted generic tool that can be applied equally well across study types. No tool was specific to allied health research requirements. Thus interpretation of critical appraisal of research reports currently needs to be considered in light of the properties and intent of the critical appraisal tool chosen for the task.

Project description:IntroductionIncreasing severity of hematuria is instinctively associated with higher likelihood of urological malignancy. However, the robustness of the evidentiary base for this assertion is unclear, particularly as it relates to the likelihood of upper urinary tract pathology. Thus, the value of axial imaging in the diagnostic workup of hematuria is unclear due to differences in the underlying patient populations, raising concern for sampling bias. We performed a systematic review to characterize the literature and association between severity of hematuria and likelihood of upper urinary tract cancer based on axial imaging.MethodsMEDLINE, EMBASE, and Cochrane were systematically searched for all studies reporting on adult patients presenting with hematuria. We used Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) for reporting of this systematic review and meta-analysis and the Newcastle-Ottawa Scale for risk of bias assessment. Degree of hematuria was classified as "microscopic," "gross," or "unspecified." Three urological malignancies (bladder, upper tract urothelial, and renal cancer) were considered both individually and in aggregate. Random-effects model with pairwise comparisons was employed to arrive at the axial imaging diagnostic yields.ResultsTwenty-nine studies were included, of which six (20.7%) reported on patients with gross hematuria only, four (13.8%) reported on patients with microscopic hematuria only, seven (24.1%) included both, and 12 (41.4%) did not define or specify the severity of hematuria. Of 29 studies, two (6.9%) were at high-risk of bias, 21 (72.4%) at intermediate-risk, and six (20.7%) at low-risk of bias using the Newcastle-Ottawa criteria. Based on axial imaging, rates of diagnoses of renal, upper tract urothelial, and bladder cancers differed with differing severity of hematuria. Notably, rates of renal and upper tract urothelial carcinoma were higher in studies of patients with unspecified hematuria severity (3.6% and 10.4%, respectively) than among patients with gross hematuria (1.5% and 1.3%, respectively). When all urological malignancies were pooled, patients with unspecified hematuria were diagnosed more frequently (19.5%) compared to those with gross (15.3%) and microscopic hematuria (4.5%, difference=1.51%, 99% confidence interval 3.6-26.5%).ConclusionsLack of granularity in the available literature, particularly with regards to patients with unspecified hematuria severity, limits the generalizability of these results and highlights the need for future studies that provide sufficient baseline information, allowing for firmer conclusions to be drawn.