Dataset Information

Oversampling and replacement strategies in propensity score matching: a critical review focused on small sample size in clinical settings

ABSTRACT:

Background

Propensity score matching is a statistical method that is often used to make inferences on the treatment effects in observational studies. In recent years, there has been widespread use of the technique in the cardiothoracic surgery literature to evaluate to potential benefits of new surgical therapies or procedures. However, the small sample size and the strong dependence of the treatment assignment on the baseline covariates that often characterize these studies make such an evaluation challenging from a statistical point of view. In such settings, the use of propensity score matching in combination with oversampling and replacement may provide a solution to these issues by increasing the initial sample size of the study and thus improving the statistical power that is needed to detect the effect of interest. In this study, we review the use of propensity score matching in combination with oversampling and replacement in small sample size settings.

Methods

We performed a series of Monte Carlo simulations to evaluate how the sample size, the proportion of treated, and the assignment mechanism affect the performances of the proposed approaches. We assessed the performances with overall balance, relative bias, root mean squared error and nominal coverage. Moreover, we illustrate the methods using a real case study from the cardiac surgery literature.

Results

Matching without replacement produced estimates with lower bias and better nominal coverage than matching with replacement when 1:1 matching was considered. In contrast to that, matching with replacement showed better balance, relative bias, and root mean squared error than matching without replacement for increasing levels of oversampling. The best nominal coverage was obtained by using the estimator that accounts for uncertainty in the matching procedure on sets of units obtained after matching with replacement.

Conclusions

The use of replacement provides the most reliable treatment effect estimates and that no more than 1 or 2 units from the control group should be matched to each treated observation. Moreover, the variance estimator that accounts for the uncertainty in the matching procedure should be used to estimate the treatment effect.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01454-z.

SUBMITTER: Bottigliengo D

PROVIDER: S-EPMC8609749 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundThe diagnostic process is a key element of medicine but it is complex and prone to errors. Infectious diseases are one of the three categories of diseases in which diagnostic errors can be most harmful to patients. In this study we aimed to estimate the effect of initial misdiagnosis of the source of infection in patients with bacteraemia on 14 day mortality using propensity score methods to adjust for confounding.MethodsData from a previously described longitudinal cohort of patients diagnosed with monobacterial bloodstream infection (BSI) at the Leiden University Medical Centre (LUMC) between 2013 and 2015 were used. Propensity score matching and inversed probability of treatment weighting (IPTW) were applied to correct for confounding. The average treatment effect on the treated (ATT), which in this study was the average effect of initial misdiagnosis on the misdiagnosed (AEMM), was estimated. Methodological issues that were encountered when applying propensity score methods were addressed by performing additional sensitivity analyses. Sensitivity analyses consisted of varying caliper in propensity score matching and using different truncated weights in inversed probability of treatment weighting.ResultsData of 887 patients were included in the study. Propensity scores ranged between 0.015 and 0.999 and 80 patients (9.9%) had a propensity score > 0.95. In the matched analyses, 35 of the 171 misdiagnosed patients died within 14 days (20.5%), versus 10 of the 171 correctly diagnosed patients (5.8%), yielding a difference of 14.6% (7.6%; 21.6%). In the total group of patients, the observed percentage of patients with an incorrect initial diagnosis that died within 14 days was 19.8% while propensity score reweighting estimated that their probability of dying would have been 6.5%, if they had been correctly diagnosed (difference 13.3% (95% CI 6.9%;19.6%)). After adjustment for all variables that showed disbalance in the propensity score a difference of 13.7% (7.4%; 19.9%) was estimated. Sensitivity analyses yielded similar results. However, performing weighted analyses without truncation yielded unstable results.ConclusionThus, we observed a substantial increase of 14 day mortality in initially misdiagnosed patients. Furthermore, several patients received propensity scores extremely close to one and were almost sure to be initially misdiagnosed.