Estimation of Population Average Treatment Effects in the FIRST Trial: Application of a Propensity Score-Based Stratification Approach.
Ontology highlight
ABSTRACT: Objective/study question
To estimate and compare sample average treatment effects (SATE) and population average treatment effects (PATE) of a resident duty hour policy change on patient and resident outcomes using data from the Flexibility in Duty Hour Requirements for Surgical Trainees Trial ("FIRST Trial").Data sources/study setting
Secondary data from the National Surgical Quality Improvement Program and the FIRST Trial (2014-2015).Study design
The FIRST Trial was a cluster-randomized pragmatic noninferiority trial designed to evaluate the effects of a resident work hour policy change to permit greater flexibility in scheduling on patient and resident outcomes. We estimated hierarchical logistic regression models to estimate the SATE of a policy change on outcomes within an intent-to-treat framework. Propensity score-based poststratification was used to estimate PATE.Data collection/extraction methods
This study was a secondary analysis of previously collected data.Principal findings
Although SATE estimates suggested noninferiority of outcomes under flexible duty hour policy versus standard policy, the noninferiority of a policy change was inconclusively noninferior based on PATE estimates due to imprecision.Conclusions
Propensity score-based poststratification can be valuable tools to address trial generalizability but may yield imprecise estimates of PATE when sparse strata exist.
SUBMITTER: Chung JW
PROVIDER: S-EPMC6051989 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Confounding due to population stratification (PS) arises when differences in both allele and disease frequencies exist in a population of mixed racial/ethnic subpopulations. Genomic control, structured association, principal components analysis (PCA), and multidimensional scaling (MDS) approaches have been proposed to address this bias using genetic markers. However, confounding due to PS can also be due to non-genetic factors. Propensity scores are widely used to address confounding in observational studies but have not been adapted to deal with PS in genetic association studies. We propose a genomic propensity score (GPS) approach to correct for bias due to PS that considers both genetic and non-genetic factors. We compare the GPS method with PCA and MDS using simulation studies. Our results show that GPS can adequately adjust and consistently correct for bias due to PS. Under no/mild, moderate, and severe PS, GPS yielded estimated with bias close to 0 (mean=-0.0044, standard error=0.0087). Under moderate or severe PS, the GPS method consistently outperforms the PCA method in terms of bias, coverage probability (CP), and type I error. Under moderate PS, the GPS method consistently outperforms the MDS method in terms of CP. PCA maintains relatively high power compared to both MDS and GPS methods under the simulated situations. GPS and MDS are comparable in terms of statistical properties such as bias, type I error, and power. The GPS method provides a novel and robust tool for obtaining less-biased estimates of genetic associations that can consider both genetic and non-genetic factors.
| S-EPMC4537699 | biostudies-literature
Project description:Currently, methods for conducting multiple treatment propensity scoring in the presence of high-dimensional covariate spaces that result from "big data" are lacking-the most prominent method relies on inverse probability treatment weighting (IPTW). However, IPTW only utilizes one element of the generalized propensity score (GPS) vector, which can lead to a loss of information and inadequate covariate balance in the presence of multiple treatments. This limitation motivates the development of a novel propensity score method that uses the entire GPS vector to establish a scalar balancing score that, when adjusted for, achieves covariate balance in the presence of potentially high-dimensional covariates. Specifically, the generalized propensity score cumulative distribution function (GPS-CDF) method is introduced. A one-parameter power function fits the CDF of the GPS vector and a resulting scalar balancing score is used for matching and/or stratification. Simulation results show superior performance of the new method compared to IPTW both in achieving covariate balance and estimating average treatment effects in the presence of multiple treatments. The proposed approach is applied to a study derived from electronic medical records to determine the causal relationship between three different vasopressors and mortality in patients with non-traumatic aneurysmal subarachnoid hemorrhage. Results suggest that the GPS-CDF method performs well when applied to large observational studies with multiple treatments that have large covariate spaces.
| S-EPMC7334100 | biostudies-literature
Project description:Continuous treatments propensity scoring remains understudied as the majority of methods are focused on the binary treatment setting. Current propensity score methods for continuous treatments typically rely on weighting in order to produce causal estimates. It has been shown that in some continuous treatment settings, weighting methods can result in worse covariate balance than had no adjustments been made to the data. Furthermore, weighting is not always stable, and resultant estimates may be unreliable due to extreme weights. These issues motivate the current development of novel propensity score stratification techniques to be used with continuous treatments. Specifically, the generalized propensity score cumulative distribution function (GPS-CDF) and the nonparametric GPS-CDF approaches are introduced. Empirical CDFs are used to stratify subjects based on pretreatment confounders in order to produce causal estimates. A detailed simulation study shows superiority of these new stratification methods based on the empirical CDF, when compared with standard weighting techniques. The proposed methods are applied to the "Mexican-American Tobacco use in Children" study to determine the causal relationship between continuous exposure to smoking imagery in movies, and smoking behavior among Mexican-American adolescents. These promising results provide investigators with new options for implementing continuous treatment propensity scoring.
| S-EPMC8629138 | biostudies-literature
Project description:When exposure is infrequent, propensity-score matching results in reduced precision because it discards a large proportion of unexposed patients. To our knowledge, the relative performance of propensity-score stratification in these circumstances has not been examined.Using an empirical example of the association of first trimester statin exposure (prevalence = 0.04%) with risk of congenital malformations and 1,000 simulated cohorts (n = 20,000) with eight combinations of exposure prevalence (0.5%, 1%, 5%, 10%) and outcome risk (3.5%, 10%), we compared four propensity-score-based approaches to confounding adjustment: (1) matching (1:1, 1:5, full), (2) stratification in 10, 50, and 100 strata by entire cohort propensity-score distribution, (3) stratification in 10, 50, and 100 strata by exposed group propensity-score distribution, (4) standardized mortality ratio (SMR) weighting. Weighted generalized linear models were used to derive effect estimates after weighting unexposed according to the distribution of the exposed in their stratum for the stratification approaches.In the empirical example, propensity-score stratification (cohort) approaches resulted in greater imbalances in covariate distributions between statin-exposed and unexposed compared with propensity-score stratification (exposed) and matching. In simulations, propensity-score stratification (exposed) resulted in smaller relative bias than the cohort approach with 10 and 50 strata, and greater precision than matching and SMR weighting at 0.5% and 1% exposure prevalence, but similar performance at 5% and 10%.For exposures with prevalence under 5%, propensity-score stratification with fine strata, based on the exposed group propensity-score distribution, produced the best results. For more common exposures, all approaches were equivalent.
| S-EPMC5497217 | biostudies-literature
Project description:We consider estimating average treatment effects (ATE) of a binary treatment in observational data when data-driven variable selection is needed to select relevant covariates from a moderately large number of available covariates X . To leverage covariates among X predictive of the outcome for efficiency gain while using regularization to fit a parametric propensity score (PS) model, we consider a dimension reduction of X based on fitting both working PS and outcome models using adaptive LASSO. A novel PS estimator, the Double-index Propensity Score (DiPS), is proposed, in which the treatment status is smoothed over the linear predictors for X from both the initial working models. The ATE is estimated by using the DiPS in a normalized inverse probability weighting estimator, which is found to maintain double robustness and also local semiparametric efficiency with a fixed number of covariates p. Under misspecification of working models, the smoothing step leads to gains in efficiency and robustness over traditional doubly robust estimators. These results are extended to the case where p diverges with sample size and working models are sparse. Simulations show the benefits of the approach in finite samples. We illustrate the method by estimating the ATE of statins on colorectal cancer risk in an electronic medical record study and the effect of smoking on C-reactive protein in the Framingham Offspring Study.
| S-EPMC7370895 | biostudies-literature
Project description:Methods based on the propensity score comprise one set of valuable tools for comparative effectiveness research and for estimating causal effects more generally. These methods typically consist of two distinct stages: (1) a propensity score stage where a model is fit to predict the propensity to receive treatment (the propensity score), and (2) an outcome stage where responses are compared in treated and untreated units having similar values of the estimated propensity score. Traditional techniques conduct estimation in these two stages separately; estimates from the first stage are treated as fixed and known for use in the second stage. Bayesian methods have natural appeal in these settings because separate likelihoods for the two stages can be combined into a single joint likelihood, with estimation of the two stages carried out simultaneously. One key feature of joint estimation in this context is "feedback" between the outcome stage and the propensity score stage, meaning that quantities in a model for the outcome contribute information to posterior distributions of quantities in the model for the propensity score. We provide a rigorous assessment of Bayesian propensity score estimation to show that model feedback can produce poor estimates of causal effects absent strategies that augment propensity score adjustment with adjustment for individual covariates. We illustrate this phenomenon with a simulation study and with a comparative effectiveness investigation of carotid artery stenting versus carotid endarterectomy among 123,286 Medicare beneficiaries hospitlized for stroke in 2006 and 2007.
| S-EPMC3622139 | biostudies-literature
Project description:In observational studies of treatments or interventions, propensity score (PS) adjustment is often useful for controlling bias in estimation of treatment effects. Regression on PS is used most often and can be highly efficient, but it can lead to biased results when model assumptions are violated. The validity of stratification on PS depends on fewer model assumptions, but this approach is less efficient than regression adjustment when the regression assumptions hold. To investigate these issues, we compare stratification and regression adjustments in a Monte Carlo simulation study. We consider two stratification approaches: equal frequency strata and an approach that attempts to choose strata that minimize the mean squared error (MSE) of the treatment effect estimate. The regression approach that we consider is a Generalized Additive Model (GAM) that estimates treatment effect controlling for a potentially nonlinear association between PS and outcome. We find that under a wide range of plausible data generating distributions the GAM approach outperforms stratification in treatment effect estimation with respect to bias, variance, and thereby MSE. We illustrate each approach in an analysis of insurance plan choice and its relation to satisfaction with asthma care.
| S-EPMC4238307 | biostudies-literature
Project description:While the two-stage randomized design allows us to unbiasedly evaluate the impact of patients' treatment preference on the outcome of interest, it may not always be practical to implement in clinical practice; patients with a strong preference may not be willing to be randomized. The more pragmatic, partially randomized preference design (PRPD) allows patients who are unwilling to be randomized, but willing to state their preference, to receive their preferred treatment in lieu of the first-stage randomization in the two-stage design, at the cost of potentially introducing bias in estimating the effects of interest. In this article, we consider the application of propensity score stratification (PSS) in a PRPD to recreate a conditional first-stage randomization based on observed covariates, enabling the estimation and inference of the overall treatment, selection and preference effects with minimum bias. We additionally derive a set of closed-form sample size formulas for detecting all three effects of interest in a PSS-PRPD. Simulation studies demonstrate the bias reduction properties of the PSS-PRPD, and validate the accuracy of the proposed sample size formulas. Our results show that 5 to 10 propensity score strata may be needed to correct for biases in effect estimates, and the exact number of strata needed to achieve the best match between the empirical power and formula prediction may depend on the degree of effect heterogeneity. Finally, we demonstrate our proposed formulas by estimating the required sample sizes to detect treatment, selection and preference effects in the context of the Harapan Study.
| S-EPMC10530658 | biostudies-literature
Project description:To illustrate the use of ensemble tree-based methods (random forest classification [RFC] and bagging) for propensity score estimation and to compare these methods with logistic regression, in the context of evaluating the effect of physical and occupational therapy on preschool motor ability among very low birth weight (VLBW) children.We used secondary data from the Early Childhood Longitudinal Study Birth Cohort (ECLS-B) between 2001 and 2006.We estimated the predicted probability of treatment using tree-based methods and logistic regression (LR). We then modeled the exposure-outcome relation using weighted LR models while considering covariate balance and precision for each propensity score estimation method.Among approximately 500 VLBW children, therapy receipt was associated with moderately improved preschool motor ability. Overall, ensemble methods produced the best covariate balance (Mean Squared Difference: 0.03-0.07) and the most precise effect estimates compared to LR (Mean Squared Difference: 0.11). The overall magnitude of the effect estimates was similar between RFC and LR estimation methods.Propensity score estimation using RFC and bagging produced better covariate balance with increased precision compared to LR. Ensemble methods are a useful alterative to logistic regression to control confounding in observational studies.
| S-EPMC3796115 | biostudies-literature
Project description:Existing studies have suggested superior performance of nonparametric machine learning over logistic regression for propensity score estimation. However, it is unclear whether the advantages of nonparametric propensity score modeling are carried to settings where there is clustering of individuals, especially when there is unmeasured cluster-level confounding. In this work we examined the performance of logistic regression (all main effects), Bayesian additive regression trees and generalized boosted modeling for propensity score weighting in clustered settings, with the clustering being accounted for by including either cluster indicators or random intercepts. We simulated data for three hypothetical observational studies of varying sample and cluster sizes. Confounders were generated at both levels, including a cluster-level confounder that is unobserved in the analyses. A binary treatment and a continuous outcome were generated based on seven scenarios with varying relationships between the treatment and confounders (linear and additive, nonlinear/nonadditive, nonadditive with the unobserved cluster-level confounder). Results suggest that when the sample and cluster sizes are large, nonparametric propensity score estimation may provide better covariate balance, bias reduction, and 95% confidence interval coverage, regardless of the degree of nonlinearity or nonadditivity in the true propensity score model. When the sample or cluster sizes are small, however, nonparametric approaches may become more vulnerable to unmeasured cluster-level confounding and thus may not be a better alternative to multilevel logistic regression. We applied the methods to the National Longitudinal Study of Adolescent to Adult Health data, estimating the effect of team sports participation during adolescence on adulthood depressive symptoms.
| S-EPMC9996644 | biostudies-literature