Dataset Information

The proposed 'concordance-statistic for benefit' provided a useful metric when modeling heterogeneous treatment effects.

ABSTRACT:

Objectives

Clinical prediction models that support treatment decisions are usually evaluated for their ability to predict the risk of an outcome rather than treatment benefit-the difference between outcome risk with vs. without therapy. We aimed to define performance metrics for a model's ability to predict treatment benefit.

Study design and setting

We analyzed data of the Synergy between Percutaneous Coronary Intervention with Taxus and Cardiac Surgery (SYNTAX) trial and of three recombinant tissue plasminogen activator trials. We assessed alternative prediction models with a conventional risk concordance-statistic (c-statistic) and a novel c-statistic for benefit. We defined observed treatment benefit by the outcomes in pairs of patients matched on predicted benefit but discordant for treatment assignment. The 'c-for-benefit' represents the probability that from two randomly chosen matched patient pairs with unequal observed benefit, the pair with greater observed benefit also has a higher predicted benefit.

Results

Compared to a model without treatment interactions, the SYNTAX score II had improved ability to discriminate treatment benefit (c-for-benefit 0.590 vs. 0.552), despite having similar risk discrimination (c-statistic 0.725 vs. 0.719). However, for the simplified stroke-thrombolytic predictive instrument (TPI) vs. the original stroke-TPI, the c-for-benefit (0.584 vs. 0.578) was similar.

Conclusion

The proposed methodology has the potential to measure a model's ability to predict treatment benefit not captured with conventional performance metrics.

SUBMITTER: van Klaveren D

PROVIDER: S-EPMC7448760 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:Background:Two recent randomized trials produced discordant results when testing the benefits and harms of treatment to reduce blood pressure (BP) in patients with cardiovascular disease (CVD). Objective:To perform a theoretical modeling study to identify whether large, clinically important differences in benefit and harm among patients (heterogeneous treatment effects [HTEs]) can be hidden in, and explain discordant results between, treat-to-target BP trials. Design:Microsimulation. Data Sources:Results of 2 trials comparing standard (systolic BP target <140 mm Hg) with intensive (systolic BP target <120 mm Hg) BP treatment and data from the National Health and Nutrition Examination Survey (2013 to 2014). Target Population:U.S. adults. Time Horizon:5 years. Perspective:Societal. Intervention:BP treatment. Outcome Measures:CVD events and mortality. Results of Base-Case Analysis:Clinically important HTEs could explain differences in outcomes between 2 trials of intensive BP treatment, particularly diminishing benefit with each additional BP agent (for example, adding a second agent reduces CVD risk [hazard ratio, 0.61], but adding a fourth agent to a third has no benefit) and increasing harm at low diastolic BP. Results of Sensitivity Analysis:Conventional treat-to-target trial designs had poor (<5%) statistical power to detect the HTEs, despite large samples (n > 20 000), and produced biased effect estimates. In contrast, a trial with sequential randomization to more intensive therapy achieved greater than 80% power and unbiased HTE estimates, despite small samples (n = 3500). Limitations:The HTEs as a function of the number of BP agents only were explored. Simulated aggregate data from the trials were used as model inputs because individual-participant data were not available. Conclusion:Clinically important heterogeneity in intensive BP treatment effects remains undetectable in conventional trial designs but can be detected in sequential randomization trial designs. Primary Funding Source:National Institutes of Health and U.S. Department of Veterans Affairs.