Dataset Information

Using internally developed risk models to assess heterogeneity in treatment effects in clinical trials.

ABSTRACT: Recent proposals suggest that risk-stratified analyses of clinical trials be routinely performed to better enable tailoring of treatment decisions to individuals. Trial data can be stratified using externally developed risk models (eg, Framingham risk score), but such models are not always available. We sought to determine whether internally developed risk models, developed directly on trial data, introduce bias compared with external models.We simulated a large patient population with known risk factors and outcomes. Clinical trials were then simulated by repeatedly drawing from the patient population assuming a specified relative treatment effect in the experimental arm, which either did or did not vary according to a subject's baseline risk. For each simulated trial, 2 internal risk models were developed on either the control population only (internal controls only) or the whole trial population blinded to treatment (internal whole trial). Bias was estimated for the internal models by comparing treatment effect predictions to predictions from the external model. Under all treatment assumptions, internal models introduced only modest bias compared with external models. The magnitude of these biases was slightly smaller for internal whole trial models than for internal controls only models. Internal whole trial models were also slightly less sensitive to bias introduced by overfitting and less sensitive to falsely identifying the existence of variability in treatment effect across the risk spectrum compared with internal controls only models.Appropriately developed internal models produce relatively unbiased estimates of treatment effect across the spectrum of risk. When estimating treatment effect, internally developed risk models using both treatment arms should, in general, be preferred to models developed on the control population.

SUBMITTER: Burke JF

PROVIDER: S-EPMC3957096 | biostudies-other | 2014 Jan

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

Using internally developed risk models to assess heterogeneity in treatment effects in clinical trials.

Burke James F JF Hayward Rodney A RA Nelson Jason P JP Kent David M DM

Circulation. Cardiovascular quality and outcomes 20140114 1

<h4>Background</h4>Recent proposals suggest that risk-stratified analyses of clinical trials be routinely performed to better enable tailoring of treatment decisions to individuals. Trial data can be stratified using externally developed risk models (eg, Framingham risk score), but such models are not always available. We sought to determine whether internally developed risk models, developed directly on trial data, introduce bias compared with external models.<h4>Methods and results</h4>We simu ...[more]

PMID: 24425710

Similar Datasets

Project description:BackgroundRisk of the outcome is a mathematical determinant of the absolute treatment benefit of an intervention, yet this can vary substantially within a trial population, complicating the interpretation of trial results.MethodsWe developed risk models using Cox or logistic regression on a set of large publicly available randomized controlled trials (RCTs). We evaluated risk heterogeneity using the extreme quartile risk ratio (EQRR, the ratio of outcome rates in the lowest risk quartile to that in the highest) and skewness using the median to mean risk ratio (MMRR, the ratio of risk in the median risk patient to the average). We also examined heterogeneity of treatment effects (HTE) across risk strata.ResultsWe describe 39 analyses using data from 32 large trials, with event rates across studies ranging from 3% to 63% (median?=?15%, 25th-75th percentile?=?9-29%). C-statistics of risk models ranged from 0.59 to 0.89 (median?=?0.70, 25th-75th percentile?=?0.65-0.71). The EQRR ranged from 1.8 to 50.7 (median?=?4.3, 25th-75th percentile?=?3.0-6.1). The MMRR ranged from 0.4 to 1.0 (median?=?0.86, 25th-75th percentile?=?0.80-0.92). EQRRs were predictably higher and MMRRs predictably lower as the c-statistic increased or the overall outcome incidence decreased. Among 18 comparisons with a significant overall treatment effect, there was a significant interaction between treatment and baseline risk on the proportional scale in only one. The difference in the absolute risk reduction between extreme risk quartiles ranged from -3.2 to 28.3% (median?=?5.1%; 25th-75th percentile?=?0.3-10.9).ConclusionsThere is typically substantial variation in outcome risk in clinical trials, commonly leading to clinically significant differences in absolute treatment effects Most patients have outcome risks lower than the trial average reflected in the summary result. Risk-stratified trial analyses are feasible and may be clinically informative, particularly when the outcome is predictable and uncommon.

Project description:BackgroundInvestigators conducting randomized clinical trials often explore treatment effect heterogeneity to assess whether treatment efficacy varies according to patient characteristics. Identifying heterogeneity is central to making informed personalized healthcare decisions. Treatment effect heterogeneity can be investigated using subpopulation treatment effect pattern plot (STEPP), a non-parametric graphical approach that constructs overlapping patient subpopulations with varying values of a characteristic. Procedures for statistical testing using subpopulation treatment effect pattern plot when the endpoint of interest is survival remain an area of active investigation.MethodsA STEPP analysis was used to explore patterns of absolute and relative treatment effects for varying levels of a breast cancer biomarker, Ki-67, in the phase III Breast International Group 1-98 randomized clinical trial, comparing letrozole to tamoxifen as adjuvant therapy for postmenopausal women with hormone receptor-positive breast cancer. Absolute treatment effects were measured by differences in 4-year cumulative incidence of breast cancer recurrence, while relative effects were measured by the subdistribution hazard ratio in the presence of competing risks using O-E (observed-minus-expected) methodology, an intuitive non-parametric method. While estimation of hazard ratio values based on O-E methodology has been shown, a similar development for the subdistribution hazard ratio has not. Furthermore, we observed that the subpopulation treatment effect pattern plot analysis may not produce results, even with 100 patients within each subpopulation. After further investigation through simulation studies, we observed inflation of the type I error rate of the traditional test statistic and sometimes singular variance-covariance matrix estimates that may lead to results not being produced. This is due to the lack of sufficient number of events within the subpopulations, which we refer to as instability of the subpopulation treatment effect pattern plot analysis. We introduce methodology designed to improve stability of the subpopulation treatment effect pattern plot analysis and generalize O-E methodology to the competing risks setting. Simulation studies were designed to assess the type I error rate of the tests for a variety of treatment effect measures, including subdistribution hazard ratio based on O-E estimation. This subpopulation treatment effect pattern plot methodology and standard regression modeling were used to evaluate heterogeneity of Ki-67 in the Breast International Group 1-98 randomized clinical trial.ResultsWe introduce methodology that generalizes O-E methodology to the competing risks setting and that improves stability of the STEPP analysis by pre-specifying the number of events across subpopulations while controlling the type I error rate. The subpopulation treatment effect pattern plot analysis of the Breast International Group 1-98 randomized clinical trial showed that patients with high Ki-67 percentages may benefit most from letrozole, while heterogeneity was not detected using standard regression modeling.ConclusionThe STEPP methodology can be used to study complex patterns of treatment effect heterogeneity, as illustrated in the Breast International Group 1-98 randomized clinical trial. For the subpopulation treatment effect pattern plot analysis, we recommend a minimum of 20 events within each subpopulation.

Dataset Information

Using internally developed risk models to assess heterogeneity in treatment effects in clinical trials.

Publications

Using internally developed risk models to assess heterogeneity in treatment effects in clinical trials.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets