Dataset Information

Applying machine learning to predict real-world individual treatment effects: insights from a virtual patient cohort.

ABSTRACT: Objective:We aimed to investigate bias in applying machine learning to predict real-world individual treatment effects.

Materials and methods:Using a virtual patient cohort, we simulated real-world healthcare data and applied random forest and gradient boosting classifiers to develop prediction models. Treatment effect was estimated as the difference between the predicted outcomes of a treatment and a control. We evaluated the impact of predictors (ie, treatment predictors [X1], confounders [X2], treatment effects modifiers [X3], and other outcome risk factors [X4]) with known effects on treatment and outcome using real-world data, and outcome imbalance on predicting individual outcome. Using counterfactuals, we evaluated percentage of patients with biased predicted individual treatment effects.

Results:The X4 had relatively more impact on model performance than X2 and X3 did. No effects were observed from X1. Moderate-to-severe outcome imbalance had a significantly negative impact on model performance, particularly among subgroups in which an outcome occurred. Bias in predicting individual treatment effects was significant and persisted even when the models had a 100% accuracy in predicting health outcome.

Discussion:Inadequate inclusion of the X2, X3, and X4 and moderate-to-severe outcome imbalance may affect model performance in predicting individual outcome and subsequently bias in predicting individual treatment effects. Machine learning models with all features and high performance for predicting individual outcome still yielded biased individual treatment effects.

Conclusions:Direct application of machine learning might not adequately address bias in predicting individual treatment effects. Further method development is needed to advance machine learning to support individualized treatment selection.

SUBMITTER: Fang G

PROVIDER: S-EPMC7647181 | biostudies-literature | 2019 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Applying machine learning to predict real-world individual treatment effects: insights from a virtual patient cohort.

Fang Gang G Annis Izabela E IE Elston-Lafata Jennifer J Cykert Samuel S

Journal of the American Medical Informatics Association : JAMIA 20191001 10

<h4>Objective</h4>We aimed to investigate bias in applying machine learning to predict real-world individual treatment effects.<h4>Materials and methods</h4>Using a virtual patient cohort, we simulated real-world healthcare data and applied random forest and gradient boosting classifiers to develop prediction models. Treatment effect was estimated as the difference between the predicted outcomes of a treatment and a control. We evaluated the impact of predictors (ie, treatment predictors [X1], c ...[more]

PMID: 31220274

Similar Datasets

Project description:BACKGROUND:Identifying individuals who are unlikely to adhere to a physical exercise regime has potential to improve physical activity interventions. The aim of this paper is to develop and test adherence prediction models using objectively measured physical activity data in the Mobile Phone-Based Physical Activity Education program (mPED) trial. To the best of our knowledge, this is the first to apply Machine Learning methods to predict exercise relapse using accelerometer-recorded physical activity data. METHODS:We use logistic regression and support vector machine methods to design two versions of a Discontinuation Prediction Score (DiPS), which uses objectively measured past data (e.g., steps and goal achievement) to provide a numerical quantity indicating the likelihood of exercise relapse in the upcoming week. The respective prediction accuracy of these two versions of DiPS are compared, and then numerical simulation is performed to explore the potential of using DiPS to selectively allocate financial incentives to participants to encourage them to increase physical activity. RESULTS:we had access to a physical activity trial data that were continuously collected every 60?sec every day for 9?months in 210 participants. By using the first 15?weeks of data as training and test on weeks 16-30, we show that both versions of DiPS have a test AUC of 0.9 with high sensitivity and specificity in predicting the probability of exercise adherence. Simulation results assuming different intervention regimes suggest the potential benefit of using DiPS as a score to allocate resources in physical activity intervention programs in reducing costs over other allocation schemes. CONCLUSIONS:DiPS is capable of making accurate and robust predictions for future weeks. The most predictive features are steps and physical activity intensity. Furthermore, the use of DiPS scores can be a promising approach to determine when or if to provide just-in-time messages and step goal adjustments to improve compliance. Further studies on the use of DiPS in the design of physical activity promotion programs are warranted. TRIAL REGISTRATION:ClinicalTrials.gov NCT01280812 Registered on January 21, 2011.

Dataset Information

Applying machine learning to predict real-world individual treatment effects: insights from a virtual patient cohort.

Publications

Applying machine learning to predict real-world individual treatment effects: insights from a virtual patient cohort.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets