Performance of binary prediction models in high-correlation low-dimensional settings: a comparison of methods.
Ontology highlight
ABSTRACT: Background
Clinical prediction models are developed widely across medical disciplines. When predictors in such models are highly collinear, unexpected or spurious predictor-outcome associations may occur, thereby potentially reducing face-validity of the prediction model. Collinearity can be dealt with by exclusion of collinear predictors, but when there is no a priori motivation (besides collinearity) to include or exclude specific predictors, such an approach is arbitrary and possibly inappropriate.Methods
We compare different methods to address collinearity, including shrinkage, dimensionality reduction, and constrained optimization. The effectiveness of these methods is illustrated via simulations.Results
In the conducted simulations, no effect of collinearity was observed on predictive outcomes (AUC, R2, Intercept, Slope) across methods. However, a negative effect of collinearity on the stability of predictor selection was found, affecting all compared methods, but in particular methods that perform strong predictor selection (e.g., Lasso). Methods for which the included set of predictors remained most stable under increased collinearity were Ridge, PCLR, LAELR, and Dropout.Conclusions
Based on the results, we would recommend refraining from data-driven predictor selection approaches in the presence of high collinearity, because of the increased instability of predictor selection, even in relatively high events-per-variable settings. The selection of certain predictors over others may disproportionally give the impression that included predictors have a stronger association with the outcome than excluded predictors.
SUBMITTER: Leeuwenberg AM
PROVIDER: S-EPMC8751246 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Over the last decades, molecular signatures have become increasingly important in oncology and are opening up a new area of personalized medicine. Nevertheless, biological relevance and statistical tools necessary for the development of these signatures have been called into question in the literature. Here, we investigate six typical selection methods for high-dimensional settings and survival endpoints, including LASSO and some of its extensions, component-wise boosting, and random survival forests (RSF). A resampling algorithm based on data splitting was used on nine high-dimensional simulated datasets to assess selection stability on training sets and the intersection between selection methods. Prognostic performances were evaluated on respective validation sets. Finally, one application on a real breast cancer dataset has been proposed. The false discovery rate (FDR) was high for each selection method, and the intersection between lists of predictors was very poor. RSF selects many more variables than the other methods and thus becomes less efficient on validation sets. Due to the complex correlation structure in genomic data, stability in the selection procedure is generally poor for selected predictors, but can be improved with a higher training sample size. In a very high-dimensional setting, we recommend the LASSO-pcvl method since it outperforms other methods by reducing the number of selected genes and minimizing FDR in most scenarios. Nevertheless, this method still gives a high rate of false positives. Further work is thus necessary to propose new methods to overcome this issue where numerous predictors are present. Pluridisciplinary discussion between clinicians and statisticians is necessary to ensure both statistical and biological relevance of the predictors included in molecular signatures.
| S-EPMC7350178 | biostudies-literature
Project description:Prognostic models based on survival data frequently make use of the Cox proportional hazards model. Developing reliable Cox models with few events relative to the number of predictors can be challenging, even in low-dimensional datasets, with a much larger number of observations than variables. In such a setting we examined the performance of methods used to estimate a Cox model, including (i) full model using all available predictors and estimated by standard techniques, (ii) backward elimination (BE), (iii) ridge regression, (iv) least absolute shrinkage and selection operator (lasso), and (v) elastic net. Based on a prospective cohort of patients with manifest coronary artery disease (CAD), we performed a simulation study to compare the predictive accuracy, calibration, and discrimination of these approaches. Candidate predictors for incident cardiovascular events we used included clinical variables, biomarkers, and a selection of genetic variants associated with CAD. The penalized methods, i.e., ridge, lasso, and elastic net, showed a comparable performance, in terms of predictive accuracy, calibration, and discrimination, and outperformed BE and the full model. Excessive shrinkage was observed in some cases for the penalized methods, mostly on the simulation scenarios having the lowest ratio of a number of events to the number of variables. We conclude that in similar settings, these three penalized methods can be used interchangeably. The full model and backward elimination are not recommended in rare event scenarios.
| S-EPMC4996851 | biostudies-literature
Project description:ImportanceThe lack of standards in methods to reduce bias for clinical algorithms presents various challenges in providing reliable predictions and in addressing health disparities.ObjectiveTo evaluate approaches for reducing bias in machine learning models using a real-world clinical scenario.Design, setting, and participantsHealth data for this cohort study were obtained from the IBM MarketScan Medicaid Database. Eligibility criteria were as follows: (1) Female individuals aged 12 to 55 years with a live birth record identified by delivery-related codes from January 1, 2014, through December 31, 2018; (2) greater than 80% enrollment through pregnancy to 60 days post partum; and (3) evidence of coverage for depression screening and mental health services. Statistical analysis was performed in 2020.ExposuresBinarized race (Black individuals and White individuals).Main outcomes and measuresMachine learning models (logistic regression [LR], random forest, and extreme gradient boosting) were trained for 2 binary outcomes: postpartum depression (PPD) and postpartum mental health service utilization. Risk-adjusted generalized linear models were used for each outcome to assess potential disparity in the cohort associated with binarized race (Black or White). Methods for reducing bias, including reweighing, Prejudice Remover, and removing race from the models, were examined by analyzing changes in fairness metrics compared with the base models. Baseline characteristics of female individuals at the top-predicted risk decile were compared for systematic differences. Fairness metrics of disparate impact (DI, 1 indicates fairness) and equal opportunity difference (EOD, 0 indicates fairness).ResultsAmong 573 634 female individuals initially examined for this study, 314 903 were White (54.9%), 217 899 were Black (38.0%), and the mean (SD) age was 26.1 (5.5) years. The risk-adjusted odds ratio comparing White participants with Black participants was 2.06 (95% CI, 2.02-2.10) for clinically recognized PPD and 1.37 (95% CI, 1.33-1.40) for postpartum mental health service utilization. Taking the LR model for PPD prediction as an example, reweighing reduced bias as measured by improved DI and EOD metrics from 0.31 and -0.19 to 0.79 and 0.02, respectively. Removing race from the models had inferior performance for reducing bias compared with the other methods (PPD: DI = 0.61; EOD = -0.05; mental health service utilization: DI = 0.63; EOD = -0.04).Conclusions and relevanceClinical prediction models trained on potentially biased data may produce unfair outcomes on the basis of the chosen metrics. This study's results suggest that the performance varied depending on the model, outcome label, and method for reducing bias. This approach toward evaluating algorithmic bias can be used as an example for the growing number of researchers who wish to examine and address bias in their data and models.
| S-EPMC8050742 | biostudies-literature
Project description:It is widely acknowledged that the predictive performance of clinical prediction models should be studied in patients that were not part of the data in which the model was derived. Out-of-sample performance can be hampered when predictors are measured differently at derivation and external validation. This may occur, for instance, when predictors are measured using different measurement protocols or when tests are produced by different manufacturers. Although such heterogeneity in predictor measurement between derivation and validation data is common, the impact on the out-of-sample performance is not well studied. Using analytical and simulation approaches, we examined out-of-sample performance of prediction models under various scenarios of heterogeneous predictor measurement. These scenarios were defined and clarified using an established taxonomy of measurement error models. The results of our simulations indicate that predictor measurement heterogeneity can induce miscalibration of prediction and affects discrimination and overall predictive accuracy, to extents that the prediction model may no longer be considered clinically useful. The measurement error taxonomy was found to be helpful in identifying and predicting effects of heterogeneous predictor measurements between settings of prediction model derivation and validation. Our work indicates that homogeneity of measurement strategies across settings is of paramount importance in prediction research.
| S-EPMC6619392 | biostudies-literature
Project description:We report performance of six different protocols for small RNAseq library preparation and of a method utilizing sequencing of probes targeting microRNAs (HTG EdgeSeq). Recently, small RNA sequencing (small RNA-seq) has been introduced as a method for quantifying circulating microRNAs (miRNAs) and enabling their global profiling without prior knowledge of target sequences. Despite its great promise, small RNA-seq has not delivered the expected outcomes, particularly due to ligation and PCR bias introduced within the workflow. In this study, we assessed the performance of all existing approaches to the small RNA-seq of miRNAs in plasma samples: original two adapter ligation approach; single adapter ligation with subsequent circularization; polyadenylation; use of randomized adapters; and use of unique molecular identifiers (UMI). Using comprehensive set of metrics, we evaluated each protocol in terms of yield, precision, accuracy, sensitivity, and ability to detect isomiRs. Moreover, we assessed performance of targeted RNA-seq method utilizing hybridization probes across relevant metrics and together with RT-qPCR we used it as a reference for accuracy evaluation. The best results were delivered by targeted RNA-seq outperforming other methods in all relevant parameters. The protocols using randomized adapters or UMIs showed consistent good performance across all of the assessed metrics. In contrast, the polyadenylation approach generated a high percentage of discarded reads and impeded the analysis of isomiRs. The single adapter ligation with subsequent circularization failed to prevent ligation bias and the traditional two adapter ligation approach achieved the worse scores in the majority of tested metrics. To sum, we provide a comprehensive comparison that can serve as a guide for new users interested in analysis of circulating miRNAs and as a reference for further comparative studies.
2020-12-31 | GSE149513 | GEO
Project description:Estimation methods for nonlinear mixed-effects modelling have considerably improved over the last decades. Nowadays, several algorithms implemented in different software are used. The present study aimed at comparing their performance for dose-response models. Eight scenarios were considered using a sigmoid E(max) model, with varying sigmoidicity and residual error models. One hundred simulated datasets for each scenario were generated. One hundred individuals with observations at four doses constituted the rich design and at two doses, the sparse design. Nine parametric approaches for maximum likelihood estimation were studied: first-order conditional estimation (FOCE) in NONMEM and R, LAPLACE in NONMEM and SAS, adaptive Gaussian quadrature (AGQ) in SAS, and stochastic approximation expectation maximization (SAEM) in NONMEM and MONOLIX (both SAEM approaches with default and modified settings). All approaches started first from initial estimates set to the true values and second, using altered values. Results were examined through relative root mean squared error (RRMSE) of the estimates. With true initial conditions, full completion rate was obtained with all approaches except FOCE in R. Runtimes were shortest with FOCE and LAPLACE and longest with AGQ. Under the rich design, all approaches performed well except FOCE in R. When starting from altered initial conditions, AGQ, and then FOCE in NONMEM, LAPLACE in SAS, and SAEM in NONMEM and MONOLIX with tuned settings, consistently displayed lower RRMSE than the other approaches. For standard dose-response models analyzed through mixed-effects models, differences were identified in the performance of estimation methods available in current software, giving material to modellers to identify suitable approaches based on an accuracy-versus-runtime trade-off.
| S-EPMC3385815 | biostudies-literature
Project description:Low salinity waterflooding (LSWF) and its variants also known as smart water or ion tuned water injection have emerged as promising enhanced oil recovery (EOR) methods. LSWF is a complex process controlled by several mechanisms and parameters involving oil, brine, and rock composition. The major mechanisms and processes controlling LSWF are still being debated in the literature. Thus, the establishment of an approach that relates these parameters to the final recovery factor (RFf) is vital. The main objective of this research work was to use a number of artificial intelligence models to develop robust predictive models based on experimental data and main parameters controlling the LSWF determined through sensitivity analysis and feature selection. The parameters include properties of oil, rock, injected brine, and connate water. Different operational parameters were considered to increase the model accuracy as well. After collecting the relevant data from 99 experimental studies reported in the literature, the database underwent a comprehensive and rigorous data preprocessing stage, which included removal of duplicates and low-variance features, missing value imputation, collinearity assessment, data characteristic assessment, outlier removal, feature selection, data splitting (80-20 rule was applied), and data scaling. Then, a number of methods such as linear regression (LR), multilayer perceptron (MLP), support vector machine (SVM), and committee machine intelligent system (CMIS) were used to link 1316 data samples assembled in this research work. Based on the obtained results, the CMIS model was proven to produce superior results compared to its counterparts such that the root mean squared rrror (RMSE) values for both training and testing data are 4.622 and 7.757, respectively. Based on the feature importance results, the presence of Ca2+ in the connate water, Na+ in the injected brine, core porosity, and total acid number of the crude oil are detected as the parameters with the highest impact on the RFf. The CMIS model proposed here can be applied with a high degree of confidence to predict the performance of LSWF in sandstone reservoirs. The database assembled for the purpose of this research work is so far the largest and most comprehensive of its kind, and it can be used to further delineate mechanisms behind LSWF and optimization of this EOR process in sandstone reservoirs.
| S-EPMC8638312 | biostudies-literature
Project description:Evaluation of the performance of eight pathogenicity prediction methods for missense variants in the dog and horse.
2022-09-25 | PRJEB56211 | EVA
Project description:Data collected from clinical trials and cohort studies, such as dementia studies, are often high-dimensional, censored, heterogeneous and contain missing information, presenting challenges to traditional statistical analysis. There is an urgent need for methods that can overcome these challenges to model this complex data. At present there is no cure for dementia and no treatment that can successfully change the course of the disease. Machine learning models that can predict the time until a patient develops dementia are important tools in helping understand dementia risks and can give more accurate results than traditional statistical methods when modelling high-dimensional, heterogeneous, clinical data. This work compares the performance and stability of ten machine learning algorithms, combined with eight feature selection methods, capable of performing survival analysis of high-dimensional, heterogeneous, clinical data. We developed models that predict survival to dementia using baseline data from two different studies. The Sydney Memory and Ageing Study (MAS) is a longitudinal cohort study of 1037 participants, aged 70-90 years, that aims to determine the effects of ageing on cognition. The Alzheimer's Disease Neuroimaging Initiative (ADNI) is a longitudinal study aimed at identifying biomarkers for the early detection and tracking of Alzheimer's disease. Using the concordance index as a measure of performance, our models achieve maximum performance values of 0.82 for MAS and 0.93 For ADNI.
| S-EPMC7683682 | biostudies-literature
Project description:As a performance measure for a prediction model, the area under the receiver operating characteristic curve (AUC) is insensitive to the addition of strong markers. A number of measures sensitive to performance change have recently been proposed; however, these relative-performance measures may lead to self-contradictory conclusions. This paper examines alternative performance measures for prediction models: the Lorenz curve-based Gini and Pietra indices, and a standardized version of the Brier score, the scaled Brier. Computer simulations are performed in order to study the sensitivity of these measures to performance change when a new marker is added to a baseline model. When the discrimination power of the added marker is concentrated in the gray zone of the baseline model, the AUC and the Gini show minimal performance improvements. The Pietra and the scaled Brier show more significant improvements in the same situation, comparatively. The Pietra and the scaled Brier indices are therefore recommended for prediction model performance measurement, in light of their ease of interpretation, clinical relevance and sensitivity to gray-zone resolving markers.
| S-EPMC3946724 | biostudies-literature