Understanding covariate shift in model performance.
Ontology highlight
ABSTRACT:
SUBMITTER: McGaughey G
PROVIDER: S-EPMC5070592 | biostudies-literature | 2016
REPOSITORIES: biostudies-literature
Publications
Understanding covariate shift in model performance.
McGaughey Georgia G Walters W Patrick WP Goldman Brian B
F1000Research 20160407
Three (3) different methods (logistic regression, covariate shift and k-NN) were applied to five (5) internal datasets and one (1) external, publically available dataset where covariate shift existed. In all cases, k-NN's performance was inferior to either logistic regression or covariate shift. Surprisingly, there was no obvious advantage for using covariate shift to reweight the training data in the examined datasets. ...[more]
Similar Datasets
Project description:Covariate shift is a prevalent setting for supervised learning in the wild when the training and test data are drawn from different time periods, different but related domains, or via different sampling strategies. This paper addresses a transfer learning setting, with covariate shift between source and target domains. Most existing methods for correcting covariate shift exploit density ratios of the features to reweight the source-domain data, and when the features are high-dimensional, the estimated density ratios may suffer large estimation variances, leading to poor prediction performance. In this work, we investigate the dependence of covariate shift correction performance on the dimensionality of the features, and propose a correction method that finds a low-dimensional representation of the features, which takes into account feature relevant to the target Y, and exploits the density ratio of this representation for importance reweighting. We discuss the factors affecting the performance of our method and demonstrate its capabilities on both pseudo-real and real-world data.
| S-EPMC6730633 | biostudies-literature
Project description:Many applications of machine-learning methods involve an iterative protocol in which data are collected, a model is trained, and then outputs of that model are used to choose what data to consider next. For example, a data-driven approach for designing proteins is to train a regression model to predict the fitness of protein sequences and then use it to propose new sequences believed to exhibit greater fitness than observed in the training data. Since validating designed sequences in the wet laboratory is typically costly, it is important to quantify the uncertainty in the model's predictions. This is challenging because of a characteristic type of distribution shift between the training and test data that arises in the design setting-one in which the training and test data are statistically dependent, as the latter is chosen based on the former. Consequently, the model's error on the test data-that is, the designed sequences-has an unknown and possibly complex relationship with its error on the training data. We introduce a method to construct confidence sets for predictions in such settings, which account for the dependence between the training and test data. The confidence sets we construct have finite-sample guarantees that hold for any regression model, even when it is used to choose the test-time input distribution. As a motivating use case, we use real datasets to demonstrate how our method quantifies uncertainty for the predicted fitness of designed proteins and can therefore be used to select design algorithms that achieve acceptable tradeoffs between high predicted fitness and low predictive uncertainty.
| S-EPMC9618043 | biostudies-literature
Project description:The accuracy and predictability of mixture models in NONMEM® may change depending on the relative size of inter-individual differences and the size of the differences in the parameters between subpopulations. This study explored the accuracy of mixture models when dealing with missing a categorical covariate under various situations that may occur in reality. We generated simulation data under various scenarios where genotypes representing extensive metabolizers (EM) and poor metabolizers (PM) of drug-metabolizing enzymes affect the clearance of a drug by different degrees, and the inter-individual variations in clearance are different for each scenario. From each simulated datum, a specific proportion of the covariate (genotype information) was randomly removed. Based on these simulation data, the proportion of each individual subpopulation and the clearance were estimated using a mixture model. Overall, the clearance estimate was more accurate when the difference in clearance between subpopulations was large, and the inter-individual variations were small. In some scenarios that showed higher ETA or epsilon shrinkage, the clearance estimates were significantly biased. The mixture model made better predictions for individuals in the EM subpopulation than for individuals in the PM subpopulation. However, the estimated values were not significantly affected by the tested ratio, if the sample size was secured to some extent. The current simulation study suggests that when the coefficient of variation of inter-individual variations of clearance exceeds 40%, the mixture model should be used carefully, and it should be taken into account that shrinkage can bias the results.
| S-EPMC7032962 | biostudies-literature
Project description:New approaches to expedite the development of safe and effective pediatric dosing regimens and first-in-child doses are urgently needed. Model-based approaches require quantitative functions on the maturation of different metabolic pathways. In this study, we directly incorporated a pediatric covariate model for the glucuronidation of morphine into a pediatric population model for zidovudine glucuronidation. This model was compared with a reference model that gave the statistically best description of the data. Both models had adequate goodness-of-fit plots and normalized prediction distribution errors (NPDE), similar population clearance values for each individual, and a ?objective function value of 13 points (?2df). This supports our hypothesis that pediatric pharmacokinetic covariate models contain system-specific information that can be used as semiphysiological functions in pediatric population models. Further research should explore the validity of the semiphysiological function for other UDP-glucuronosyltransferase 2B7 substrates and patient populations and reveal how this function can be used for pediatric physiologically based pharmacokinetic models.CPT: Pharmacometrics & Systems Pharmacology (2012) 1, e9; doi:10.1038/psp.2012.11; advance online publication 3 October 2012.
| S-EPMC3603431 | biostudies-literature
Project description:BackgroundBayesian hierarchical models have been proposed to combine evidence from different types of study designs. However, when combining evidence from randomised and non-randomised controlled studies, imbalances in patient characteristics between study arms may bias the results. The objective of this study was to assess the performance of a proposed Bayesian approach to adjust for imbalances in patient level covariates when combining evidence from both types of study designs.Methodology/principal findingsSimulation techniques, in which the truth is known, were used to generate sets of data for randomised and non-randomised studies. Covariate imbalances between study arms were introduced in the non-randomised studies. The performance of the Bayesian hierarchical model adjusted for imbalances was assessed in terms of bias. The data were also modelled using three other Bayesian approaches for synthesising evidence from randomised and non-randomised studies. The simulations considered six scenarios aimed at assessing the sensitivity of the results to changes in the impact of the imbalances and the relative number and size of studies of each type. For all six scenarios considered, the Bayesian hierarchical model adjusted for differences within studies gave results that were unbiased and closest to the true value compared to the other models.Conclusions/significanceWhere informed health care decision making requires the synthesis of evidence from randomised and non-randomised study designs, the proposed hierarchical Bayesian method adjusted for differences in patient characteristics between study arms may facilitate the optimal use of all available evidence leading to unbiased results compared to unadjusted analyses.
| S-EPMC3189931 | biostudies-literature
Project description:Automatic emotion recognition is one of the most challenging tasks. To detect emotion from nonstationary EEG signals, a sophisticated learning algorithm that can represent high-level abstraction is required. This study proposes the utilization of a deep learning network (DLN) to discover unknown feature correlation between input signals that is crucial for the learning task. The DLN is implemented with a stacked autoencoder (SAE) using hierarchical feature learning approach. Input features of the network are power spectral densities of 32-channel EEG signals from 32 subjects. To alleviate overfitting problem, principal component analysis (PCA) is applied to extract the most important components of initial input features. Furthermore, covariate shift adaptation of the principal components is implemented to minimize the nonstationary effect of EEG signals. Experimental results show that the DLN is capable of classifying three different levels of valence and arousal with accuracy of 49.52% and 46.03%, respectively. Principal component based covariate shift adaptation enhances the respective classification accuracy by 5.55% and 6.53%. Moreover, DLN provides better performance compared to SVM and naive Bayes classifiers.
| S-EPMC4165739 | biostudies-other
Project description:In population pharmacokinetic analyses, missing categorical data are often encountered. We evaluated several methods of performing covariate analyses with partially missing categorical covariate data. Missing data methods consisted of discarding data (DROP), additional effect parameter for the group with missing data (EXTRA), and mixture methods in which the mixing probability was fixed to the observed fraction of categories (MIX(obs)), based on the likelihood of the concentration data (MIX(conc)), or combined likelihood of observed covariate data and concentration data (MIX(joint)). Simulations were implemented to study bias and imprecision of the methods in datasets with equal-sized and unbalanced category ratios for a binary covariate as well as datasets with non-random missingness (MNAR). Additionally, the performance and feasibility of implementation was assessed in two real datasets. At either low (10%) or high (50%) levels of missingness, all methods performed similarly well. Performance was similar for situations with unbalanced datasets (3:1 covariate distribution) and balanced datasets. In the MNAR scenario, the MIX methods showed a higher bias in the estimation of CL and covariate effect than EXTRA. All methods could be applied to real datasets, except DROP. All methods perform similarly at the studied levels of missingness, but the DROP and EXTRA methods provided less bias than the mixture methods in the case of MNAR. However, EXTRA was associated with inflated type I error rates of covariate selection, while DROP handled data inefficiently.
| S-EPMC3385822 | biostudies-other
Project description:AimsTo clarify the hypothesis tests associated with the full covariate modelling (FCM) approach in population pharmacokinetic analysis, investigate the potential impact of multiplicity in population pharmacokinetic analysis, and evaluate simultaneous confidence intervals (SCI) as an approach to control multiplicity.MethodsClinical trial simulations were performed using a simple one-compartment pharmacokinetic model. Different numbers of covariates, sample sizes, effect sizes of covariates, and correlations among covariates were explored. The false positive rate (FPR) and power were evaluated.ResultsThe FPR for the FCM approach dramatically increases with number of covariates. The chance of incorrectly selecting ≥1 seemingly clinically relevant covariates can be increased from 5% to a 40-70% range for 10-20 covariates. The SCI approach may provide appropriate control of the family-wise FPR, allowing more appropriate decision making. As a result, the power detecting real effects without incorrectly identifying non-existing effects can be greatly improved by the SCI approach compared to the approach in current practice. The performance of the SCI approach is driven by the ratio of sample size to number of covariates. The FPR can be controlled at 5% and 10% using the SCI approach when the ratio was ≥20 and 10, respectively.ConclusionThe FCM approach still lies within the framework of statistical testing, and therefore multiplicity is an issue for this approach. It is imperative to consider multiplicity reporting and adjustments in FCM modelling practice to ensure more appropriate decision making.
| S-EPMC6005596 | biostudies-literature
Project description:When modeling pharmacokinetic (PK) data, identifying covariates is important in explaining interindividual variability, and thus increasing the predictive value of the model. Nonlinear mixed-effects modeling with stepwise covariate modeling is frequently used to build structural covariate models, and the most commonly used software-NONMEM-provides estimations for the fixed-effect parameters (e.g., drug clearance), interindividual and residual unidentified random effects. The aim of covariate modeling is not only to find covariates that significantly influence the population PK parameters, but also to provide dosing recommendations for a certain drug under different conditions, e.g., organ dysfunction, combination chemotherapy. A true covariate is usually seen as one that carries unique information on a structural model parameter. Covariate models have improved our understanding of the pharmacology of many anticancer drugs, including busulfan or melphalan that are part of high-dose pretransplant treatments, the antifolate methotrexate whose elimination is strongly dependent on GFR and comedication, the taxanes and tyrosine kinase inhibitors, the latter being subject of cytochrome p450 3A4 (CYP3A4) associated metabolism. The purpose of this review article is to provide a tool to help understand population covariate analysis and their potential implications for the clinic. Accordingly, several population covariate models are listed, and their clinical relevance is discussed. The target audience of this article are clinical oncologists with a special interest in clinical and mathematical pharmacology.
| S-EPMC3291194 | biostudies-literature
Project description:A tuning-free procedure is proposed to estimate the covariate-adjusted Gaussian graphical model. For each finite subgraph, this estimator is asymptotically normal and efficient. As a consequence, a confidence interval can be obtained for each edge. The procedure enjoys easy implementation and efficient computation through parallel estimation on subgraphs or edges. We further apply the asymptotic normality result to perform support recovery through edge-wise adaptive thresholding. This support recovery procedure is called ANTAC, standing for Asymptotically Normal estimation with Thresholding after Adjusting Covariates. ANTAC outperforms other methodologies in the literature in a range of simulation studies. We apply ANTAC to identify gene-gene interactions using an eQTL dataset. Our result achieves better interpretability and accuracy in comparison with CAMPE.
| S-EPMC4974017 | biostudies-literature