Browse
Submit Data
Databases
API
Help

Dataset Information

14 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Performance of Regression Models as a Function of Experiment Noise.

ABSTRACT:

Background

A challenge in developing machine learning regression models is that it is difficult to know whether maximal performance has been reached on the test dataset, or whether further model improvement is possible. In biology, this problem is particularly pronounced as sample labels (response variables) are typically obtained through experiments and therefore have experiment noise associated with them. Such label noise puts a fundamental limit to the metrics of performance attainable by regression models on the test dataset.

Results

We address this challenge by deriving an expected upper bound for the coefficient of determination (R ²) for regression models when tested on the holdout dataset. This upper bound depends only on the noise associated with the response variable in a dataset as well as its variance. The upper bound estimate was validated via Monte Carlo simulations and then used as a tool to bootstrap performance of regression models trained on biological datasets, including protein sequence data, transcriptomic data, and genomic data.

Conclusions

The new method for estimating upper bounds for model performance on test data should aid researchers in developing ML regression models that reach their maximum potential. Although we study biological datasets in this work, the new upper bound estimates will hold true for regression models from any research field or application area where response variables have associated noise.

SUBMITTER: Li G

PROVIDER: S-EPMC8243133 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Similar Datasets

Regression Models for Identifying Noise Sources in Magnetic Resonance Images.

Project description:Stochastic noise, susceptibility artifacts, magnetic field and radiofrequency inhomogeneities, and other noise components in magnetic resonance images (MRIs) can introduce serious bias into any measurements made with those images. We formally introduce three regression models including a Rician regression model and two associated normal models to characterize stochastic noise in various magnetic resonance imaging modalities, including diffusion-weighted imaging (DWI) and functional MRI (fMRI). Estimation algorithms are introduced to maximize the likelihood function of the three regression models. We also develop a diagnostic procedure for systematically exploring MR images to identify noise components other than simple stochastic noise, and to detect discrepancies between the fitted regression models and MRI data. The diagnostic procedure includes goodness-of-fit statistics, measures of influence, and tools for graphical display. The goodness-of-fit statistics can assess the key assumptions of the three regression models, whereas measures of influence can isolate outliers caused by certain noise components, including motion artifacts. The tools for graphical display permit graphical visualization of the values for the goodness-of-fit statistic and influence measures. Finally, we conduct simulation studies to evaluate performance of these methods, and we analyze a real dataset to illustrate how our diagnostic procedure localizes subtle image artifacts by detecting intravoxel variability that is not captured by the regression models.

| S-EPMC2771876 | biostudies-literature

Wavelet-Based Scalar-on-Function Finite Mixture Regression Models.

Project description:Classical finite mixture regression is useful for modeling the relationship between scalar predictors and scalar responses arising from subpopulations defined by the di ering associations between those predictors and responses. The classical finite mixture regression model is extended to incorporate functional predictors by taking a wavelet-based approach in which both the functional predictors and the component-specific coefficient functions are represented in terms of an appropriate wavelet basis. By using the wavelet representation of the model, the coefficients corresponding to the functional covariates become the predictors. In this setting, there are typically many more predictors than observations. Hence a lasso-type penalization is employed to simultaneously perform feature selection and estimation. Specification of the model is discussed and a fitting algorithm is provided. The wavelet-based approach is evaluated on synthetic data as well as applied to a real data set from a study of the relationship between cognitive ability and di usion tensor imaging measures in subjects with multiple sclerosis.

| S-EPMC4620087 | biostudies-literature

Performance evaluation of cetacean species distribution models developed using generalized additive models and boosted regression trees.

Project description:Species distribution models (SDMs) are important management tools for highly mobile marine species because they provide spatially and temporally explicit information on animal distribution. Two prevalent modeling frameworks used to develop SDMs for marine species are generalized additive models (GAMs) and boosted regression trees (BRTs), but comparative studies have rarely been conducted; most rely on presence-only data; and few have explored how features such as species distribution characteristics affect model performance. Since the majority of marine species BRTs have been used to predict habitat suitability, we first compared BRTs to GAMs that used presence/absence as the response variable. We then compared results from these habitat suitability models to GAMs that predict species density (animals per km2) because density models built with a subset of the data used here have previously received extensive validation. We compared both the explanatory power (i.e., model goodness of fit) and predictive power (i.e., performance on a novel dataset) of the GAMs and BRTs for a taxonomically diverse suite of cetacean species using a robust set of systematic survey data (1991-2014) within the California Current Ecosystem. Both BRTs and GAMs were successful at describing overall distribution patterns throughout the study area for the majority of species considered, but when predicting on novel data, the density GAMs exhibited substantially greater predictive power than both the presence/absence GAMs and BRTs, likely due to both the different response variables and fitting algorithms. Our results provide an improved understanding of some of the strengths and limitations of models developed using these two methods. These results can be used by modelers developing SDMs and resource managers tasked with the spatial management of marine species to determine the best modeling technique for their question of interest.

| S-EPMC7319248 | biostudies-literature

Evaluating the performance of machine-learning regression models for pharmacokinetic drug-drug interactions.

Project description:Combination therapy or concomitant drug administration can be associated with pharmacokinetic drug-drug interactions, increasing the risk of adverse drug events and reduced drug efficacy. Thus far, machine-learning models have been developed that can classify drug-drug interactions. However, to enable quantification of the pharmacokinetic effects of a drug-drug interaction, regression-based machine learning should be explored. Therefore, this study investigated the use of regression-based machine learning to predict changes in drug exposure caused by pharmacokinetic drug-drug interactions. Fold changes in exposure relative to substrate drug monotherapy were collected from 120 clinical drug-drug interaction studies extracted from the Washington Drug Interaction Database and SimCYP compound library files. Drug characteristics (features) were collected such as structure, physicochemical properties, in vitro pharmacokinetic properties, cytochrome P450 metabolic activity, and population characteristics. Three different regression-based supervised machine-learning models were then applied to the prediction task: random forest, elastic net, and support vector regressor. Model performance was evaluated using fivefold cross-validation. Strongest performance was observed with support vector regression, with 78% of predictions within twofold of the observed exposure changes. The results show that changes in drug exposure can be predicted with reasonable accuracy using regression-based machine-learning models trained on data available early in drug discovery. This has potential applications in enabling earlier drug-drug interaction risk assessment for new drug candidates.

| S-EPMC9835131 | biostudies-literature

Validation of noise models for single-cell transcriptomics

Project description:Single-cell transcriptomics has recently emerged as a powerful technology to explore gene expression heterogeneity amongst single cells. Here we identify two major sources of technical variability, sampling noise and global cell-to-cell variation in sequencing efficiency. We propose noise models to correct for this and after validation by single-molecule FISH experiments, we apply these models to demonstrate that growing mES cells in 2i instead of serum/LIF globally reduces gene expression variability. J1 mouse embryonic stem cells (mESCs) were cutured in 2i or in serum medium. Cells were dissociated into a single cell suspension and picked under a stereomicroscope using a 30Î¼m glass capillary and mouth pipette. Picked cells were deposited in the lid of an 0.5ml LoBind eppendorf tube and snap frozen in liquid nitrogen. For the pool and split controls, approximately 1 million cells were lysed, the amount of RNA was quantified on a bioanalyzer (Agilent) using the Eukaryote Total RNA pico kit. 20pg aliquots of total RNA were used for each pool and split control. Single cells and controls were processed using the previously described CEL-seq technique, with a few alterations. A 4bp random barcode as unique molecular identifier (UMI) was added to the primer in between the cell specific barcode and the poly T stretch. Libraries were sequenced on an Illumina HighSeq 2500 using 50bp paired end sequencing. For cells and controls, two libraries were sequenced on two lanes in total for each condition.

2014-04-20 | E-GEOD-54695 | biostudies-arrayexpress

Predictive performance of regression models to estimate Chlorophyll-a concentration based on Landsat imagery.

Project description:Chlorophyll-a (Chl-a) concentration is a key parameter to describe water quality in marine and freshwater environments. Nowadays, several products with Chl-a have derived from satellite imagery, but they are not available or reliable sometimes for coastal and/or small water bodies. Thus, in the last decade several methods have been described to estimate Chl-a with high-resolution (30 m) satellite imagery, such as Landsat, but a standardized method to estimate Chl-a from Landsat imagery has not been accepted yet. Therefore, this study evaluated the predictive performance of regression models (Simple Linear Regression [SLR], Multiple Linear Regression [MLR] and Generalized Additive Models [GAMs]) to estimate Chl-a based on Landsat imagery, using in situ Chl-a data collected (synchronized with the overpass of Landsat 8 satellite) and spectral reflectance in the visible light portion (bands 1-4) and near infrared (band 5). These bands were selected because of Chl-a absorbance/reflectance properties in these wavelengths. According to goodness of fit, GAM outperformed SLR and MLR. However, the model validation showed that MLR performed better in predicting log-transformed Chl-a. Thus, MLR, constructed by using four spectral bands (1, 2, 3, and 5), was considered the best method to predict Chl-a. The coefficients of this model suggested that log-transformed Chl-a concentration had a positive linear relationship with bands 1 (coastal/aerosol), 3 (green), and 5 (NIR). On the other hand, band 2 (blue) suggested a negative relationship, which implied high coherence with Chl-a absorbance/reflectance properties measured in the laboratory, indicating that Landsat 8 images could be applied effectively to estimate Chl-a concentrations in coastal environments.

| S-EPMC6185857 | biostudies-literature

Performance of multi-city land use regression models for nitrogen dioxide and fine particles.

Project description:BackgroundLand use regression (LUR) models have been developed mostly to explain intraurban variations in air pollution based on often small local monitoring campaigns. Transferability of LUR models from city to city has been investigated, but little is known about the performance of models based on large numbers of monitoring sites covering a large area.ObjectivesWe aimed to develop European and regional LUR models and to examine their transferability to areas not used for model development.MethodsWe evaluated LUR models for nitrogen dioxide (NO2) and particulate matter (PM; PM2.5, PM2.5 absorbance) by combining standardized measurement data from 17 (PM) and 23 (NO2) ESCAPE (European Study of Cohorts for Air Pollution Effects) study areas across 14 European countries for PM and NO2. Models were evaluated with cross-validation (CV) and hold-out validation (HV). We investigated the transferability of the models by successively excluding each study area from model building.ResultsThe European model explained 56% of the concentration variability across all sites for NO2, 86% for PM2.5, and 70% for PM2.5 absorbance. The HV R2s were only slightly lower than the model R2 (NO2, 54%; PM2.5, 80%; PM2.5 absorbance, 70%). The European NO2, PM2.5, and PM2.5 absorbance models explained a median of 59%, 48%, and 70% of within-area variability in individual areas. The transferred models predicted a modest-to-large fraction of variability in areas that were excluded from model building (median R2: NO2, 59%; PM2.5, 42%; PM2.5 absorbance, 67%).ConclusionsUsing a large data set from 23 European study areas, we were able to develop LUR models for NO2 and PM metrics that predicted measurements made at independent sites and areas reasonably well. This finding is useful for assessing exposure in health studies conducted in areas where no measurements were conducted.

| S-EPMC4123024 | biostudies-literature

Performance evaluation of survival regression models in analysing Swedish dental implant complication data with frailty.

Project description:The use of inappropriate methods for estimating the effects of covariates in survival data with frailty leads to erroneous conclusions in medical research. This study evaluated the performance of 13 survival regression models in assessing the factors associated with the timing of complications in implant-supported dental restorations in a Swedish cohort. Data were obtained from randomly selected cohort (n = 596) of Swedish patients provided with dental restorations supported in 2003. Patients were evaluated over 9 years of implant loss, peri-implantitis or technical complications. Best Model was identified using goodness, AIC and BIC. The loglikelihood, the AIC and BIC were consistently lower in flexible parametric model with frailty (df = 2) than other models. Adjusted hazard of implant complications was 45% (adjusted Hazard Ratio (aHR) = 1.449; 95% Confidence Interval (CI): 1.153-1.821, p = 0.001) higher among patients with periodontitis. While controlling for other variables, the hazard of implant complications was about 5 times (aHR = 4.641; 95% CI: 2.911-7.401, p<0.001) and 2 times (aHR = 2.338; 95% CI: 1.553-3.519, p<0.001) higher among patients with full- and partial-jaw restorations than those with single crowns. Flexible parametric survival model with frailty are the most suitable for modelling implant complications among the studied patients.

| S-EPMC7790411 | biostudies-literature

Comparing performance between log-binomial and robust Poisson regression models for estimating risk ratios under model misspecification.

Project description:Log-binomial and robust (modified) Poisson regression models are popular approaches to estimate risk ratios for binary response variables. Previous studies have shown that comparatively they produce similar point estimates and standard errors. However, their performance under model misspecification is poorly understood.In this simulation study, the statistical performance of the two models was compared when the log link function was misspecified or the response depended on predictors through a non-linear relationship (i.e. truncated response).Point estimates from log-binomial models were biased when the link function was misspecified or when the probability distribution of the response variable was truncated at the right tail. The percentage of truncated observations was positively associated with the presence of bias, and the bias was larger if the observations came from a population with a lower response rate given that the other parameters being examined were fixed. In contrast, point estimates from the robust Poisson models were unbiased.Under model misspecification, the robust Poisson model was generally preferable because it provided unbiased estimates of risk ratios.

| S-EPMC6013902 | biostudies-other

Additive Function-on-Function Regression.

Project description:We study additive function-on-function regression where the mean response at a particular time point depends on the time point itself, as well as the entire covariate trajectory. We develop a computationally efficient estimation methodology based on a novel combination of spline bases with an eigenbasis to represent the trivariate kernel function. We discuss prediction of a new response trajectory, propose an inference procedure that accounts for total variability in the predicted response curves, and construct pointwise prediction intervals. The estimation/inferential procedure accommodates realistic scenarios, such as correlated error structure as well as sparse and/or irregular designs. We investigate our methodology in finite sample size through simulations and two real data applications. Supplementary Material for this article is available online.

| S-EPMC5956967 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data