Bootstrap Bias Corrected Cross Validation Applied to Super Learning
Ontology highlight
ABSTRACT:
SUBMITTER: Krzhizhanovskaya V
PROVIDER: S-EPMC7304018 | biostudies-literature | 2020 May
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Model validation is the most important part of building a supervised model. For building a model with good generalization performance one must have a sensible data splitting strategy, and this is crucial for model validation. In this study, we conducted a comparative study on various reported data splitting methods. The MixSim model was employed to generate nine simulated datasets with different probabilities of mis-classification and variable sample sizes. Then partial least squares for discriminant analysis and support vector machines for classification were applied to these datasets. Data splitting methods tested included variants of cross-validation, bootstrapping, bootstrapped Latin partition, Kennard-Stone algorithm (K-S) and sample set partitioning based on joint X-Y distances algorithm (SPXY). These methods were employed to split the data into training and validation sets. The estimated generalization performances from the validation sets were then compared with the ones obtained from the blind test sets which were generated from the same distribution but were unseen by the training/validation procedure used in model construction. The results showed that the size of the data is the deciding factor for the qualities of the generalization performance estimated from the validation set. We found that there was a significant gap between the performance estimated from the validation set and the one from the test set for the all the data splitting methods employed on small datasets. Such disparity decreased when more samples were available for training/validation, and this is because the models were then moving towards approximations of the central limit theory for the simulated datasets used. We also found that having too many or too few samples in the training set had a negative effect on the estimated model performance, suggesting that it is necessary to have a good balance between the sizes of training set and validation set to have a reliable estimation of model performance. We also found that systematic sampling method such as K-S and SPXY generally had very poor estimation of the model performance, most likely due to the fact that they are designed to take the most representative samples first and thus left a rather poorly representative sample set for model performance estimation.
| S-EPMC6373628 | biostudies-literature
Project description:We present deep-learning-enabled super-resolution across different fluorescence microscopy modalities. This data-driven approach does not require numerical modeling of the imaging process or the estimation of a point-spread-function, and is based on training a generative adversarial network (GAN) to transform diffraction-limited input images into super-resolved ones. Using this framework, we improve the resolution of wide-field images acquired with low-numerical-aperture objectives, matching the resolution that is acquired using high-numerical-aperture objectives. We also demonstrate cross-modality super-resolution, transforming confocal microscopy images to match the resolution acquired with a stimulated emission depletion (STED) microscope. We further demonstrate that total internal reflection fluorescence (TIRF) microscopy images of subcellular structures within cells and tissues can be transformed to match the results obtained with a TIRF-based structured illumination microscope. The deep network rapidly outputs these super-resolved images, without any iterations or parameter search, and could serve to democratize super-resolution imaging.
| S-EPMC7276094 | biostudies-literature
Project description:Online estimators update a current estimate with a new incoming batch of data without having to revisit past data thereby providing streaming estimates that are scalable to big data. We develop flexible, ensemble-based online estimators of an infinite-dimensional target parameter, such as a regression function, in the setting where data are generated sequentially by a common conditional data distribution given summary measures of the past. This setting encompasses a wide range of time-series models and, as special case, models for independent and identically distributed data. Our estimator considers a large library of candidate online estimators and uses online cross-validation to identify the algorithm with the best performance. We show that by basing estimates on the cross-validation-selected algorithm, we are asymptotically guaranteed to perform as well as the true, unknown best-performing algorithm. We provide extensions of this approach including online estimation of the optimal ensemble of candidate online estimators. We illustrate excellent performance of our methods using simulations and a real data example where we make streaming predictions of infectious disease incidence using data from a large database. Copyright © 2017 John Wiley & Sons, Ltd.
| S-EPMC5671383 | biostudies-literature
Project description:We demonstrate that the open-source i2b2 (Informatics for Integrating Biology and the Bedside) data model can be used to bootstrap rural health analytics and learning networks. These networks promote communication and research initiatives by providing the infrastructure necessary for sharing data and insights across a group of healthcare and research partners. Data integration remains a crucial challenge in connecting rural healthcare sites with a common data sharing and learning network due to the lack of interoperability and standards within electronic health records. The i2b2 data model acts as a point of convergence for disparate data from multiple healthcare sites. A consistent and natural data model for healthcare data is essential for overcoming integration issues, but challenges such as those caused by weak data standardization must still be addressed. We describe our experience in the context of building the West Virginia/Kentucky Health Analytics and Learning Network, a collaborative, multi-state effort connecting rural healthcare sites.
| S-EPMC5324727 | biostudies-other
Project description:High-throughput expression profiling allows simultaneous measure of tens of thousands of genes at once. These data have motivated the development of reliable biomarkers for disease subtypes identification and diagnosis. Many methods have been developed in the literature for analyzing these data, such as diagonal discriminant analysis, support vector machines, and k-nearest neighbor methods. The diagonal discriminant methods have been shown to perform well for high-dimensional data with small sample sizes. Despite its popularity, the independence assumption is unlikely to be true in practice. Recently, a gene module based linear discriminant analysis strategy has been proposed by utilizing the correlation among genes in discriminant analysis. However, the approach can be underpowered when the samples of the two classes are unbalanced. In this paper, we propose to correct the biases in the discriminant scores of block-diagonal discriminant analysis. In simulation studies, our proposed method outperforms other approaches in various settings. We also illustrate our proposed discriminant analysis method for analyzing microarray data studies.
| S-EPMC3856189 | biostudies-literature
Project description:Mendelian randomization (MR) is an instrumental variable approach used to infer causal relationships between exposures and outcomes and can apply to summary data from genome-wide association studies (GWAS). Since GWAS summary statistics are subject to estimation errors, most existing MR approaches suffer from measurement error bias, whose scale and direction are influenced by weak instrumental variables and GWAS sample overlap, respectively. We introduce MRBEE (MR using Bias-corrected Estimating Equation), a novel multivariable MR method capable of simultaneously removing measurement error bias and identifying horizontal pleiotropy. In simulations, we showed that MRBEE is capable of effectively removing measurement error bias in the presence of weak instrumental variables and sample overlap. In two independent real data analyses, we discovered that the causal effect of BMI on coronary artery disease risk is entirely mediated by blood pressure, and that existing MR methods may underestimate the causal effect of cannabis use disorder on schizophrenia risk compared to MRBEE. MRBEE possesses significant potential for advancing genetic research by providing a valuable tool to study causality between multiple risk factors and disease outcomes, particularly as a large number of GWAS summary statistics become publicly available.
| S-EPMC10103949 | biostudies-literature
Project description:MotivationIt is commonly assumed in pattern recognition that cross-validation error estimation is 'almost unbiased' as long as the number of folds is not too small. While this is true for random sampling, it is not true with separate sampling, where the populations are independently sampled, which is a common situation in bioinformatics.ResultsWe demonstrate, via analytical and numerical methods, that classical cross-validation can have strong bias under separate sampling, depending on the difference between the sampling ratios and the true population probabilities. We propose a new separate-sampling cross-validation error estimator, and prove that it satisfies an 'almost unbiased' theorem similar to that of random-sampling cross-validation. We present two case studies with previously published data, which show that the results can change drastically if the correct form of cross-validation is used.Availability and implementationThe source code in C++, along with the Supplementary Materials, is available at: http://gsp.tamu.edu/Publications/supplementary/zollanvari13/.
| S-EPMC4296143 | biostudies-literature
Project description:Gene set analysis as it is typically applied to genome-wide methylation assays is severely biased as a result of differences in the numbers and sizes of CpG islands associated with different classes of genes. We demonstrate this bias using published data from a study of differential methylation in lung cancer and a data set we generated to study methylation changes in patients with long-standing ulcerative colitis and show that several of the gene sets that appear enriched would also be identified with randomized data. We also report a method to correct the bias. Application of the corrected method to the lung cancer and ulcerative colitis data sets provides novel and potentially interesting biological insights into the role of methylation in cancer development and chronic inflammation.
2012-07-13 | GSE39188 | GEO
Project description:Observing the cell surface and underlying cytoskeleton at nanoscale resolution using super-resolution microscopy has enabled many insights into cell signaling and function. However, the nanoscale dynamics of tissue-specific immune cells have been relatively little studied. Tissue macrophages, for example, are highly autofluorescent, severely limiting the utility of light microscopy. Here, we report a correction technique to remove autofluorescent noise from stochastic optical reconstruction microscopy (STORM) data sets. Simulations and analysis of experimental data identified a moving median filter as an accurate and robust correction technique, which is widely applicable across challenging biological samples. Here, we used this method to visualize lung macrophages activated through Fc receptors by antibody-coated glass slides. Accurate, nanoscale quantification of macrophage morphology revealed that activation induced the formation of cellular protrusions tipped with MHC class I protein. These data are consistent with a role for lung macrophage protrusions in antigen presentation. Moreover, the tetraspanin protein CD81, known to mark extracellular vesicles, appeared in ring-shaped structures (mean diameter 93 ± 50 nm) at the surface of activated lung macrophages. Thus, a moving median filter correction technique allowed us to quantitatively analyze extracellular secretions and membrane structure in tissue-derived immune cells.
| S-EPMC7822748 | biostudies-literature
Project description:Diagonal discriminant rules have been successfully used for high-dimensional classification problems, but suffer from the serious drawback of biased discriminant scores. In this article, we propose improved diagonal discriminant rules with bias-corrected discriminant scores for high-dimensional classification. We show that the proposed discriminant scores dominate the standard ones under the quadratic loss function. Analytical results on why the bias-corrected rules can potentially improve the predication accuracy are also provided. Finally, we demonstrate the improvement of the proposed rules over the original ones through extensive simulation studies and real case studies.
| S-EPMC3164859 | biostudies-other