Dataset Information

An Efficient Data Partitioning to Improve Classification Performance While Keeping Parameters Interpretable.

ABSTRACT: Supervised machine learning methods typically require splitting data into multiple chunks for training, validating, and finally testing classifiers. For finding the best parameters of a classifier, training and validation are usually carried out with cross-validation. This is followed by application of the classifier with optimized parameters to a separate test set for estimating the classifier's generalization performance. With limited data, this separation of test data creates a difficult trade-off between having more statistical power in estimating generalization performance versus choosing better parameters and fitting a better model. We propose a novel approach that we term "Cross-validation and cross-testing" improving this trade-off by re-using test data without biasing classifier performance. The novel approach is validated using simulated data and electrophysiological recordings in humans and rodents. The results demonstrate that the approach has a higher probability of discovering significant results than the standard approach of cross-validation and testing, while maintaining the nominal alpha level. In contrast to nested cross-validation, which is maximally efficient in re-using data, the proposed approach additionally maintains the interpretability of individual parameters. Taken together, we suggest an addition to currently used machine learning approaches which may be particularly useful in cases where model weights do not require interpretation, but parameters do.

SUBMITTER: Korjus K

PROVIDER: S-EPMC5001642 | biostudies-other | 2016

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

An Efficient Data Partitioning to Improve Classification Performance While Keeping Parameters Interpretable.

Korjus Kristjan K Hebart Martin N MN Vicente Raul R

PloS one 20160826 8

Supervised machine learning methods typically require splitting data into multiple chunks for training, validating, and finally testing classifiers. For finding the best parameters of a classifier, training and validation are usually carried out with cross-validation. This is followed by application of the classifier with optimized parameters to a separate test set for estimating the classifier's generalization performance. With limited data, this separation of test data creates a difficult trad ...[more]

PMID: 27564393

Similar Datasets

Project description:Single-cell RNA sequencing (scRNA-seq) is an invaluable tool for profiling cells in complex tissues and dissecting activation states that lack well-defined surface protein expression. For immune cells, the transcriptomic profile captured by scRNA- seq cannot always identify cell states and subsets defined by conventional flow cytometry. Emerging technologies have enabled multimodal sequencing of single cells, such as paired sequencing of the transcriptome and surface proteome by CITE-seq, but integrating these high dimensional modalities for accurate cell type annotation remains a challenge in the field. Here, we describe a machine learning tool called MultiModal Classifier Hierarchy (MMoCHi) for the cell-type annotation of CITE-seq data. Our classifier involves several steps: 1) we use landmark registration to remove batch-related staining artifacts in CITE-Seq protein expression, 2) the user defines a hierarchy of classifications based on cell type similarity and ontology and provides markers (protein or gene expression) for the identification of ground truth populations within the dataset by threshold gating, 3) progressing through this user-defined hierarchy, we train a random forest classifier using all available modalities (surface proteome and transcriptome data), and 4) we use these forests to predict cell types across the entire dataset. Applying MMoCHi to CITE-seq data of immune cells isolated from eight distinct tissue sites of two human organ donors yields high-purity cell type annotations encompassing the broad array of immune cell states in the dataset. This includes T and B cell memory subsets, macrophages and monocytes, and natural killer cells, as well as rare populations of plasmacytoid dendritic cells, innate T cells, and innate lymphoid cell subsets. We validate the use of feature importances extracted from the classifier hierarchy to select robust genes for improved identification of T cell memory subsets by scRNA-seq. Together, MMoCHi provides a comprehensive system of tools for the batch-correction and cell- type annotation of CITE-seq data. Moreover, this tool provides flexibility in classification hierarchy design allowing for cell type annotations to reflect a researcher’s specific experimental design. This flexibility also renders MMoCHi readily extendable beyond immune cell annotation, and potentially adaptable to other sequencing modalities.

Dataset Information

An Efficient Data Partitioning to Improve Classification Performance While Keeping Parameters Interpretable.

Publications

An Efficient Data Partitioning to Improve Classification Performance While Keeping Parameters Interpretable.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets