Dataset Information

Multiple-rule bias in the comparison of classification rules.

ABSTRACT:

Motivation

There is growing discussion in the bioinformatics community concerning overoptimism of reported results. Two approaches contributing to overoptimism in classification are (i) the reporting of results on datasets for which a proposed classification rule performs well and (ii) the comparison of multiple classification rules on a single dataset that purports to show the advantage of a certain rule.

Results

This article provides a careful probabilistic analysis of the second issue and the 'multiple-rule bias', resulting from choosing a classification rule having minimum estimated error on the dataset. It quantifies this bias corresponding to estimating the expected true error of the classification rule possessing minimum estimated error and it characterizes the bias from estimating the true comparative advantage of the chosen classification rule relative to the others by the estimated comparative advantage on the dataset. The analysis is applied to both synthetic and real data using a number of classification rules and error estimators.

Availability

We have implemented in C code the synthetic data distribution model, classification rules, feature selection routines and error estimation methods. The code for multiple-rule analysis is implemented in MATLAB. The source code is available at http://gsp.tamu.edu/Publications/supplementary/yousefi11a/. Supplementary simulation results are also included.

SUBMITTER: Yousefi MR

PROVIDER: S-EPMC3106200 | biostudies-literature | 2011 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Multiple-rule bias in the comparison of classification rules.

Yousefi Mohammadmahdi R MR Hua Jianping J Dougherty Edward R ER

Bioinformatics (Oxford, England) 20110505 12

<h4>Motivation</h4>There is growing discussion in the bioinformatics community concerning overoptimism of reported results. Two approaches contributing to overoptimism in classification are (i) the reporting of results on datasets for which a proposed classification rule performs well and (ii) the comparison of multiple classification rules on a single dataset that purports to show the advantage of a certain rule.<h4>Results</h4>This article provides a careful probabilistic analysis of the secon ...[more]

PMID: 21546390

Similar Datasets

Project description:MotivationGene expression-based multiclass prediction, such as tumor subtyping, is a non-trivial bioinformatic problem. Most classifier methods operate by comparing expression levels relative to other samples. Methods that base predictions on the expression pattern within a sample have been proposed as an alternative. As these methods are invariant to the cohort composition and can be applied to a sample in isolation, they can collectively be termed single sample predictors (SSP). Such predictors could potentially be used for preprocessing-free classification of new samples and be built to function across different expression platforms where proper batch and dataset normalization is challenging. Here we evaluate the behavior of several multiclass single sample predictors based on binary gene-pair rules (k-Top Scoring Pairs, Absolute Intrinsic Molecular Subtyping, and a new Random Forest approach) and compare them to centroids built with centered or raw expression values, with the criteria that an optimal predictor should have high accuracy, overcome differences in tumor purity, be robust across expression platforms, and provide an informative prediction output score.ResultsWe found that gene-pair based SSPs showed excellent performance on many expression-based classification tasks. The three methods differed in prediction score output, handling of tied scores, and behavior in low purity samples. The k-Top Scoring Pairs and Random Forest approach both achieved high classification accuracy while providing an informative prediction score. Although gene-pair-based SSPs have been touted as being cross-platform compatible (through training on mixed platform data), out-of-the-box compatibility with a new dataset remains a potential issue that warrants cohort-to-cohort verification.AvailabilityOur R package 'multiclassPairs' (https://cran.r-project.org/package=multiclassPairs) (https://doi.org/10.1093/bioinformatics/btab088) is freely available and enables easy training, prediction, and visualization using the gene-pair rule-based Random Forest SSP method and provides additional multiclass functionalities to the switchBox k-Top-Scoring Pairs package.Supplementary informationSupplementary data are available at Bioinformatics online.

Project description:ImportanceComputed tomographic (CT) scanning is the standard for the rapid diagnosis of intracranial injury, but it is costly and exposes patients to ionizing radiation. The Pediatric Emergency Care Applied Research Network (PECARN) rules for identifying children with minor head trauma who are at very low risk of clinically important traumatic brain injury (ciTBI) are widely used to triage CT imaging.ObjectiveTo examine whether optimal classification trees (OCTs), which are novel machine-learning classifiers, improve on PECARN rules' predictive accuracy.Design, setting, and participantsA secondary analysis of prospective, publicly available data on emergency department visits for head trauma used by the PECARN group to develop their tool was conducted to derive OCT-based prediction rules for ciTBI in a development cohort and compare their predictive performance vs the PECARN rules in a validation cohort among children who were younger than 2 years and 2 years or older. Data on 42 412 children with head trauma and without severely altered mental status who were examined between June 1, 2004, and September 30, 2006, were gathered from 25 emergency departments in North America participating in PECARN. Data analysis was conducted from September 15, 2016, to December 18, 2018.Main outcomes and measuresThe outcome was ciTBI, with predictive performance measured by estimating the sensitivity, specificity, positive predictive value, negative predictive value, positive likelihood ratio, and negative likelihood ratio for the OCT and the PECARN rules. The OCT and PECARN rules' performance was compared by estimating ratios for each measure.ResultsOf the 42 412 children (15 996 [37.7%] girls) included in the analysis, 10 718 were younger than 2 years (25.3%; mean [SD] age, 11.6 [0.6] months) and 31 694 were 2 years or older (74.7%; age, 9.1 [4.9] years). Compared with PECARN rules, OCTs misclassified 0 vs 1 child with ciTBI in the younger and 10 vs 9 children with ciTBI in the older cohort, and correctly identified more children with very low risk of ciTBI in the younger (7605 vs 5701) and older (20 594 vs 18 134) cohorts. In the validation cohorts, compared with the PECARN rules, the OCTs had statistically significantly better specificity (in the younger cohort: 69.3%; 95% CI, 67.4%-71.2% vs 52.8%; 95% CI, 50.8%-54.9%; in the older cohort: 65.6%; 95% CI, 64.5%-66.8% vs 57.6%; 95% CI, 56.4%-58.8%), positive predictive value (odds ratios, 1.54; 95% CI, 1.36-1.74 and 1.23; 95% CI, 1.17-1.30, in younger and older children, respectively), and positive likelihood ratio (risk ratios, 1.54; 95% CI, 1.36-1.74 and 1.23; 95% CI, 1.17-1.30, in younger and older children, respectively). There were no statistically significant differences in the sensitivity, negative predictive value, and negative likelihood ratio between the 2 sets of rules.Conclusions and relevanceIf implemented, OCTs may help reduce the number of unnecessary CT scans, without missing more patients with ciTBI than the PECARN rules.

Dataset Information

Multiple-rule bias in the comparison of classification rules.

Motivation

Results

Availability

Publications

Multiple-rule bias in the comparison of classification rules.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure