Dataset Information

100% classification accuracy considered harmful: the normalized information transfer factor explains the accuracy paradox.

ABSTRACT: The most widely spread measure of performance, accuracy, suffers from a paradox: predictive models with a given level of accuracy may have greater predictive power than models with higher accuracy. Despite optimizing classification error rate, high accuracy models may fail to capture crucial information transfer in the classification task. We present evidence of this behavior by means of a combinatorial analysis where every possible contingency matrix of 2, 3 and 4 classes classifiers are depicted on the entropy triangle, a more reliable information-theoretic tool for classification assessment. Motivated by this, we develop from first principles a measure of classification performance that takes into consideration the information learned by classifiers. We are then able to obtain the entropy-modulated accuracy (EMA), a pessimistic estimate of the expected accuracy with the influence of the input distribution factored out, and the normalized information transfer factor (NIT), a measure of how efficient is the transmission of information from the input to the output set of classes. The EMA is a more natural measure of classification performance than accuracy when the heuristic to maximize is the transfer of information through the classifier instead of classification error count. The NIT factor measures the effectiveness of the learning process in classifiers and also makes it harder for them to "cheat" using techniques like specialization, while also promoting the interpretability of results. Their use is demonstrated in a mind reading task competition that aims at decoding the identity of a video stimulus based on magnetoencephalography recordings. We show how the EMA and the NIT factor reject rankings based in accuracy, choosing more meaningful and interpretable classifiers.

SUBMITTER: Valverde-Albacete FJ

PROVIDER: S-EPMC3888391 | biostudies-literature | 2014

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

100% classification accuracy considered harmful: the normalized information transfer factor explains the accuracy paradox.

Valverde-Albacete Francisco J FJ Peláez-Moreno Carmen C

PloS one 20140110 1

The most widely spread measure of performance, accuracy, suffers from a paradox: predictive models with a given level of accuracy may have greater predictive power than models with higher accuracy. Despite optimizing classification error rate, high accuracy models may fail to capture crucial information transfer in the classification task. We present evidence of this behavior by means of a combinatorial analysis where every possible contingency matrix of 2, 3 and 4 classes classifiers are depict ...[more]

PMID: 24427282

Similar Datasets

Project description:IntroductionThe purpose of this study was to determine which components of sports medicine fellowships are most important to applicants when reviewing fellowship websites during the application process.MethodsAn anonymous survey was distributed to 492 fellowship applicants from the 2017-2018 and 2018-2019 cycles. The survey included questions about the importance of including components of fellow education, recruitment, and experience on program websites. The weighted average of responses determined each component's rank, with 5 being "very important" and 1 being "not at all important." Responses were analyzed by application cycle, current position, and sex using the Wilcoxon rank-sum test.ResultsSixty-five applicants participated in the survey and completed the demographics section, resulting in a 13.2% response rate. According to participants, the most important components to include on fellowship websites were exposure to advanced operative sports medicine techniques (weighted average, 4.62), complexity of cases performed (4.52), and number of cases performed (4.50). Analysis demonstrated statistically significant differences in opinion between application cycles for flexibility for conducting a remote interview (P = .0074), jobs obtained by previous fellows (P = .019), national rank of department (P = .021), program's geographic location (P = .026), protected academic time (P = .038), current positions for criteria for fellows' performance evaluations (P = .028), program's geographic location (P = .0097), and protected academic time (P = .0079). There were statistically significant differences in opinion between current positions regarding flexibility for conducting a remote interview (P = .0026), jobs obtained by previous fellows (P = .012), and national rank of department (P = .0013).ConclusionsOrthopaedic sports medicine fellowship applicants believe that it is most important to include information about the volume and complexity of fellows' cases and their day-to-day commitments on program websites.Clinical relevanceThis information would enable applicants to identify programs that will support professional development and allow program directors to communicate expectations.

Project description:Color patterns are complex traits under selective pressures from conspecifics, mutualists, and antagonists. To evaluate the salience of a pattern or the similarity between colors, several visual models are available. Color discrimination models estimate the perceptual difference between any two colors. Their application to a diversity of taxonomic groups has become common in the literature to answer behavioral, ecological, and evolutionary questions. To use these models, we need information about the visual system of our beholder species. However, many color patterns are simultaneously subject to selective pressures from different species, often from different taxonomic groups, with different visual systems. Furthermore, we lack information about the visual system of many species, leading ecologists to use surrogate values or theoretical estimates for model parameters.Here, we present a modification of the segment classification method proposed by Endler (Biological Journal of the Linnean Society, 1990 41, 315-352): the normalized segment classification model (NSC). We explain its logic and use, exploring how NSC differs from other visual models. We also compare its predictions with available experimental data.Even though the NSC model includes no information about the visual system of the receiver species, it performed better than traditional color discrimination models when predicting the output of some behavioral tasks. Although vision scientists define color as independent of stimulus brightness, a likely explanation for the goodness of fit of the NSC model is that its distance measure depends on brightness differences, and achromatic information can influence the decision-making process of animals when chromatic information is missing.Species-specific models may be insufficient for the study of color patterns in a community context. The NSC model offers a species-independent solution for color analyses, allowing us to calculate color differences when we ignore the intended viewer of a signal or when different species impose selective pressures on the signal.

Dataset Information

100% classification accuracy considered harmful: the normalized information transfer factor explains the accuracy paradox.

Publications

100% classification accuracy considered harmful: the normalized information transfer factor explains the accuracy paradox.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure