Dataset Information

Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions.

ABSTRACT: Multivariate classification is used in neuroimaging studies to infer brain activation or in medical applications to infer diagnosis. Their results are often assessed through either a binomial or a permutation test. Here, we simulated classification results of generated random data to assess the influence of the cross-validation scheme on the significance of results. Distributions built from classification of random data with cross-validation did not follow the binomial distribution. The binomial test is therefore not adapted. On the contrary, the permutation test was unaffected by the cross-validation scheme. The influence of the cross-validation was further illustrated on real-data from a brain-computer interface experiment in patients with disorders of consciousness and from an fMRI study on patients with Parkinson disease. Three out of 16 patients with disorders of consciousness had significant accuracy on binomial testing, but only one showed significant accuracy using permutation testing. In the fMRI experiment, the mental imagery of gait could discriminate significantly between idiopathic Parkinson's disease patients and healthy subjects according to the permutation test but not according to the binomial test. Hence, binomial testing could lead to biased estimation of significance and false positive or negative results. In our view, permutation testing is thus recommended for clinical application of classification with cross-validation.

SUBMITTER: Noirhomme Q

PROVIDER: S-EPMC4053638 | biostudies-literature | 2014

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions.

Noirhomme Quentin Q Lesenfants Damien D Gomez Francisco F Soddu Andrea A Schrouff Jessica J Garraux Gaëtan G Luxen André A Phillips Christophe C Laureys Steven S

NeuroImage. Clinical 20140413

Multivariate classification is used in neuroimaging studies to infer brain activation or in medical applications to infer diagnosis. Their results are often assessed through either a binomial or a permutation test. Here, we simulated classification results of generated random data to assess the influence of the cross-validation scheme on the significance of results. Distributions built from classification of random data with cross-validation did not follow the binomial distribution. The binomial ...[more]

PMID: 24936420

Similar Datasets

Project description:PurposeMultiple clinical and epidemiological studies have provided estimates of fibromyalgia prevalence and sex ratio, but different criteria sets and methodology, as well as bias, have led to widely varying (0.4%->11%) estimates of prevalence and female predominance (>90% to <61%). In general, studies have failed to distinguish Criteria based fibromyalgia (CritFM) from Clinical fibromyalgia (ClinFM). In the current study we compare CritFM with ClinFM to investigate gender and other biases in the diagnosis of fibromyalgia.MethodsWe used a rheumatic disease databank and 2016 fibromyalgia criteria to study prevalence and sex ratios in a selection biased sample of 1761 referred and diagnosed fibromyalgia patients and in an unbiased sample of 4342 patients with no diagnosis with respect to fibromyalgia. We compared diagnostic and clinical variables according to gender, and we reanalyzed a German population study (GPS) (n = 2435) using revised 2016 criteria for fibromyalgia.ResultsIn the selection-biased sample of referred patients with fibromyalgia, more than 90% were women. However, when an unselected sample of rheumatoid arthritis (RA) patients was studied for the presence of fibromyalgia, women represented 58.7% of fibromyalgia cases. Women had slightly more symptoms than men, including generalized pain (36.8% vs. 32.4%), count of 37 symptoms (4.7 vs. 3.7) and mean polysymptomatic distress scores (10.2 vs. 8.2). We also found a linear relation between the probability of being females and fibromyalgia and fibromyalgia severity. Women in the GPS represented 59.2% of cases.DiscussionThe perception of fibromyalgia as almost exclusively (?90%) a women's disorder is not supported by data in unbiased studies. Using validated self-report criteria and unbiased selection, the female proportion of fibromyalgia cases was ?60% in the unbiased studies, and the observed CritFM prevalence of fibromyalgia in the GPS was ~2%. ClinFM is the public face of fibromyalgia, but is severely affected by selection and confirmation bias in the clinic and publications, underestimating men with fibromyalgia and overestimating women. We recommend the use of 2016 fibromyalgia criteria for clinical diagnosis and epidemiology because of its updated scoring and generalized pain requirement. Fibromyalgia and generalized pain positivity, widespread pain (WPI), symptom severity scale (SSS) and polysymptomatic distress (PSD) scale should always be reported.

Project description:Breast cancer claims 11,400 lives on average every year in the UK, making it one of the deadliest diseases. Mammography is the gold standard for detecting early signs of breast cancer, which can help cure the disease during its early stages. However, incorrect mammography diagnoses are common and may harm patients through unnecessary treatments and operations (or a lack of treatment). Therefore, systems that can learn to detect breast cancer on their own could help reduce the number of incorrect interpretations and missed cases. Various deep learning techniques, which can be used to implement a system that learns how to detect instances of breast cancer in mammograms, are explored throughout this paper. Convolution Neural Networks (CNNs) are used as part of a pipeline based on deep learning techniques. A divide and conquer approach is followed to analyse the effects on performance and efficiency when utilising diverse deep learning techniques such as varying network architectures (VGG19, ResNet50, InceptionV3, DenseNet121, MobileNetV2), class weights, input sizes, image ratios, pre-processing techniques, transfer learning, dropout rates, and types of mammogram projections. This approach serves as a starting point for model development of mammography classification tasks. Practitioners can benefit from this work by using the divide and conquer results to select the most suitable deep learning techniques for their case out-of-the-box, thus reducing the need for extensive exploratory experimentation. Multiple techniques are found to provide accuracy gains relative to a general baseline (VGG19 model using uncropped 512 × 512 pixels input images with a dropout rate of 0.2 and a learning rate of 1 × 10-3) on the Curated Breast Imaging Subset of DDSM (CBIS-DDSM) dataset. These techniques involve transfer learning pre-trained ImagetNet weights to a MobileNetV2 architecture, with pre-trained weights from a binarised version of the mini Mammography Image Analysis Society (mini-MIAS) dataset applied to the fully connected layers of the model, coupled with using weights to alleviate class imbalance, and splitting CBIS-DDSM samples between images of masses and calcifications. Using these techniques, a 5.6% gain in accuracy over the baseline model was accomplished. Other deep learning techniques from the divide and conquer approach, such as larger image sizes, do not yield increased accuracies without the use of image pre-processing techniques such as Gaussian filtering, histogram equalisation and input cropping.

Project description:Assessments of genomic prediction accuracies using artificial intelligent (AI) algorithms (i.e., machine and deep learning methods) are currently not available or very limited in aquaculture species. The principal aim of this study was to examine the predictive performance of these new methods for disease resistance to Edwardsiella ictaluri in a population of striped catfish Pangasianodon hypophthalmus and to make comparisons with four common methods, i.e., pedigree-based best linear unbiased prediction (PBLUP), genomic-based best linear unbiased prediction (GBLUP), single-step GBLUP (ssGBLUP) and a nonlinear Bayesian approach (notably BayesR). Our analyses using machine learning (i.e., ML-KAML) and deep learning (i.e., DL-MLP and DL-CNN) together with the four common methods (PBLUP, GBLUP, ssGBLUP, and BayesR) were conducted for two main disease resistance traits (i.e., survival status coded as 0 and 1 and survival time, i.e., days that the animals were still alive after the challenge test) in a pedigree consisting of 560 individual animals (490 offspring and 70 parents) genotyped for 14,154 single nucleotide polymorphism (SNPs). The results using 6,470 SNPs after quality control showed that machine learning methods outperformed PBLUP, GBLUP, and ssGBLUP, with the increases in the prediction accuracies for both traits by 9.1-15.4%. However, the prediction accuracies obtained from machine learning methods were comparable to those estimated using BayesR. Imputation of missing genotypes using AlphaFamImpute increased the prediction accuracies by 5.3-19.2% in all the methods and data used. On the other hand, there were insignificant decreases (0.3-5.6%) in the prediction accuracies for both survival status and survival time when multivariate models were used in comparison to univariate analyses. Interestingly, the genomic prediction accuracies based on only highly significant SNPs (P < 0.00001, 318-400 SNPs for survival status and 1,362-1,589 SNPs for survival time) were somewhat lower (0.3-15.6%) than those obtained from the whole set of 6,470 SNPs. In most of our analyses, the accuracies of genomic prediction were somewhat higher for survival time than survival status (0/1 data). It is concluded that although there are prospects for the application of genomic selection to increase disease resistance to E. ictaluri in striped catfish breeding programs, further evaluation of these methods should be made in independent families/populations when more data are accumulated in future generations to avoid possible biases in the genetic parameters estimates and prediction accuracies for the disease-resistant traits studied in this population of striped catfish P. hypophthalmus.

Dataset Information

Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions.

Publications

Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets