Dataset Information

Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions.

ABSTRACT: Multivariate classification is used in neuroimaging studies to infer brain activation or in medical applications to infer diagnosis. Their results are often assessed through either a binomial or a permutation test. Here, we simulated classification results of generated random data to assess the influence of the cross-validation scheme on the significance of results. Distributions built from classification of random data with cross-validation did not follow the binomial distribution. The binomial test is therefore not adapted. On the contrary, the permutation test was unaffected by the cross-validation scheme. The influence of the cross-validation was further illustrated on real-data from a brain-computer interface experiment in patients with disorders of consciousness and from an fMRI study on patients with Parkinson disease. Three out of 16 patients with disorders of consciousness had significant accuracy on binomial testing, but only one showed significant accuracy using permutation testing. In the fMRI experiment, the mental imagery of gait could discriminate significantly between idiopathic Parkinson's disease patients and healthy subjects according to the permutation test but not according to the binomial test. Hence, binomial testing could lead to biased estimation of significance and false positive or negative results. In our view, permutation testing is thus recommended for clinical application of classification with cross-validation.

SUBMITTER: Noirhomme Q

PROVIDER: S-EPMC4053638 | biostudies-literature | 2014

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions.

Noirhomme Quentin Q Lesenfants Damien D Gomez Francisco F Soddu Andrea A Schrouff Jessica J Garraux Gaëtan G Luxen André A Phillips Christophe C Laureys Steven S

NeuroImage. Clinical 20140413

Multivariate classification is used in neuroimaging studies to infer brain activation or in medical applications to infer diagnosis. Their results are often assessed through either a binomial or a permutation test. Here, we simulated classification results of generated random data to assess the influence of the cross-validation scheme on the significance of results. Distributions built from classification of random data with cross-validation did not follow the binomial distribution. The binomial ...[more]

PMID: 24936420

Similar Datasets

Project description:PurposeMultiple clinical and epidemiological studies have provided estimates of fibromyalgia prevalence and sex ratio, but different criteria sets and methodology, as well as bias, have led to widely varying (0.4%->11%) estimates of prevalence and female predominance (>90% to <61%). In general, studies have failed to distinguish Criteria based fibromyalgia (CritFM) from Clinical fibromyalgia (ClinFM). In the current study we compare CritFM with ClinFM to investigate gender and other biases in the diagnosis of fibromyalgia.MethodsWe used a rheumatic disease databank and 2016 fibromyalgia criteria to study prevalence and sex ratios in a selection biased sample of 1761 referred and diagnosed fibromyalgia patients and in an unbiased sample of 4342 patients with no diagnosis with respect to fibromyalgia. We compared diagnostic and clinical variables according to gender, and we reanalyzed a German population study (GPS) (n = 2435) using revised 2016 criteria for fibromyalgia.ResultsIn the selection-biased sample of referred patients with fibromyalgia, more than 90% were women. However, when an unselected sample of rheumatoid arthritis (RA) patients was studied for the presence of fibromyalgia, women represented 58.7% of fibromyalgia cases. Women had slightly more symptoms than men, including generalized pain (36.8% vs. 32.4%), count of 37 symptoms (4.7 vs. 3.7) and mean polysymptomatic distress scores (10.2 vs. 8.2). We also found a linear relation between the probability of being females and fibromyalgia and fibromyalgia severity. Women in the GPS represented 59.2% of cases.DiscussionThe perception of fibromyalgia as almost exclusively (?90%) a women's disorder is not supported by data in unbiased studies. Using validated self-report criteria and unbiased selection, the female proportion of fibromyalgia cases was ?60% in the unbiased studies, and the observed CritFM prevalence of fibromyalgia in the GPS was ~2%. ClinFM is the public face of fibromyalgia, but is severely affected by selection and confirmation bias in the clinic and publications, underestimating men with fibromyalgia and overestimating women. We recommend the use of 2016 fibromyalgia criteria for clinical diagnosis and epidemiology because of its updated scoring and generalized pain requirement. Fibromyalgia and generalized pain positivity, widespread pain (WPI), symptom severity scale (SSS) and polysymptomatic distress (PSD) scale should always be reported.

Project description:Assessments of genomic prediction accuracies using artificial intelligent (AI) algorithms (i.e., machine and deep learning methods) are currently not available or very limited in aquaculture species. The principal aim of this study was to examine the predictive performance of these new methods for disease resistance to Edwardsiella ictaluri in a population of striped catfish Pangasianodon hypophthalmus and to make comparisons with four common methods, i.e., pedigree-based best linear unbiased prediction (PBLUP), genomic-based best linear unbiased prediction (GBLUP), single-step GBLUP (ssGBLUP) and a nonlinear Bayesian approach (notably BayesR). Our analyses using machine learning (i.e., ML-KAML) and deep learning (i.e., DL-MLP and DL-CNN) together with the four common methods (PBLUP, GBLUP, ssGBLUP, and BayesR) were conducted for two main disease resistance traits (i.e., survival status coded as 0 and 1 and survival time, i.e., days that the animals were still alive after the challenge test) in a pedigree consisting of 560 individual animals (490 offspring and 70 parents) genotyped for 14,154 single nucleotide polymorphism (SNPs). The results using 6,470 SNPs after quality control showed that machine learning methods outperformed PBLUP, GBLUP, and ssGBLUP, with the increases in the prediction accuracies for both traits by 9.1-15.4%. However, the prediction accuracies obtained from machine learning methods were comparable to those estimated using BayesR. Imputation of missing genotypes using AlphaFamImpute increased the prediction accuracies by 5.3-19.2% in all the methods and data used. On the other hand, there were insignificant decreases (0.3-5.6%) in the prediction accuracies for both survival status and survival time when multivariate models were used in comparison to univariate analyses. Interestingly, the genomic prediction accuracies based on only highly significant SNPs (P < 0.00001, 318-400 SNPs for survival status and 1,362-1,589 SNPs for survival time) were somewhat lower (0.3-15.6%) than those obtained from the whole set of 6,470 SNPs. In most of our analyses, the accuracies of genomic prediction were somewhat higher for survival time than survival status (0/1 data). It is concluded that although there are prospects for the application of genomic selection to increase disease resistance to E. ictaluri in striped catfish breeding programs, further evaluation of these methods should be made in independent families/populations when more data are accumulated in future generations to avoid possible biases in the genetic parameters estimates and prediction accuracies for the disease-resistant traits studied in this population of striped catfish P. hypophthalmus.

Project description:ImportanceWhen confronted with new medical symptoms, many people turn to the internet to understand why they are ill as well as whether and where they should get care. Such searches may be harmful because they may facilitate misdiagnosis and inappropriate triage.ObjectiveTo empirically measure the association of an internet search for health information with diagnosis, triage, and anxiety by laypeople.Design, setting, and participantsThis survey study used a nationally representative sample of US adults who were recruited through an online platform between April 1, 2019, and April 15, 2019. A total of 48 validated case vignettes of both common (eg, viral illness) and severe (eg, heart attack) conditions were used. Participants were asked to relay their diagnosis, triage, and anxiety regarding 1 of these cases before and after searching the internet for health information.ExposuresShort, validated case vignettes written at or below the sixth-grade reading level randomly assigned to participants.Main outcomes and measuresCorrect diagnosis, correct triage, and flipping (changing) or anchoring (not changing) diagnosis and triage decisions were the main outcomes. Multivariable modeling was performed to identify patient factors associated with correct triage and diagnosis.ResultsOf the 5000 participants, 2549 were female (51.0%), 3819 were White (76.4%), and the mean (SD) age was 45.0 (16.9) years. Mean internet search time was 12.1 (95% CI, 10.7-13.5) minutes per case. No difference in triage accuracy was found before and after search (74.5% vs 74.1%; difference, -0.4 [95% CI, -1.4 to 0.6]; P = .06), but improved diagnostic accuracy was found (49.8% vs 54.0%; difference, 4.2% [95% CI, 3.1%-5.3%]; P < .001). Most participants (4254 [85.1%]) were anchored on their diagnosis. Of the 14.9% of participants (n = 746) who flipped their diagnosis, 9.6% (n = 478) flipped from incorrect to correct and 5.4% (n = 268) flipped from correct to incorrect. The following groups had an increased rate of correct diagnosis: adults 40 years or older (eg, 40-49 years: 5.1 [95% CI, 0.8-9.4] percentage points better than those aged <30 years; P = .02), women (9.4 [95% CI, 6.8-12.0] percentage points better than men; P < .001), and those with perceived poor health status (16.3 [95% CI, 6.9-25.6] percentage points better than those with excellent status; P = .001) and with more than 2 chronic diseases (6.8 [95% CI, 1.5-12.1] percentage points better than those with 0 conditions; P = .01).Conclusions and relevanceThis study found that an internet search for health information was associated with small increases in diagnostic accuracy but not with triage accuracy.

Dataset Information

Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions.

Publications

Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets