Dataset Information

Classification Accuracy of Neuroimaging Biomarkers in Attention-Deficit/Hyperactivity Disorder: Effects of Sample Size and Circular Analysis.

ABSTRACT: BACKGROUND:Motivated by an inconsistency between reports of high diagnosis-classification accuracies and known heterogeneity in attention-deficit/hyperactivity disorder (ADHD), this study assessed classification accuracy in studies of ADHD as a function of methodological factors that can bias results. We hypothesized that high classification results in ADHD diagnosis are inflated by methodological factors. METHODS:We reviewed 69 studies (of 95 studies identified) that used neuroimaging features to predict ADHD diagnosis. Based on reported methods, we assessed the prevalence of circular analysis, which inflates classification accuracy, and evaluated the relationship between sample size and accuracy to test if small-sample models tend to report higher classification accuracy, also an indicator of bias. RESULTS:Circular analysis was detected in 15.9% of ADHD classification studies, lack of independent test set was noted in 13%, and insufficient methodological detail to establish its presence was noted in another 11.6%. Accuracy of classification ranged from 60% to 80% in the 59.4% of reviewed studies that met criteria for independence of feature selection, model construction, and test datasets. Moreover, there was a negative relationship between accuracy and sample size, implying additional bias contributing to reported accuracies at lower sample sizes. CONCLUSIONS:High classification accuracies in neuroimaging studies of ADHD appear to be inflated by circular analysis and small sample size. Accuracies on independent datasets were consistent with known heterogeneity of the disorder. Steps to resolve these issues, and a shift toward accounting for sample heterogeneity and prediction of future outcomes, will be crucial in future classification studies in ADHD.

SUBMITTER: Pulini AA

PROVIDER: S-EPMC6310118 | biostudies-literature | 2019 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Classification Accuracy of Neuroimaging Biomarkers in Attention-Deficit/Hyperactivity Disorder: Effects of Sample Size and Circular Analysis.

Pulini Alfredo A AA Kerr Wesley T WT Loo Sandra K SK Lenartowicz Agatha A

Biological psychiatry. Cognitive neuroscience and neuroimaging 20180627 2

<h4>Background</h4>Motivated by an inconsistency between reports of high diagnosis-classification accuracies and known heterogeneity in attention-deficit/hyperactivity disorder (ADHD), this study assessed classification accuracy in studies of ADHD as a function of methodological factors that can bias results. We hypothesized that high classification results in ADHD diagnosis are inflated by methodological factors.<h4>Methods</h4>We reviewed 69 studies (of 95 studies identified) that used neuroim ...[more]

PMID: 30064848

Similar Datasets

Project description:BACKGROUND:A challenge facing clinical neuroscientists is how best to synthesize diverse and sometimes inconsistent evidence for neuropsychological deficits and brain system dysfunction found in psychiatric disorders into models that guide etiological and treatment research. Multiple-pathway models suggest that psychiatric symptoms might arise from pathophysiology in different neural systems. This study tested dual-pathway model predictions for attention-deficit/hyperactivity disorder (ADHD) that reward and executive function cognitive deficits should be related to abnormalities in corresponding functionally specialized neural systems. METHODS:Behavioral inhibition and preference for immediate rewards were assessed in N = 251 adolescent boys and girls ages 12 to 18 diagnosed with DSM-IV combined-subtype ADHD or non-ADHD control subjects. Following taxometric analyses of test performance, the resulting subgroups were compared on a functional magnetic resonance imaging monetary incentive delay task probing reward anticipation and go/no-go task of motor response inhibition. RESULTS:Three ADHD subgroups were identified consistent with different proposed pathways-ADHD with executive function/motor inhibition deficits, ADHD with both executive and reward deficits, and ADHD with relatively normal test performance. Each cognitive domain mapped to different ADHD brain dysfunction features as expected. However, no brain abnormalities were found common to all ADHD subgroups despite the fact they had nearly identical ADHD-related clinical characteristics. CONCLUSIONS:The results suggest that combined-subtype ADHD is a collection of discrete disorders for which a comparable behavioral end point arises through different neurobiological pathways. The findings raise caution about applying common cause, single-deficit conceptual models to individual ADHD patients and should prompt researchers to consider biologically defined, multifactorial etiological models for other psychiatric diagnoses.

Project description:In a recent review, it was suggested that much larger cohorts are needed to prove the diagnostic value of neuroimaging biomarkers in psychiatry. While within a sample, an increase of diagnostic accuracy of schizophrenia (SZ) with number of subjects (N) has been shown, the relationship between N and accuracy is completely different between studies. Using data from a recent meta-analysis of machine learning (ML) in imaging SZ, we found that while low-N studies can reach 90% and higher accuracy, above N/2?=?50 the maximum accuracy achieved steadily drops to below 70% for N/2?>?150. We investigate the role N plays in the wide variability in accuracy results in SZ studies (63-97%). We hypothesize that the underlying cause of the decrease in accuracy with increasing N is sample heterogeneity. While smaller studies more easily include a homogeneous group of subjects (strict inclusion criteria are easily met; subjects live close to study site), larger studies inevitably need to relax the criteria/recruit from large geographic areas. A SZ prediction model based on a heterogeneous group of patients with presumably a heterogeneous pattern of structural or functional brain changes will not be able to capture the whole variety of changes, thus being limited to patterns shared by most patients. In addition to heterogeneity (sample size), we investigate other factors influencing accuracy and introduce a ML effect size. We derive a simple model of how the different factors, such as sample heterogeneity and study setup determine this ML effect size, and explain the variation in prediction accuracies found from the literature, both in cross-validation and independent sample testing. From this, we argue that smaller-N studies may reach high prediction accuracy at the cost of lower generalizability to other samples. Higher-N studies, on the other hand, will have more generalization power, but at the cost of lower accuracy. In conclusion, when comparing results from different ML studies, the sample sizes should be taken into account. To assess the generalizability of the models, validation (by direct application) of the prediction models should be tested in independent samples. The prediction of more complex measures such as outcome, which are expected to have an underlying pattern of more subtle brain abnormalities (lower effect size), will require large samples.

Dataset Information

Classification Accuracy of Neuroimaging Biomarkers in Attention-Deficit/Hyperactivity Disorder: Effects of Sample Size and Circular Analysis.

Publications

Classification Accuracy of Neuroimaging Biomarkers in Attention-Deficit/Hyperactivity Disorder: Effects of Sample Size and Circular Analysis.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets