Dataset Information

Detecting the impact of subject characteristics on machine learning-based diagnostic applications.

ABSTRACT: Collection of high-dimensional, longitudinal digital health data has the potential to support a wide-variety of research and clinical applications including diagnostics and longitudinal health tracking. Algorithms that process these data and inform digital diagnostics are typically developed using training and test sets generated from multiple repeated measures collected across a set of individuals. However, the inclusion of repeated measurements is not always appropriately taken into account in the analytical evaluations of predictive performance. The assignment of repeated measurements from each individual to both the training and the test sets ("record-wise" data split) is a common practice and can lead to massive underestimation of the prediction error due to the presence of "identity confounding." In essence, these models learn to identify subjects, in addition to diagnostic signal. Here, we present a method that can be used to effectively calculate the amount of identity confounding learned by classifiers developed using a record-wise data split. By applying this method to several real datasets, we demonstrate that identity confounding is a serious issue in digital health studies and that record-wise data splits for machine learning- based applications need to be avoided.

SUBMITTER: Chaibub Neto E

PROVIDER: S-EPMC6789029 | biostudies-literature | 2019

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Detecting the impact of subject characteristics on machine learning-based diagnostic applications.

Chaibub Neto Elias E Pratap Abhishek A Perumal Thanneer M TM Tummalacherla Meghasyam M Snyder Phil P Bot Brian M BM Trister Andrew D AD Friend Stephen H SH Mangravite Lara L Omberg Larsson L

NPJ digital medicine 20191011

Collection of high-dimensional, longitudinal digital health data has the potential to support a wide-variety of research and clinical applications including diagnostics and longitudinal health tracking. Algorithms that process these data and inform digital diagnostics are typically developed using training and test sets generated from multiple repeated measures collected across a set of individuals. However, the inclusion of repeated measurements is not always appropriately taken into account in ...[more]

PMID: 31633058

Similar Datasets

Project description:Background:To evaluate the association of multiparametric and multiregional MRI-features with key molecular characteristics in patients with newly-diagnosed glioblastoma. Methods:Retrospective data evaluation was approved by the local ethics committee of the University of Heidelberg (ethics approval number: S-320/2012) and informed consent was waived. Preoperative MRI-features were correlated with key molecular characteristics within a single-institutional cohort of 152 patients with newly-diagnosed glioblastoma. Preoperative MRI-features (n=31) included multiparametric (anatomical, diffusion-, perfusion-, and susceptibility-weighted images) and multiregional (contrast enhancing and non-enhancing FLAIR-hyperintense) information with (histogram) quantification of tumor volumes, volume ratios, apparent diffusion coefficients, cerebral blood flow / volume (CBF / CBV) and intratumoral susceptibility signals. Molecular characteristics determined with the Illumina Infinium HumanMethylation450 array included global DNA-methylation subgroups (e.g. mesenchymal (MES), RTK I “PGFRA”, RTK II “classic”), MGMT-promoter methylation status and hallmark copy-number-variations (EGFR-, PDGFRA-, MDM4- and CDK4-amplification; PTEN-, CDKN2A-, NF1- and RB1-loss). Univariate analyses (voxel-lesion-symptom-mapping for tumor location, Wilcoxon-test for all other MRI-features) as well as machine-learning models were applied to study the strength of association and discriminative value of MRI-features for predicting underlying molecular characteristics. Results: There was no tumor location predilection for any of the assessed molecular parameters (permutation-adjusted p>0.05 each). Univariate imaging parameter associations were noted for EGFR amplification and CDKN2A loss, both demonstrating increased nrCBV and nrCBF values (performance of these parameters, as assessed by the area under the ROC curve ranged from 63 to 69%, FDR-adjusted p<0.05, respectively). Subjecting all MRI-features to machine-learning-based classification allowed to predict EGFR amplification status and the RTK II “classic” GB subgroup with a moderate, yet significantly greater accuracy (63% for EGFR [p<0.01] and 61% for RTK II [p=0.01]) than the prediction by chance, whereas prediction accuracy for all other molecular parameters was non-significant (p>0.05, all models). Conclusions: In summary, we found univariate associations between established MRI-features and molecular characteristics, however not of sufficient strength to allow the generation of machine-learning classification models for reliable and clinically meaningful prediction of the assessed molecular characteristics in patients with newly-diagnosed glioblastoma.

Project description:BackgroundDeveloping machine learning models to support health analytics requires increased understanding about statistical properties of self-rated expression statements used in health-related communication and decision making. To address this, our current research analyzes self-rated expression statements concerning the coronavirus COVID-19 epidemic and with a new methodology identifies how statistically significant differences between groups of respondents can be linked to machine learning results.MethodsA quantitative cross-sectional study gathering the "need for help" ratings for twenty health-related expression statements concerning the coronavirus epidemic on an 11-point Likert scale, and nine answers about the person's health and wellbeing, sex and age. The study involved online respondents between 30 May and 3 August 2020 recruited from Finnish patient and disabled people's organizations, other health-related organizations and professionals, and educational institutions (n = 673). We propose and experimentally motivate a new methodology of influence analysis concerning machine learning to be applied for evaluating how machine learning results depend on and are influenced by various properties of the data which are identified with traditional statistical methods.ResultsWe found statistically significant Kendall rank-correlations and high cosine similarity values between various health-related expression statement pairs concerning the "need for help" ratings and a background question pair. With tests of Wilcoxon rank-sum, Kruskal-Wallis and one-way analysis of variance (ANOVA) between groups we identified statistically significant rating differences for several health-related expression statements in respect to groupings based on the answer values of background questions, such as the ratings of suspecting to have the coronavirus infection and having it depending on the estimated health condition, quality of life and sex. Our new methodology enabled us to identify how statistically significant rating differences were linked to machine learning results thus helping to develop better human-understandable machine learning models.ConclusionsThe self-rated "need for help" concerning health-related expression statements differs statistically significantly depending on the person's background information, such as his/her estimated health condition, quality of life and sex. With our new methodology statistically significant rating differences can be linked to machine learning results thus enabling to develop better machine learning to identify, interpret and address the patient's needs for well-personalized care.

Project description:Knee osteoarthritis (KOA) is characterized by pain and decreased gait function. We aimed to find KOA-related gait features based on patient reported outcome measures (PROMs) and develop regression models using machine learning algorithms to estimate KOA severity. The study included 375 volunteers with variable KOA grades. The severity of KOA was determined using the Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC). WOMAC scores were used to classify disease severity into three groups. A total of 1087 features were extracted from the gait data. An ANOVA and student's t-test were performed and only features that were significant were selected for inclusion in the machine learning algorithm. Three WOMAC subscales (physical function, pain and stiffness) were further divided into three classes. An ANOVA was performed to determine which selected features were significantly related to the subscales. Both linear regression models and a random forest regression was used to estimate patient the WOMAC scores. Forty-three features were selected based on ANOVA and student's t-test results. The following number of features were selected from each joint: 12 from hip, 1 feature from pelvic, 17 features from knee, 9 features from ankle, 1 feature from foot, and 3 features from spatiotemporal parameters. A significance level of < 0.0001 and < 0.00003 was set for the ANOVA and t-test, respectively. The physical function, pain, and stiffness subscales were related to 41, 10, and 16 features, respectively. Linear regression models showed a correlation of 0.723 and the machine learning algorithm showed a correlation of 0.741. The severity of KOA was predicted by gait analysis features, which were incorporated to develop an objective estimation model for KOA severity. The identified features may serve as a tool to guide rehabilitation and progress assessments. In addition, the estimation model presented here suggests an approach for clinical application of gait analysis data for KOA evaluation.

Dataset Information

Detecting the impact of subject characteristics on machine learning-based diagnostic applications.

Publications

Detecting the impact of subject characteristics on machine learning-based diagnostic applications.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets