Dataset Information

CAGI5: Objective performance assessments of predictions based on the Evolutionary Action equation.

ABSTRACT: Many computational approaches estimate the effect of coding variants, but their predictions often disagree with each other. These contradictions confound users and raise questions regarding reliability. Performance assessments can indicate the expected accuracy for each method and highlight advantages and limitations. The Critical Assessment of Genome Interpretation (CAGI) community aims to organize objective and systematic assessments: They challenge predictors on unpublished experimental and clinical data and assign independent assessors to evaluate the submissions. We participated in CAGI experiments as predictors, using the Evolutionary Action (EA) method to estimate the fitness effect of coding mutations. EA is untrained, uses homology information, and relies on a formal equation: The fitness effect equals the functional sensitivity to residue changes multiplied by the magnitude of the substitution. In previous CAGI experiments (between 2011 and 2016), our submissions aimed to predict the protein activity of single mutants. In 2018 (CAGI5), we also submitted predictions regarding clinical associations, folding stability, and matching genomic data with phenotype. For all these diverse challenges, we used EA to predict the fitness effect of variants, adjusted to specifically address each question. Our submissions had consistently good performance, suggesting that EA predicts reliably the effects of genetic variants.

SUBMITTER: Katsonis P

PROVIDER: S-EPMC6900054 | biostudies-literature | 2019 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

CAGI5: Objective performance assessments of predictions based on the Evolutionary Action equation.

Katsonis Panagiotis P Lichtarge Olivier O

Human mutation 20190807 9

Many computational approaches estimate the effect of coding variants, but their predictions often disagree with each other. These contradictions confound users and raise questions regarding reliability. Performance assessments can indicate the expected accuracy for each method and highlight advantages and limitations. The Critical Assessment of Genome Interpretation (CAGI) community aims to organize objective and systematic assessments: They challenge predictors on unpublished experimental and c ...[more]

PMID: 31317604

Similar Datasets

Project description:Background:Digital biomarkers that measure physical activity and mobility are of great interest in the assessment of chronic diseases such as rheumatoid arthritis, as it provides insights on patients' quality of life that can be reliably compared across a whole population. Objective:To investigate the feasibility of analyzing iPhone sensor data collected remotely by means of a mobile software application in order to derive meaningful information on functional ability in rheumatoid arthritis patients. Methods:Two objective, active tasks were made available to the study participants: a wrist joint motion test and a walk test, both performed remotely and without any medical supervision. During these tasks, gyroscope and accelerometer time-series data were captured. Processing schemes were developed using machine learning techniques such as logistic regression as well as explicitly programmed algorithms to assess data quality in both tasks. Motion-specific features including wrist joint range of motion (ROM) in flexion-extension (for the wrist motion test) and gait parameters (for the walk test) were extracted from high quality data and compared with subjective pain and mobility parameters, separately captured via the application. Results:Out of 646 wrist joint motion samples collected, 289 (45%) were high quality. Data collected for the walk test included 2,583 samples (through 867 executions of the test) from which 651 (25%) were high quality. Further analysis of high-quality data highlighted links between reduced mobility and increased symptom severity. ANOVA testing showed statistically significant differences in wrist joint ROM between groups with light-moderate (220 participants) versus severe (36 participants) wrist pain (p < 0.001) as well as in average step times between groups with slight versus moderate problems walking about (p < 0.03). Conclusion:These findings demonstrate the potential to capture and quantify meaningful objective clinical information remotely using iPhone sensors and represent an early step towards the development of patient-centric digital endpoints for clinical trials in rheumatoid arthritis.

Project description:Data classification is a fundamental task in data mining. Within this field, the classification of multi-labeled data has been seriously considered in recent years. In such problems, each data entity can simultaneously belong to several categories. Multi-label classification is important because of many recent real-world applications in which each entity has more than one label. To improve the performance of multi-label classification, feature selection plays an important role. It involves identifying and removing irrelevant and redundant features that unnecessarily increase the dimensions of the search space for the classification problems. However, classification may fail with an extreme decrease in the number of relevant features. Thus, minimizing the number of features and maximizing the classification accuracy are two desirable but conflicting objectives in multi-label feature selection. In this article, we introduce a multi-objective optimization algorithm customized for selecting the features of multi-label data. The proposed algorithm is an enhanced variant of a decomposition-based multi-objective optimization approach, in which the multi-label feature selection problem is divided into single-objective subproblems that can be simultaneously solved using an evolutionary algorithm. This approach leads to accelerating the optimization process and finding more diverse feature subsets. The proposed method benefits from a local search operator to find better solutions for each subproblem. We also define a pool of genetic operators to generate new feature subsets based on old generation. To evaluate the performance of the proposed algorithm, we compare it with two other multi-objective feature selection approaches on eight real-world benchmark datasets that are commonly used for multi-label classification. The reported results of multi-objective method evaluation measures, such as hypervolume indicator and set coverage, illustrate an improvement in the results obtained by the proposed method. Moreover, the proposed method achieved better results in terms of classification accuracy with fewer features compared with state-of-the-art methods.

Project description:BackgroundResidency programs select medical students for interviews and employment using metrics such as the United States Medical Licensing Examination (USMLE) scores, grade-point average (GPA), and class rank/quartile. It is unclear whether these metrics predict performance as an intern. This study tested the hypothesis that performance on these metrics would predict intern performance.MethodsThis single institution, retrospective cohort analysis included 244 graduates from four classes (2015-2018) who completed an Accreditation Council for Graduate Medical Education (ACGME) certified internship and were evaluated by program directors (PDs) at the end of the year. PDs provided a global assessment rating and ratings addressing ACGME competencies (response rate = 47%) with five response options: excellent = 5, very good = 4, acceptable = 3, marginal = 2, unacceptable = 1. PDs also classified interns as outstanding = 4, above average = 3, average = 2, and below average = 1 relative to other interns from the same residency program. Mean USMLE scores (Step 1 and Step 2CK), third-year GPA, class rank, and core competency ratings were compared using Welch's ANOVA and follow-up pairwise t-tests.ResultsBetter performance on PD evaluations at the end of intern year was associated with higher USMLE Step 1 (p = 0.006), Step 2CK (p = 0.030), medical school GPA (p = 0.020) and class rank (p = 0.016). Interns rated as average had lower USMLE scores, GPA, and class rank than those rated as above average or outstanding; there were no significant differences between above average and outstanding interns. Higher rating in each of the ACGME core competencies was associated with better intern performance (p < 0.01).ConclusionsBetter performance as an intern was associated with higher USMLE scores, medical school GPA and class rank. When USMLE Step 1 reporting changes from numeric scores to pass/fail, residency programs can use other metrics to select medical students for interviews and employment.

Dataset Information

CAGI5: Objective performance assessments of predictions based on the Evolutionary Action equation.

Publications

CAGI5: Objective performance assessments of predictions based on the Evolutionary Action equation.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets