Dataset Information

A strategy for evaluating pathway analysis methods.

ABSTRACT: Researchers have previously developed a multitude of methods designed to identify biological pathways associated with specific clinical or experimental conditions of interest, with the aim of facilitating biological interpretation of high-throughput data. Before practically applying such pathway analysis (PA) methods, we must first evaluate their performance and reliability, using datasets where the pathways perturbed by the conditions of interest have been well characterized in advance. However, such 'ground truths' (or gold standards) are often unavailable. Furthermore, previous evaluation strategies that have focused on defining 'true answers' are unable to systematically and objectively assess PA methods under a wide range of conditions.In this work, we propose a novel strategy for evaluating PA methods independently of any gold standard, either established or assumed. The strategy involves the use of two mutually complementary metrics, recall and discrimination. Recall measures the consistency of the perturbed pathways identified by applying a particular analysis method to an original large dataset and those identified by the same method to a sub-dataset of the original dataset. In contrast, discrimination measures specificity-the degree to which the perturbed pathways identified by a particular method to a dataset from one experiment differ from those identifying by the same method to a dataset from a different experiment. We used these metrics and 24 datasets to evaluate six widely used PA methods. The results highlighted the common challenge in reliably identifying significant pathways from small datasets. Importantly, we confirmed the effectiveness of our proposed dual-metric strategy by showing that previous comparative studies corroborate the performance evaluations of the six methods obtained by our strategy.Unlike any previously proposed strategy for evaluating the performance of PA methods, our dual-metric strategy does not rely on any ground truth, either established or assumed, of the pathways perturbed by a specific clinical or experimental condition. As such, our strategy allows researchers to systematically and objectively evaluate pathway analysis methods by employing any number of datasets for a variety of conditions.

SUBMITTER: Yu C

PROVIDER: S-EPMC5640951 | biostudies-literature | 2017 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A strategy for evaluating pathway analysis methods.

Yu Chenggang C Woo Hyung Jun HJ Yu Xueping X Oyama Tatsuya T Wallqvist Anders A Reifman Jaques J

BMC bioinformatics 20171013 1

<h4>Background</h4>Researchers have previously developed a multitude of methods designed to identify biological pathways associated with specific clinical or experimental conditions of interest, with the aim of facilitating biological interpretation of high-throughput data. Before practically applying such pathway analysis (PA) methods, we must first evaluate their performance and reliability, using datasets where the pathways perturbed by the conditions of interest have been well characterized ...[more]

PMID: 29029625

Similar Datasets

Project description:BACKGROUND: The current paradigm of arthroscopic training lacks objective evaluation of technical ability and its adequacy is concerning given the accelerating complexity of the field. To combat insufficiencies, emphasis is shifting towards skill acquisition outside the operating room and sophisticated assessment tools. We reviewed (1) the validity of cadaver and surgical simulation in arthroscopic training, (2) the role of psychomotor analysis and arthroscopic technical ability, (3) what validated assessment tools are available to evaluate technical competency, and (4) the quantification of arthroscopic proficiency. METHODS: The Medline and Embase databases were searched for published articles in the English literature pertaining to arthroscopic competence, arthroscopic assessment and evaluation and objective measures of arthroscopic technical skill. Abstracts were independently evaluated and exclusion criteria included articles outside the scope of knee and shoulder arthroscopy as well as original articles about specific therapies, outcomes and diagnoses leaving 52 articles cited in this review. RESULTS: Simulated arthroscopic environments exhibit high levels of internal validity and consistency for simple arthroscopic tasks, however the ability to transfer complex skills to the operating room has not yet been established. Instrument and force trajectory data can discriminate between technical ability for basic arthroscopic parameters and may serve as useful adjuncts to more comprehensive techniques. There is a need for arthroscopic assessment tools for standardized evaluation and objective feedback of technical skills, yet few comprehensive instruments exist, especially for the shoulder. Opinion on the required arthroscopic experience to obtain proficiency remains guarded and few governing bodies specify absolute quantities. CONCLUSIONS: Further validation is required to demonstrate the transfer of complex arthroscopic skills from simulated environments to the operating room and provide objective parameters to base evaluation. There is a deficiency of validated assessment tools for technical competencies and little consensus of what constitutes a sufficient case volume within the arthroscopy community.

Project description:BackgroundFloating catchment methods have recently been applied to identify priority regions for Automated External Defibrillator (AED) deployment, to aid in improving Out of Hospital Cardiac Arrest (OHCA) survival. This approach models access as a supply-to-demand ratio for each area, targeting areas with high demand and low supply for AED placement. These methods incorporate spatial covariates on OHCA occurrence, but do not provide precise AED locations, which are critical to the initial intent of such location analysis research. Exact AED locations can be determined using optimisation methods, but they do not incorporate known spatial risk factors for OHCA, such as income and demographics. Combining these two approaches would evaluate AED placement impact, describe drivers of OHCA occurrence, and identify areas that may not be appropriately covered by AED placement strategies. There are two aims in this paper. First, to develop geospatial models of OHCA that account for and display uncertainty. Second, to evaluate the AED placement methods using geospatial models of accessibility. We first identify communities with the greatest gap between demand and supply for allocating AEDs. We then use this information to evaluate models for precise AED location deployment.MethodsCase study data set consisted of 2802 OHCA events and 719 AEDs. Spatial OHCA occurrence was described using a geospatial model, with possible spatial correlation accommodated by introducing a conditional autoregressive (CAR) prior on the municipality-level spatial random effect. This model was fit with Integrated Nested Laplacian Approximation (INLA), using covariates for population density, proportion male, proportion over 65 years, financial strength, and the proportion of land used for transport, commercial, buildings, recreation, and urban areas. Optimisation methods for AED locations were applied to find the top 100 AED placement locations. AED access was calculated for current access and 100 AED placements. Priority rankings were then given for each area based on their access score and predicted number of OHCA events.ResultsOf the 2802 OHCA events, 64.28% occurred in rural areas, and 35.72% in urban areas. Additionally, over 70% of individuals were aged over 65. Supply of AEDs was less than demand in most areas. Priority regions for AED placement were identified, and access scores were evaluated for AED placement methodology by ranking the access scores and the predicted OHCA count. AED placement methodology placed AEDs in areas with the highest priority, but placed more AEDs in areas with more predicted OHCA events in each grid cell.ConclusionThe methods in this paper incorporate OHCA spatial risk factors and OHCA coverage to identify spatial regions most in need of resources. These methods can be used to help understand how AED allocation methods affect OHCA accessibility, which is of significant practical value for communities when deciding AED placements.

Project description:It has been suggested that pathway analysis can complement single-SNP analysis in exploring genomewide association data. Pathway analysis incorporates the available biological knowledge of genes and SNPs and is expected to improve the chances of revealing the underlying genetic architecture of complex traits. Methods for pathway analysis can be classified as competitive (enrichment) or self-contained (association) according to the hypothesis tested. Although association tests are statistically more powerful than enrichment tests they can be difficult to calibrate because biases in analysis accumulate across multiple SNPs or genes. Furthermore, enrichment tests can be more scientifically relevant than association tests, as they detect pathways with relatively more evidence for association than the remaining genes. Here we show how some well known association tests can be simply adapted to test for enrichment, and compare their performance to some established enrichment tests. We propose versions of the Adaptive Rank Truncated Product (ARTP), Tail Strength Measure and Fisher's combination of p-values for testing the enrichment null hypothesis. We compare the behaviour of these proposed methods with the established Hypergeometric Test and Gene-Set Enrichment Analysis (GSEA). The results of the simulation study show that the modified version of the ARTP method has generally the best performance across the situations considered. The methods were also applied for finding enriched pathways for body mass index (BMI) and platelet function phenotypes. The pathway analysis of BMI identified the Vasoactive Intestinal Peptide pathway as significantly associated with BMI. This pathway has been previously reported as associated with BMI and the risk of obesity. The ARTP method was the method that identified the largest number of enriched pathways across all tested pathway databases and phenotypes. The simulation and data application results are in agreement with previous work on association tests and suggests that the ARTP should be preferred for both enrichment and association testing.

Project description:Per- and polyfluoroalkyl substances (PFAS) are a very large (thousands of chemicals) category; these substances have important industrial and consumer product applications. PFAS are highly persistent in the environment, and some are known to pose human health hazard. Regulatory agencies worldwide are considering restrictions and outright bans of PFAS; however, little data exists to make informed decisions. Therefore, a prioritization strategy is urgently needed for evaluation of potential hazards of PFAS. Structure-based grouping could expedite selection of PFAS for testing; still, the hypothesis that structure-effect relationships exist requires confirmation. We tested 26 structurally diverse PFAS from 8 groups in two human cell types from organs that are thought to be targets for PFAS. We used human induced pluripotent stem cell-derived hepatocytes and cardiomyocytes and tested concentration-response effects on both cell function and gene expression. Few phenotypic effects were observed in hepatocytes, but negative chronotropy was observed for 8 of 26 PFAS. Substance- and cell-dependent transcriptomic changes were more pronounced; however, little evidence of group-specific effects was observed. In hepatocytes, we found up-regulation of stress-related and extracellular matrix organization pathways, and down-regulation of fat metabolism. In cardiomyocytes, contractility-related pathways were most affected. Using these data, we derived phenotypic and transcriptomic point of departure estimates and compared them to predicted PFAS exposures. The conservative estimates for bioactivity and exposure were used to derive margin-of-exposure (MOE) for each PFAS. We found that most (23 of 26) PFAS had MOE>1. Overall, our data suggests that chemical structure-based grouping of PFAS may not be an appropriate strategy to predict their biological effects. This means that testing of the individual PFAS would be needed for confident decision-making. Our proposed strategy of using two human cell types and considering both phenotypic and transcriptomic effects, combined with dose-response analysis, may be used for prioritization of PFAS.

Dataset Information

A strategy for evaluating pathway analysis methods.

Publications

A strategy for evaluating pathway analysis methods.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets