Dataset Information

Biospark: scalable analysis of large numerical datasets from biological simulations and experiments using Hadoop and Spark.

ABSTRACT: Data-parallel programming techniques can dramatically decrease the time needed to analyze large datasets. While these methods have provided significant improvements for sequencing-based analyses, other areas of biological informatics have not yet adopted them. Here, we introduce Biospark, a new framework for performing data-parallel analysis on large numerical datasets. Biospark builds upon the open source Hadoop and Spark projects, bringing domain-specific features for biology. AVAILABILITY AND IMPLEMENTATION:Source code is licensed under the Apache 2.0 open source license and is available at the project website: https://www.assembla.com/spaces/roberts-lab-public/wiki/Biospark CONTACT: eroberts@jhu.eduSupplementary information: Supplementary data are available at Bioinformatics online.

SUBMITTER: Klein M

PROVIDER: S-EPMC6276899 | biostudies-literature | 2017 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Biospark: scalable analysis of large numerical datasets from biological simulations and experiments using Hadoop and Spark.

Klein Max M Sharma Rati R Bohrer Chris H CH Avelis Cameron M CM Roberts Elijah E

Bioinformatics (Oxford, England) 20160922 2

Data-parallel programming techniques can dramatically decrease the time needed to analyze large datasets. While these methods have provided significant improvements for sequencing-based analyses, other areas of biological informatics have not yet adopted them. Here, we introduce Biospark, a new framework for performing data-parallel analysis on large numerical datasets. Biospark builds upon the open source Hadoop and Spark projects, bringing domain-specific features for biology.<h4>Availability ...[more]

PMID: 27663493

Similar Datasets

Project description:Bacteria need to survive in a wide range of environments. Currently, there is an incomplete understanding of the genetic basis for mechanisms underpinning survival in stressful conditions, such as the presence of anti-microbials. Transposon directed insertion-site sequencing (TraDIS) is a powerful tool to identify genes and networks which are involved in survival and fitness under a given condition by simultaneously assaying the fitness of millions of mutants, thereby relating genotype to phenotype and contributing to an understanding of bacterial cell biology. A recent refinement of this approach allows the roles of essential genes in conditional stress survival to be inferred by altering their expression. These advancements combined with the rapidly falling costs of sequencing now allows comparisons between multiple experiments to identify commonalities in stress responses to different conditions. This capacity however poses a new challenge for analysis of multiple data sets in conjunction. To address this analysis need, we have developed 'AlbaTraDIS'; a software application for rapid large-scale comparative analysis of TraDIS experiments that predicts the impact of transposon insertions on nearby genes. AlbaTraDIS can identify genes which are up or down regulated, or inactivated, between multiple conditions, producing a filtered list of genes for further experimental validation as well as several accompanying data visualisations. We demonstrate the utility of our new approach by applying it to identify genes used by Escherichia coli to survive in a wide range of different concentrations of the biocide Triclosan. AlbaTraDIS identified all well characterised Triclosan resistance genes, including the primary target, fabI. A number of new loci were also implicated in Triclosan resistance and the predicted phenotypes for a selection of these were validated experimentally with results being consistent with predictions. AlbaTraDIS provides a simple and rapid method to analyse multiple transposon mutagenesis data sets allowing this technology to be used at large scale. To our knowledge this is the only tool currently available that can perform these tasks. AlbaTraDIS is written in Python 3 and is available under the open source licence GNU GPL 3 from https://github.com/quadram-institute-bioscience/albatradis.

Dataset Information

Biospark: scalable analysis of large numerical datasets from biological simulations and experiments using Hadoop and Spark.

Publications

Biospark: scalable analysis of large numerical datasets from biological simulations and experiments using Hadoop and Spark.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets