Dataset Information

GDC 2: Compression of large collections of genomes.

ABSTRACT: The fall of prices of the high-throughput genome sequencing changes the landscape of modern genomics. A number of large scale projects aimed at sequencing many human genomes are in progress. Genome sequencing also becomes an important aid in the personalized medicine. One of the significant side effects of this change is a necessity of storage and transfer of huge amounts of genomic data. In this paper we deal with the problem of compression of large collections of complete genomic sequences. We propose an algorithm that is able to compress the collection of 1092 human diploid genomes about 9,500 times. This result is about 4 times better than what is offered by the other existing compressors. Moreover, our algorithm is very fast as it processes the data with speed 200?MB/s on a modern workstation. In a consequence the proposed algorithm allows storing the complete genomic collections at low cost, e.g., the examined collection of 1092 human genomes needs only about 700?MB when compressed, what can be compared to about 6.7?TB of uncompressed FASTA files. The source code is available at http://sun.aei.polsl.pl/REFRESH/index.php?page=projects&project=gdc&subpage=about.

SUBMITTER: Deorowicz S

PROVIDER: S-EPMC4479802 | biostudies-other | 2015

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

GDC 2: Compression of large collections of genomes.

Deorowicz Sebastian S Danek Agnieszka A Niemiec Marcin M

Scientific reports 20150625

The fall of prices of the high-throughput genome sequencing changes the landscape of modern genomics. A number of large scale projects aimed at sequencing many human genomes are in progress. Genome sequencing also becomes an important aid in the personalized medicine. One of the significant side effects of this change is a necessity of storage and transfer of huge amounts of genomic data. In this paper we deal with the problem of compression of large collections of complete genomic sequences. We ...[more]

PMID: 26108279

Similar Datasets

Project description:BackgroundHigh Throughput Sequencing (HTS) is now heavily exploited for genome (re-) sequencing, metagenomics, epigenomics, and transcriptomics and requires different, but computer intensive bioinformatic analyses. When a reference genome is available, mapping reads on it is the first step of this analysis. Read mapping programs owe their efficiency to the use of involved genome indexing data structures, like the Burrows-Wheeler transform. Recent solutions index both the genome, and the k-mers of the reads using hash-tables to further increase efficiency and accuracy. In various contexts (e.g. assembly or transcriptome analysis), read processing requires to determine the sub-collection of reads that are related to a given sequence, which is done by searching for some k-mers in the reads. Currently, many developments have focused on genome indexing structures for read mapping, but the question of read indexing remains broadly unexplored. However, the increase in sequence throughput urges for new algorithmic solutions to query large read collections efficiently.ResultsHere, we present a solution, named Gk arrays, to index large collections of reads, an algorithm to build the structure, and procedures to query it. Once constructed, the index structure is kept in main memory and is repeatedly accessed to answer queries like "given a k-mer, get the reads containing this k-mer (once/at least once)". We compared our structure to other solutions that adapt uncompressed indexing structures designed for long texts and show that it processes queries fast, while requiring much less memory. Our structure can thus handle larger read collections. We provide examples where such queries are adapted to different types of read analysis (SNP detection, assembly, RNA-Seq).ConclusionsGk arrays constitute a versatile data structure that enables fast and more accurate read analysis in various contexts. The Gk arrays provide a flexible brick to design innovative programs that mine efficiently genomics, epigenomics, metagenomics, or transcriptomics reads. The Gk arrays library is available under Cecill (GPL compliant) license from http://www.atgc-montpellier.fr/ngs/.

Project description:AbstractThis study details a workflow used to accession a large stonefly (Plecoptera) collection resulting from several donations. The eastern North American material of Kenneth W. Stewart (deceased, University of North Texas), the entire collection of Stanley W. Szczytko (deceased, University of Wisconsin, Stevens Point), and a small portion of the Barry C. Poulton collection (active, United States Geological Survey, Columbia, Missouri) were donated to the Illinois Natural History Survey in 2013. These 5,767 vials of specimens were processed to help preserve the specimen legacy of these world renowned Plecoptera researchers. The workflow used an industrialized approach to organize the specimens taxonomically, image the specimens and labels, and place the specimens into new storage. Utilizing the images as a verbatim data source, we transcribed labels in iterative steps that yielded more information with each pass. The data were normalized, locations georeferenced, all specimen data formatted to meet Darwin Core Archive format for occurrence data, and a data set created using Pensoft's Integrated Publishing Toolkit. This is the first time that any of the specimen data has been made available electronically. We also provide two important electronic supplements that include the Bill P. Stark (active, Mississippi College) Oklahoma field notebook for 1971 and 1972 detailing locations for many coded stonefly specimens in the Stewart collection, and the coded locations of B. C. Poulton's Arkansas and Missouri study. Again, we have linked coded labels in vials to normalized and georefenced site data. We confirmed 243 stonefly species were contained within the collections, and the potential for many more species exists among the specimens identified to family and genus level. Twenty-one new state, province, and other significant stonefly records are reported herein with all identifications verified by the senior author, often through consultation with other stonefly taxonomists. Researchers are encouraged to utilize the specimen data, form collaborations with the authors, and borrow specimens for research.

Dataset Information

GDC 2: Compression of large collections of genomes.

Publications

GDC 2: Compression of large collections of genomes.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure