Browse
Submit Data
Databases
API
Help

Dataset Information

37 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

CARGO: effective format-free compressed storage of genomic information.

ABSTRACT: The recent super-exponential growth in the amount of sequencing data generated worldwide has put techniques for compressed storage into the focus. Most available solutions, however, are strictly tied to specific bioinformatics formats, sometimes inheriting from them suboptimal design choices; this hinders flexible and effective data sharing. Here, we present CARGO (Compressed ARchiving for GenOmics), a high-level framework to automatically generate software systems optimized for the compressed storage of arbitrary types of large genomic data collections. Straightforward applications of our approach to FASTQ and SAM archives require a few lines of code, produce solutions that match and sometimes outperform specialized format-tailored compressors and scale well to multi-TB datasets. All CARGO software components can be freely downloaded for academic and non-commercial use from http://bio-cargo.sourceforge.net.

SUBMITTER: Roguski L

PROVIDER: S-EPMC4937321 | biostudies-literature | 2016 Jul

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

CARGO: effective format-free compressed storage of genomic information.

Roguski Łukasz Ł Ribeca Paolo P

Nucleic acids research 20160429 12

The recent super-exponential growth in the amount of sequencing data generated worldwide has put techniques for compressed storage into the focus. Most available solutions, however, are strictly tied to specific bioinformatics formats, sometimes inheriting from them suboptimal design choices; this hinders flexible and effective data sharing. Here, we present CARGO (Compressed ARchiving for GenOmics), a high-level framework to automatically generate software systems optimized for the compressed s ...[more]

PMID: 27131376

Similar Datasets

A privacy-preserving solution for compressed storage and selective retrieval of genomic data.

Project description:In clinical genomics, the continuous evolution of bioinformatic algorithms and sequencing platforms makes it beneficial to store patients' complete aligned genomic data in addition to variant calls relative to a reference sequence. Due to the large size of human genome sequence data files (varying from 30 GB to 200 GB depending on coverage), two major challenges facing genomics laboratories are the costs of storage and the efficiency of the initial data processing. In addition, privacy of genomic data is becoming an increasingly serious concern, yet no standard data storage solutions exist that enable compression, encryption, and selective retrieval. Here we present a privacy-preserving solution named SECRAM (Selective retrieval on Encrypted and Compressed Reference-oriented Alignment Map) for the secure storage of compressed aligned genomic data. Our solution enables selective retrieval of encrypted data and improves the efficiency of downstream analysis (e.g., variant calling). Compared with BAM, the de facto standard for storing aligned genomic data, SECRAM uses 18% less storage. Compared with CRAM, one of the most compressed nonencrypted formats (using 34% less storage than BAM), SECRAM maintains efficient compression and downstream data processing, while allowing for unprecedented levels of security in genomic data storage. Compared with previous work, the distinguishing features of SECRAM are that (1) it is position-based instead of read-based, and (2) it allows random querying of a subregion from a BAM-like file in an encrypted form. Our method thus offers a space-saving, privacy-preserving, and effective solution for the storage of clinical genomic data.

| S-EPMC5131820 | biostudies-literature

Terminator-free template-independent enzymatic DNA synthesis for digital information storage.

Project description:DNA is an emerging medium for digital data and its adoption can be accelerated by synthesis processes specialized for storage applications. Here, we describe a de novo enzymatic synthesis strategy designed for data storage which harnesses the template-independent polymerase terminal deoxynucleotidyl transferase (TdT) in kinetically controlled conditions. Information is stored in transitions between non-identical nucleotides of DNA strands. To produce strands representing user-defined content, nucleotide substrates are added iteratively, yielding short homopolymeric extensions whose lengths are controlled by apyrase-mediated substrate degradation. With this scheme, we synthesize DNA strands carrying 144 bits, including addressing, and demonstrate retrieval with streaming nanopore sequencing. We further devise a digital codec to reduce requirements for synthesis accuracy and sequencing coverage, and experimentally show robust data retrieval from imperfectly synthesized strands. This work provides distributive enzymatic synthesis and information-theoretic approaches to advance digital information storage in DNA.

| S-EPMC6546792 | biostudies-literature

Why is adiabatic compressed air energy storage yet to become a viable energy storage option?

Project description:Recent theoretical studies have predicted that adiabatic compressed air energy storage (ACAES) can be an effective energy storage option in the future. However, major experimental projects and commercial ventures have so far failed to yield any viable prototypes. Here we explore the underlying reasons behind this failure. By developing an analytical idealized model of a typical ACAES design, we derive a design-dependent efficiency limit for a system with hypothetical, perfect components. This previously overlooked limit, equal to 93.6% under continuous cycling for a typical design, arises from irreversibility associated with the transient pressure in the system. Although the exact value is design dependent, the methodology we present for finding the limit is applicable for a wide range of designs. Turning to real systems, the limit alone does not fully explain the failure of practical ACAES research. However, reviewing the available evidence alongside our analytical model, we reason that underestimation of the system complexity, difficulty with the integration of off-the-shelf components, and a number of misleading performance claims are the primary reasons hindering ACAES development.

| S-EPMC8111675 | biostudies-literature

Using information of relatives in genomic prediction to apply effective stratified medicine.

Project description:Genomic prediction shows promise for personalised medicine in which diagnosis and treatment are tailored to individuals based on their genetic profiles for complex diseases. We present a theoretical framework to demonstrate that prediction accuracy can be improved by targeting more informative individuals in the data set used to generate the predictors ("discovery sample") to include those with genetically close relationships with the subjects put forward for risk prediction. Increase of prediction accuracy from closer relationships is achieved under an additive model and does not rely on any family or interaction effects. Using theory, simulations and real data analyses, we show that the predictive accuracy or the area under the receiver operating characteristic curve (AUC) increased exponentially with decreasing effective size (Ne), i.e. when individuals are closely related. For example, with the sample size of discovery set N = 3000, heritability h2 = 0.5 and population prevalence K = 0.1, AUC value approached to 0.9 and the top percentile of the estimated genetic profile scores had 23 times higher proportion of cases than the general population. This suggests that there is considerable room to increase prediction accuracy by using a design that does not exclude closer relationships.

| S-EPMC5299615 | biostudies-literature

Toward highly effective loading of DNA in hydrogels for high-density and long-term information storage.

Project description:Digital information, when converted into a DNA sequence, provides dense, stable, energy-efficient, and sustainable data storage. The most stable method for encapsulating DNA has been in an inorganic matrix of silica, iron oxide, or both, but are limited by low DNA uptake and complex recovery techniques. This study investigated a rationally designed thermally responsive functionally graded (TRFG) hydrogel as a simple and cost-effective method for storing DNA. The TRFG hydrogel shows high DNA uptake, long-term protection, and reusability due to nondestructive DNA extraction. The high loading capacity was achieved by directly absorbing DNA from the solution, which is then retained because of its interaction with a hyperbranched cationic polymer loaded into a negatively charged hydrogel matrix used as a support and because of its thermoresponsive nature, which allows DNA concentration within the hydrogel through multiple swelling/deswelling cycles. We were able to achieve a high DNA data density of 7.0 × 109 gigabytes per gram using a hydrogel-based system.

| S-EPMC10171811 | biostudies-literature

Electrical energy storage using compressed gas in depleted hydraulically fractured wells.

Project description:Renewable forms of electricity generation like solar and wind require low-cost energy storage solutions to meet climate change deployment goals. Here, we explore the use of depleted hydraulically fractured ("fracked") oil and gas wells to store electrical energy in the form of compressed natural gas to be released to spin an expander/generator when electrical demand is high. Our reservoir model indicates that the same dual-porosity geological environment of fracked wells used to liberate hydrocarbons is also suitable for storing and releasing gas in a diurnal or seasonal cycle. Round-trip storage efficiency is calculated to be 40%-70% depending on the natural reservoir temperature. Levelized cost of storage is estimated to be $70-270/MWh, on par with pumped hydro storage. This study indicates that repurposed "fracked" wells could provide a much-needed low-cost seasonal energy storage solution at the TWh scale.

| S-EPMC8640470 | biostudies-literature

A new type of bladeless turbine for compressed gas energy storage system

Project description:Nowadays, energy storage engineering is an important means to relieve the problem of energy shortage. In this investigation, we design a kind of vaneless turbine originating from a Tesla turbine with a diameter and an air gap of 250 mm and 0.5 mm, respectively. Importantly, such a vaneless turbine removes the feature of an air outlet in the middle and adopts other ways of entering and leaving on both sides, so as to strengthen the rotor, because there is no need for a large hole in the middle of the rotor. Furthermore, the rotor dynamics characteristics of the vaneless turbine are calculated by six different modes. We also obtain the critical speed in different modes. Moreover, the flow field performances, such as the velocity and pressure of fluid (air), are investigated using the finite element simulation method. In addition, the bench test is built to obtain the output characteristics of a vaneless turbine. The maximum output torque is about 5.56 Nm at 992 rpm, and the maximal rotational speed of the vaneless turbine can reach 3200 rpm. Our work provides new ideas and guidance for the design and research of the new generation of the vaneless turbine.

| S-EPMC9548558 | biostudies-literature

GTRAC: fast retrieval from compressed collections of genomic variants.

Project description:MotivationThe dramatic decrease in the cost of sequencing has resulted in the generation of huge amounts of genomic data, as evidenced by projects such as the UK10K and the Million Veteran Project, with the number of sequenced genomes ranging in the order of 10 K to 1 M. Due to the large redundancies among genomic sequences of individuals from the same species, most of the medical research deals with the variants in the sequences as compared with a reference sequence, rather than with the complete genomic sequences. Consequently, millions of genomes represented as variants are stored in databases. These databases are constantly updated and queried to extract information such as the common variants among individuals or groups of individuals. Previous algorithms for compression of this type of databases lack efficient random access capabilities, rendering querying the database for particular variants and/or individuals extremely inefficient, to the point where compression is often relinquished altogether.ResultsWe present a new algorithm for this task, called GTRAC, that achieves significant compression ratios while allowing fast random access over the compressed database. For example, GTRAC is able to compress a Homo sapiens dataset containing 1092 samples in 1.1 GB (compression ratio of 160), while allowing for decompression of specific samples in less than a second and decompression of specific variants in 17 ms. GTRAC uses and adapts techniques from information theory, such as a specialized Lempel-Ziv compressor, and tailored succinct data structures.Availability and implementationThe GTRAC algorithm is available for download at: https://github.com/kedartatwawadi/GTRAC CONTACT: : kedart@stanford.eduSupplementary informationSupplementary data are available at Bioinformatics online.

| S-EPMC5013914 | biostudies-literature

SeqArray-a storage-efficient high-performance data format for WGS variant calls.

Project description:Motivation:Whole-genome sequencing (WGS) data are being generated at an unprecedented rate. Analysis of WGS data requires a flexible data format to store the different types of DNA variation. Variant call format (VCF) is a general text-based format developed to store variant genotypes and their annotations. However, VCF files are large and data retrieval is relatively slow. Here we introduce a new WGS variant data format implemented in the R/Bioconductor package 'SeqArray' for storing variant calls in an array-oriented manner which provides the same capabilities as VCF, but with multiple high compression options and data access using high-performance parallel computing. Results:Benchmarks using 1000 Genomes Phase 3 data show file sizes are 14.0?Gb (VCF), 12.3?Gb (BCF, binary VCF), 3.5?Gb (BGT) and 2.6?Gb (SeqArray) respectively. Reading genotypes in the SeqArray package are two to three times faster compared with the htslib C library using BCF files. For the allele frequency calculation, the implementation in the SeqArray package is over 5 times faster than PLINK v1.9 with VCF and BCF files, and over 16 times faster than vcftools. When used in conjunction with R/Bioconductor packages, the SeqArray package provides users a flexible, feature-rich, high-performance programming environment for analysis of WGS variant data. Availability and Implementation:http://www.bioconductor.org/packages/SeqArray. Contact:zhengx@u.washington.edu. Supplementary information:Supplementary data are available at Bioinformatics online.

| S-EPMC5860110 | biostudies-literature

Scool: a new data storage format for single-cell Hi-C data.

Project description:MotivationSingle-cell Hi-C research currently lacks an efficient, easy to use and shareable data storage format. Recent studies have used a variety of sub-optimal solutions: publishing raw data only, text-based interaction matrices, or reusing established Hi-C storage formats for single interaction matrices. These approaches are storage and pre-processing intensive, require long labour time and are often error-prone.ResultsThe single-cell cooler file format (scool) provides an efficient, user-friendly and storage-saving approach for single-cell Hi-C data. It is a flavour of the established cooler format and guarantees stable API support.Availability and implementationThe single-cell cooler format is part of the cooler file format as of API version 0.8.9. It is available via pip, conda and github: https://github.com/mirnylab/cooler.Supplementary informationSupplementary data are available at Bioinformatics online.

| S-EPMC8337000 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data