DigestiFlow: from BCL to FASTQ with ease
Ontology highlight
ABSTRACT: Summary
Management of raw-sequencing data and its pre-processing (conversion into sequences and demultiplexing) remains a challenging topic for groups running sequencing devices. They face many challenges in such efforts and solutions ranging from manual management of spreadsheets to very complex and customized laboratory information management systems handling much more than just sequencing raw data. In this article, we describe the software package DigestiFlow that focuses on the management of Illumina flow cell sample sheets and raw data. It allows for automated extraction of information from flow cell data and management of sample sheets. Furthermore, it allows for the automated and reproducible conversion of Illumina base calls to sequences and the demultiplexing thereof using bcl2fastq and Picard Tools, followed by quality control report generation. Availability and implementation
The software is available under the MIT license at https://github.com/bihealth/digestiflow-server. The client software components are available via Bioconda. Supplementary information
Supplementary data are available at Bioinformatics online.
SUBMITTER: Holtgrewe M
PROVIDER: S-EPMC7703778 | biostudies-literature | 2019 Nov
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:FASTQ has emerged as a common file format for sharing sequencing read data combining both the sequence and an associated per base quality score, despite lacking any formal definition to date, and existing in at least three incompatible variants. This article defines the FASTQ format, covering the original Sanger standard, the Solexa/Illumina variants and conversion between them, based on publicly available information such as the MAQ documentation and conventions recently agreed by the Open Bioinformatics Foundation projects Biopython, BioPerl, BioRuby, BioJava and EMBOSS. Being an open access publication, it is hoped that this description, with the example files provided as Supplementary Data, will serve in future as a reference for this important file format.
| S-EPMC2847217 | biostudies-literature
Project description:BackgroundDue to the technological progress in Next Generation Sequencing (NGS), the amount of genomic data that is produced daily has seen a tremendous increase. This increase has shifted the bottleneck of genomic projects from sequencing to computation and specifically storing, managing and analyzing the large amount of NGS data. Compression tools can reduce the physical storage used to save large amount of genomic data as well as the bandwidth used to transfer this data. Recently, DNA sequence compression has gained much attention among researchers.ResultsIn this paper, we study different techniques and algorithms used to compress genomic data. Most of these techniques take advantage of some properties that are unique to DNA sequences in order to improve the compression rate, and usually perform better than general-purpose compressors. By exploring the performance of available algorithms, we produce a powerful compression tool for NGS data called MZPAQ. Results show that MZPAQ outperforms state-of-the-art tools on all benchmark datasets obtained from a recent survey in terms of compression ratio. MZPAQ offers the best compression ratios regardless of the sequencing platform or the size of the data.ConclusionsCurrently, MZPAQ's strength is its higher compression ratio as well as its compatibility with all major sequencing platforms. MZPAQ is more suitable when the size of compressed data is crucial, such as long-term storage and data transfer. More efforts will be made in the future to target other aspects such as compression speed and memory utilization.
| S-EPMC6547476 | biostudies-literature
Project description:Storage and transmission of the data produced by modern DNA sequencing instruments has become a major concern, which prompted the Pistoia Alliance to pose the SequenceSqueeze contest for compression of FASTQ files. We present several compression entries from the competition, Fastqz and Samcomp/Fqzcomp, including the winning entry. These are compared against existing algorithms for both reference based compression (CRAM, Goby) and non-reference based compression (DSRC, BAM) and other recently published competition entries (Quip, SCALCE). The tools are shown to be the new Pareto frontier for FASTQ compression, offering state of the art ratios at affordable CPU costs. All programs are freely available on SourceForge. Fastqz: https://sourceforge.net/projects/fastqz/, fqzcomp: https://sourceforge.net/projects/fqzcomp/, and samcomp: https://sourceforge.net/projects/samcomp/.
| S-EPMC3606433 | biostudies-literature
Project description:BackgroundThe exponential growth of next generation sequencing (NGS) data has posed big challenges to data storage, management and archive. Data compression is one of the effective solutions, where reference-based compression strategies can typically achieve superior compression ratios compared to the ones not relying on any reference.ResultsThis paper presents a lossless light-weight reference-based compression algorithm namely LW-FQZip to compress FASTQ data. The three components of any given input, i.e., metadata, short reads and quality score strings, are first parsed into three data streams in which the redundancy information are identified and eliminated independently. Particularly, well-designed incremental and run-length-limited encoding schemes are utilized to compress the metadata and quality score streams, respectively. To handle the short reads, LW-FQZip uses a novel light-weight mapping model to fast map them against external reference sequence(s) and produce concise alignment results for storage. The three processed data streams are then packed together with some general purpose compression algorithms like LZMA. LW-FQZip was evaluated on eight real-world NGS data sets and achieved compression ratios in the range of 0.111-0.201. This is comparable or superior to other state-of-the-art lossless NGS data compression algorithms.ConclusionsLW-FQZip is a program that enables efficient lossless FASTQ data compression. It contributes to the state of art applications for NGS data storage and transmission. LW-FQZip is freely available online at: http://csse.szu.edu.cn/staff/zhuzx/LWFQZip.
| S-EPMC4459677 | biostudies-literature
Project description:This repository contains all the FASTQ files for the five data modalities (scRNA-seq, scATAC-seq, Multiome, CITE-seq+scVDJ-seq, and spatial transcriptomics) used in the article \\"An Atlas of Cells in The Human Tonsil,\\" published in Immunity in 2024. Inspired by the TCGA barcodes, we have named each fastq file with the following convention: [TECHNOLOGY].[DONOR_ID].[SUBPROJECT].[GEM_ID].[LIBRARY_ID].[LIBRARY_TYPE].[LANE].[READ].fastq.gz which allows to retrieve all metadata from the name itself. Here is a full description of each field: - TECHNOLOGY: scRNA-seq, scATAC-seq, Multiome, CITE-seq+scVDJ-seq, and spatial transcriptomics (Visium). We also include the fastq files associated with the multiome experiments performed on two mantle cell lymphoma patients (MCL). - DONOR_ID: identifier for each of the 17 patients included in the cohort. We provide the donor-level metadata in the file \\"tonsil_atlas_donor_metadata.csv\\", including the hospital, sex, age, age group, cause for tonsillectomy and cohort type for every donor. - SUBPROJECT: each subproject corresponds to one run of the 10x Genomics Chromium™ Chip. - GEM_ID: each run of the 10x Genomics Chromium™ Chip consists of up to 8 \\"GEM wells\\" (see https://www.10xgenomics.com/support/software/cell-ranger/getting-started/cr-glossary): a set of partitioned cells (Gel Beads-in-emulsion) from a single 10x Genomics Chromium™ Chip channel. We give a unique identifier to each of these channels. - LIBRARY_ID: one or more sequencing libraries can be derived from a GEM well. For instance, multiome yields two libraries (ATAC and RNA) and CITE-seq+scVDJ yields 4 libraries (RNA, ADT, BCR, TCR). - LIBRARY_TYPE: the type of library for each library_id. Note that we used cell hashing () for a subset of the scRNA-seq libraries, and thus the library_type can be \\"not_hashed\\", \\"hashed_cdna\\" (RNA expression) or \\"hashed_hto\\" (the hashtag oligonucleotides). - LANE: to increase sequencing depth, each library was sequenced in more than one lane. Important: all lanes corresponding to the same sequencing library need to be inputed together to cellranger, because they come from the same set of cells. - READ: for scATAC-seq we have three reads (R1, R2 or R3), see cellranger-atac's documentation. While we find these names to be the most useful, they need to be changed to follow cellranger's conventions. We provide a code snippet in the README file of the GitHub repository associated with the tonsil atlas to convert between both formats (https://github.com/Single-Cell-Genomics-Group-CNAG-CRG/TonsilAtlas/). Besides the fastq files, cellranger (and other mappers) require additional files, which we also provide in this repository: - cell_hashing_metadata.csv: as mentioned above, we ran cell hashing (10.1186/s13059-018-1603-1) to detect doublets and reduce cost per cell. This file provides the sequence of the hashtag oligonucleotides in cellranger convention to allow demultiplexing. - cite_seq_feature_reference.csv: similar to the previous file, this one links each protein surface marker to the hashtag oligonucleotide that identified it in the CITE-seq experiment. - V10M16-059.gpr and V19S23-039.gpr: these correspond to the two slides of the two Visium experiments performed in the tonsil atlas. They are needed to run spaceranger. - [GEM_ID]_[SLIDE]_[CAPTURE_AREA].jpg: 8 images associated with the Visium experiments. Here, GEM_ID refers to each of the 4 capture areas in each slide. - [TECHNOLOGY]_sequencing_metadata.csv: the GEM-level metadata for each technology. It includes the relationship between subproject, gem_id, library_id, library_type and donor_id. These are the other repositories associated with the tonsil atlas: - Expression and accessibility matrices: https://zenodo.org/records/10373041 - Seurat objects: https://zenodo.org/records/8373756 - HCATonsilData package: https://bioconductor.org/packages/release/data/experiment/html/HCATonsilData.html - Azimuth: https://azimuth.hubmapconsortium.org/ - Github: https://github.com/Single-Cell-Genomics-Group-CNAG-CRG/TonsilAtlas
2024-05-02 | E-MTAB-13687 | biostudies-arrayexpress
Project description:Background: The first step of virtually all next generation sequencing analysis involves the splitting of the raw sequencing data into separate files using sample-specific barcodes, a process known as "demultiplexing". However, we found that existing software for this purpose was either too inflexible or too computationally intensive for fast, streamlined processing of raw, single end fastq files containing combinatorial barcodes. Results: Here, we introduce a fast and uniquely flexible demultiplexer, named Ultraplex, which splits a raw FASTQ file containing barcodes either at a single end or at both 5' and 3' ends of reads, trims the sequencing adaptors and low-quality bases, and moves unique molecular identifiers (UMIs) into the read header, allowing subsequent removal of PCR duplicates. Ultraplex is able to perform such single or combinatorial demultiplexing on both single- and paired-end sequencing data, and can process an entire Illumina HiSeq lane, consisting of nearly 500 million reads, in less than 20 minutes. Conclusions: Ultraplex greatly reduces computational burden and pipeline complexity for the demultiplexing of complex sequencing libraries, such as those produced by various CLIP and ribosome profiling protocols, and is also very user friendly, enabling streamlined, robust data processing. Ultraplex is available on PyPi and Conda and via Github.
| S-EPMC8287537 | biostudies-literature
Project description:UnlabelledMany Next Generation Sequencing analyses involve the basic manipulation of input sequence data before downstream processing (e.g. searching for specific sequences, format conversion or basic file statistics). The rapidly increasing data volumes involved in NGS make any dataset manipulation a time-consuming and error-prone process. I have developed fqtools; a fast and reliable FASTQ file manipulation suite that can process the full set of valid FASTQ files, including those with multi-line sequences, whilst identifying invalid files. Fqtools is faster than similar tools, and is designed for use in automatic processing pipelines.Availability and implementationfqtools is open source and is available at: https://github.com/alastair-droop/fqtoolsSupplementary informationSupplementary data are available at Bioinformatics online.Contacta.p.droop@leeds.ac.uk.
| S-EPMC4908325 | biostudies-literature