Browse
Submit Data
Databases
API
Help

Dataset Information

0 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

When the levee breaks: a practical guide to sketching algorithms for processing the flood of genomic data.

ABSTRACT: Considerable advances in genomics over the past decade have resulted in vast amounts of data being generated and deposited in global archives. The growth of these archives exceeds our ability to process their content, leading to significant analysis bottlenecks. Sketching algorithms produce small, approximate summaries of data and have shown great utility in tackling this flood of genomic data, while using minimal compute resources. This article reviews the current state of the field, focusing on how the algorithms work and how genomicists can utilize them effectively. References to interactive workbooks for explaining concepts and demonstrating workflows are included at https://github.com/will-rowe/genome-sketching .

SUBMITTER: Rowe WPM

PROVIDER: S-EPMC6744645 | biostudies-literature | 2019 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

When the levee breaks: a practical guide to sketching algorithms for processing the flood of genomic data.

Rowe Will P M WPM

Genome biology 20190913 1

Considerable advances in genomics over the past decade have resulted in vast amounts of data being generated and deposited in global archives. The growth of these archives exceeds our ability to process their content, leading to significant analysis bottlenecks. Sketching algorithms produce small, approximate summaries of data and have shown great utility in tackling this flood of genomic data, while using minimal compute resources. This article reviews the current state of the field, focusing o ...[more]

PMID: 31519212

Similar Datasets

Sketching algorithms for genomic data analysis and querying in a secure enclave.

Project description:Genome-wide association studies (GWAS), especially on rare diseases, may necessitate exchange of sensitive genomic data between multiple institutions. Since genomic data sharing is often infeasible due to privacy concerns, cryptographic methods, such as secure multiparty computation (SMC) protocols, have been developed with the aim of offering privacy-preserving collaborative GWAS. Unfortunately, the computational overhead of these methods remain prohibitive for human-genome-scale data. Here we introduce SkSES (https://github.com/ndokmai/sgx-genome-variants-search), a hardware-software hybrid approach for privacy-preserving collaborative GWAS, which improves the running time of the most advanced cryptographic protocols by two orders of magnitude. The SkSES approach is based on trusted execution environments (TEEs) offered by current-generation microprocessors-in particular, Intel's SGX. To overcome the severe memory limitation of the TEEs, SkSES employs novel 'sketching' algorithms that maintain essential statistical information on genomic variants in input VCF files. By additionally incorporating efficient data compression and population stratification reduction methods, SkSES identifies the top k genomic variants in a cohort quickly, accurately and in a privacy-preserving manner.

| S-EPMC7423249 | biostudies-literature

Statistical properties of sketching algorithms.

Project description:Sketching is a probabilistic data compression technique that has been largely developed by the computer science community. Numerical operations on big datasets can be intolerably slow; sketching algorithms address this issue by generating a smaller surrogate dataset. Typically, inference proceeds on the compressed dataset. Sketching algorithms generally use random projections to compress the original dataset, and this stochastic generation process makes them amenable to statistical analysis. We argue that the sketched data can be modelled as a random sample, thus placing this family of data compression methods firmly within an inferential framework. In particular, we focus on the Gaussian, Hadamard and Clarkson-Woodruff sketches and their use in single-pass sketching algorithms for linear regression with huge samples. We explore the statistical properties of sketched regression algorithms and derive new distributional results for a large class of sketching estimators. A key result is a conditional central limit theorem for data-oblivious sketches. An important finding is that the best choice of sketching algorithm in terms of mean squared error is related to the signal-to-noise ratio in the source dataset. Finally, we demonstrate the theory and the limits of its applicability on two datasets.

| S-EPMC7612324 | biostudies-literature

On randomized sketching algorithms and the Tracy-Widom law.

Project description:There is an increasing body of work exploring the integration of random projection into algorithms for numerical linear algebra. The primary motivation is to reduce the overall computational cost of processing large datasets. A suitably chosen random projection can be used to embed the original dataset in a lower-dimensional space such that key properties of the original dataset are retained. These algorithms are often referred to as sketching algorithms, as the projected dataset can be used as a compressed representation of the full dataset. We show that random matrix theory, in particular the Tracy-Widom law, is useful for describing the operating characteristics of sketching algorithms in the tall-data regime when the sample size n is much greater than the number of variables d. Asymptotic large sample results are of particular interest as this is the regime where sketching is most useful for data compression. In particular, we develop asymptotic approximations for the success rate in generating random subspace embeddings and the convergence probability of iterative sketching algorithms. We test a number of sketching algorithms on real large high-dimensional datasets and find that the asymptotic expressions give accurate predictions of the empirical performance.Supplementary informationThe online version contains supplementary material available at 10.1007/s11222-022-10148-5.

| S-EPMC9852177 | biostudies-literature

A Levee to the Flood: Pre-injury Neuroinflammation and Immune Stress Influence Traumatic Brain Injury Outcome.

Project description:Increasing evidence demonstrates that aging influences the brain's response to traumatic brain injury (TBI), setting the stage for neurodegenerative pathology like Alzheimer's disease (AD). This topic is often dominated by discussions of post-injury aging and inflammation, which can diminish the consideration of those same factors before TBI. In fact, pre-TBI aging and inflammation may be just as critical in mediating outcomes. For example, elderly individuals suffer from the highest rates of TBI of all severities. Additionally, pre-injury immune challenges or stressors may alter pathology and outcome independent of age. The inflammatory response to TBI is malleable and influenced by previous, coincident, and subsequent immune insults. Therefore, pre-existing conditions that elicit or include an inflammatory response could substantially influence the brain's ability to respond to traumatic injury and ultimately affect chronic outcome. The purpose of this review is to detail how age-related cellular and molecular changes, as well as genetic risk variants for AD affect the neuroinflammatory response to TBI. First, we will review the sources and pathology of neuroinflammation following TBI. Then, we will highlight the significance of age-related, endogenous sources of inflammation, including changes in cytokine expression, reactive oxygen species processing, and mitochondrial function. Heightened focus is placed on the mitochondria as an integral link between inflammation and various genetic risk factors for AD. Together, this review will compile current clinical and experimental research to highlight how pre-existing inflammatory changes associated with infection and stress, aging, and genetic risk factors can alter response to TBI.

| S-EPMC8790486 | biostudies-literature

Data structures and compression algorithms for genomic sequence data.

Project description:MotivationThe continuing exponential accumulation of full genome data, including full diploid human genomes, creates new challenges not only for understanding genomic structure, function and evolution, but also for the storage, navigation and privacy of genomic data. Here, we develop data structures and algorithms for the efficient storage of genomic and other sequence data that may also facilitate querying and protecting the data.ResultsThe general idea is to encode only the differences between a genome sequence and a reference sequence, using absolute or relative coordinates for the location of the differences. These locations and the corresponding differential variants can be encoded into binary strings using various entropy coding methods, from fixed codes such as Golomb and Elias codes, to variables codes, such as Huffman codes. We demonstrate the approach and various tradeoffs using highly variables human mitochondrial genome sequences as a testbed. With only a partial level of optimization, 3615 genome sequences occupying 56 MB in GenBank are compressed down to only 167 KB, achieving a 345-fold compression rate, using the revised Cambridge Reference Sequence as the reference sequence. Using the consensus sequence as the reference sequence, the data can be stored using only 133 KB, corresponding to a 433-fold level of compression, roughly a 23% improvement. Extensions to nuclear genomes and high-throughput sequencing data are discussed.AvailabilityData are publicly available from GenBank, the HapMap web site, and the MITOMAP database. Supplementary materials with additional results, statistics, and software implementations are available from http://mammag.web.uci.edu/bin/view/Mitowiki/ProjectDNACompression.

| S-EPMC2705231 | biostudies-literature

Damming the genomic data flood using a comprehensive analysis and storage data structure.

Project description:Data generation, driven by rapid advances in genomic technologies, is fast outpacing our analysis capabilities. Faced with this flood of data, more hardware and software resources are added to accommodate data sets whose structure has not specifically been designed for analysis. This leads to unnecessarily lengthy processing times and excessive data handling and storage costs. Current efforts to address this have centered on developing new indexing schemas and analysis algorithms, whereas the root of the problem lies in the format of the data itself. We have developed a new data structure for storing and analyzing genotype and phenotype data. By leveraging data normalization techniques, database management system capabilities and the use of a novel multi-table, multidimensional database structure we have eliminated the following: (i) unnecessarily large data set size due to high levels of redundancy, (ii) sequential access to these data sets and (iii) common bottlenecks in analysis times. The resulting novel data structure horizontally divides the data to circumvent traditional problems associated with the use of databases for very large genomic data sets. The resulting data set required 86% less disk space and performed analytical calculations 6248 times faster compared to a standard approach without any loss of information. Database URL: http://castor.pharmacogenomics.ca.

| S-EPMC3004464 | biostudies-literature

A practical guide to applying machine learning to infant EEG data.

Project description:Electroencephalography (EEG) has been widely adopted by the developmental cognitive neuroscience community, but the application of machine learning (ML) in this domain lags behind adult EEG studies. Applying ML to infant data is particularly challenging due to the low number of trials, low signal-to-noise ratio, high inter-subject variability, and high inter-trial variability. Here, we provide a step-by-step tutorial on how to apply ML to classify cognitive states in infants. We describe the type of brain attributes that are widely used for EEG classification and also introduce a Riemannian geometry based approach for deriving connectivity estimates that account for inter-trial and inter-subject variability. We present pipelines for learning classifiers using trials from a single infant and from multiple infants, and demonstrate the application of these pipelines on a standard infant EEG dataset of forty 12-month-old infants collected under an auditory oddball paradigm. While we classify perceptual states induced by frequent versus rare stimuli, the presented pipelines can be easily adapted for other experimental designs and stimuli using the associated code that we have made publicly available.

| S-EPMC8943418 | biostudies-literature

Trust, but Verify II: A Practical Guide to Chemogenomics Data Curation.

Project description:There is a growing public concern about the lack of reproducibility of experimental data published in peer-reviewed scientific literature. Herein, we review the most recent alerts regarding experimental data quality and discuss initiatives taken thus far to address this problem, especially in the area of chemical genomics. Going beyond just acknowledging the issue, we propose a chemical and biological data curation workflow that relies on existing cheminformatics approaches to flag, and when appropriate, correct possibly erroneous entries in large chemogenomics data sets. We posit that the adherence to the best practices for data curation is important for both experimental scientists who generate primary data and deposit them in chemical genomics databases and computational researchers who rely on these data for model development.

| S-EPMC5657146 | biostudies-literature

A practical evaluation of alignment algorithms for RNA variant calling analysis

Project description:We performed RNA-seq with ten pieces of breast cancer (invasive ductal carcinoma; luminal B type) tissue and three pieces of adjacent normal tissue from a single patient. These RNA-seq data were used to evaluate the performance of splice-aware aligners.

2018-02-06 | GSE110114 | GEO

Decision Algorithms for Direct Oral Anticoagulant Use in Patients With Nonvalvular Atrial Fibrillation: A Practical Guide for Neurologists.

Project description:Direct oral anticoagulants (DOACs) are valid alternative options to vitamin K antagonists due to their limited interactions with drugs or food and the fact that they do not require regular coagulation monitoring. To this regard, recent practice guidelines recommend that DOACs should be considered as first-line anticoagulant therapy for stroke prevention in patients with nonvalvular atrial fibrillation (NVAF). This review (1) outlines current international guidelines for the management of DOACs to prevent stroke in patients with NVAF, (2) outlines indications for elderly patients as well as specific settings including acute coronary syndromes and intracranial hemorrhage, and (3) offers a practical guide for the use of DOACs in neurological settings.

| S-EPMC6714651 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data