Browse
Submit Data
Databases
API
Help

Dataset Information

0 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Scalable Workflow-Driven Hydrologic Analysis in HydroFrame

ABSTRACT: The HydroFrame project is a community platform designed to facilitate integrated hydrologic modeling across the US. As a part of HydroFrame, we seek to design innovative workflow solutions that create pathways to enable hydrologic analysis for three target user groups: the modeler, the analyzer, and the domain science educator. We present the initial progress on the HydroFrame community platform using an automated Kepler workflow. This workflow performs end-to-end hydrology simulations involving data ingestion, preprocessing, analysis, modeling, and visualization. We demonstrate how different modules of the workflow can be reused and repurposed for the three target user groups. The Kepler workflow ensures complete reproducibility through a built-in provenance framework that collects workflow specific parameters, software versions, and hardware system configuration. In addition, we aim to optimize the utilization of large-scale computational resources to adjust to the needs of all three user groups. Towards this goal, we present a design that leverages provenance data and machine learning techniques to predict performance and forecast failures using an automatic performance collection component of the pipeline.

SUBMITTER: Krzhizhanovskaya V

PROVIDER: S-EPMC7302287 | biostudies-literature | 2020 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Similar Datasets

BISR-RNAseq: an efficient and scalable RNAseq analysis workflow with interactive report generation.

Project description:BackgroundRNA sequencing has become an increasingly affordable way to profile gene expression patterns. Here we introduce a workflow implementing several open-source softwares that can be run on a high performance computing environment.ResultsDeveloped as a tool by the Bioinformatics Shared Resource Group (BISR) at the Ohio State University, we have applied the pipeline to a few publicly available RNAseq datasets downloaded from GEO in order to demonstrate the feasibility of this workflow. Source code is available here: workflow: https://code.bmi.osumc.edu/gadepalli.3/BISR-RNAseq-ICIBM2019 and shiny: https://code.bmi.osumc.edu/gadepalli.3/BISR_RNASeq_ICIBM19. Example dataset is demonstrated here: https://dataportal.bmi.osumc.edu/RNA_Seq/.ConclusionThe workflow allows for the analysis (alignment, QC, gene-wise counts generation) of raw RNAseq data and seamless integration of quality analysis and differential expression results into a configurable R shiny web application.

| S-EPMC6923898 | biostudies-literature

A scalable workflow to characterize the human exposome.

Project description:Complementing the genome with an understanding of the human exposome is an important challenge for contemporary science and technology. Tens of thousands of chemicals are used in commerce, yet cost for targeted environmental chemical analysis limits surveillance to a few hundred known hazards. To overcome limitations which prevent scaling to thousands of chemicals, we develop a single-step express liquid extraction and gas chromatography high-resolution mass spectrometry analysis to operationalize the human exposome. We show that the workflow supports quantification of environmental chemicals in human plasma (200 µL) and tissue (≤100 mg) samples. The method also provides high resolution, sensitivity and selectivity for exposome epidemiology of mass spectral features without a priori knowledge of chemical identity. The simplicity of the method can facilitate harmonization of environmental biomonitoring between laboratories and enable population level human exposome research with limited sample volume.

| S-EPMC8458492 | biostudies-literature

What the Phage: a scalable workflow for the identification and analysis of phage sequences.

Project description:Phages are among the most abundant and diverse biological entities on earth. Phage prediction from sequence data is a crucial first step to understanding their impact on the environment. A variety of bacteriophage prediction tools have been developed over the years. They differ in algorithmic approach, results, and ease of use. We, therefore, developed "What the Phage" (WtP), an easy-to-use and parallel multitool approach for phage prediction combined with an annotation and classification downstream strategy, thus supporting the user's decision-making process by summarizing the results of the different prediction tools in charts and tables. WtP is reproducible and scales to thousands of datasets through a workflow manager (Nextflow). WtP is freely available under a GPL-3.0 license (https://github.com/replikation/What_the_Phage).

| S-EPMC9673492 | biostudies-literature

KNIME-CDK: Workflow-driven cheminformatics.

Project description:BACKGROUND: Cheminformaticians have to routinely process and analyse libraries of small molecules. Among other things, that includes the standardization of molecules, calculation of various descriptors, visualisation of molecular structures, and downstream analysis. For this purpose, scientific workflow platforms such as the Konstanz Information Miner can be used if provided with the right plug-in. A workflow-based cheminformatics tool provides the advantage of ease-of-use and interoperability between complementary cheminformatics packages within the same framework, hence facilitating the analysis process. RESULTS: KNIME-CDK comprises functions for molecule conversion to/from common formats, generation of signatures, fingerprints, and molecular properties. It is based on the Chemistry Development Toolkit and uses the Chemical Markup Language for persistence. A comparison with the cheminformatics plug-in RDKit shows that KNIME-CDK supports a similar range of chemical classes and adds new functionality to the framework. We describe the design and integration of the plug-in, and demonstrate the usage of the nodes on ChEBI, a library of small molecules of biological interest. CONCLUSIONS: KNIME-CDK is an open-source plug-in for the Konstanz Information Miner, a free workflow platform. KNIME-CDK is build on top of the open-source Chemistry Development Toolkit and allows for efficient cross-vendor structural cheminformatics. Its ease-of-use and modularity enables researchers to automate routine tasks and data analysis, bringing complimentary cheminformatics functionality to the workflow environment.

| S-EPMC3765822 | biostudies-literature

A Multivariate Analysis-Driven Workflow to Tackle Uncertainties in Miniaturized NIR Data.

Project description:This study focuses on exploring and understanding measurement errors in analytical procedures involving miniaturized near-infrared instruments. Despite recent spreading in different application fields, there remains a lack of emphasis on the accuracy and reliability of these devices, which is a critical concern for accurate scientific outcomes. The study investigates multivariate measurement errors, revealing their complex nature and the influence that preprocessing techniques can have. The research introduces a possible workflow for practical error analysis in experiments involving diverse samples and instruments. Notably, it investigates how sample characteristics impact errors in the case of solid pills and tablets, typical pharmaceutical samples. ASCA was used for understanding critical instrumental factors and the potential and limitations of the method in the current application were discussed. The joint interpretation of multivariate error matrices and their resume through image histograms and K index are discussed in order to evaluate the impact of common preprocessing methods and to assess their influence on signals.

| S-EPMC10745448 | biostudies-literature

iLAP: a workflow-driven software for experimental protocol development, data acquisition and analysis.

Project description:BackgroundIn recent years, the genome biology community has expended considerable effort to confront the challenges of managing heterogeneous data in a structured and organized way and developed laboratory information management systems (LIMS) for both raw and processed data. On the other hand, electronic notebooks were developed to record and manage scientific data, and facilitate data-sharing. Software which enables both, management of large datasets and digital recording of laboratory procedures would serve a real need in laboratories using medium and high-throughput techniques.ResultsWe have developed iLAP (Laboratory data management, Analysis, and Protocol development), a workflow-driven information management system specifically designed to create and manage experimental protocols, and to analyze and share laboratory data. The system combines experimental protocol development, wizard-based data acquisition, and high-throughput data analysis into a single, integrated system. We demonstrate the power and the flexibility of the platform using a microscopy case study based on a combinatorial multiple fluorescence in situ hybridization (m-FISH) protocol and 3D-image reconstruction. iLAP is freely available under the open source license AGPL from http://genome.tugraz.at/iLAP/.ConclusioniLAP is a flexible and versatile information management system, which has the potential to close the gap between electronic notebooks and LIMS and can therefore be of great value for a broad scientific community.

| S-EPMC2789074 | biostudies-literature

Scalable workflow for characterization of cell-cell communication in COVID-19 patients.

Project description:COVID-19 patients display a wide range of disease severity, ranging from asymptomatic to critical symptoms with high mortality risk. Our ability to understand the interaction of SARS-CoV-2 infected cells within the lung, and of protective or dysfunctional immune responses to the virus, is critical to effectively treat these patients. Currently, our understanding of cell-cell interactions across different disease states, and how such interactions may drive pathogenic outcomes, is incomplete. Here, we developed a generalizable and scalable workflow for identifying cells that are differentially interacting across COVID-19 patients with distinct disease outcomes and use this to examine eight public single-cell RNA-seq datasets (six from peripheral blood mononuclear cells, one from bronchoalveolar lavage and one from nasopharyngeal), with a total of 211 individual samples. By characterizing the cell-cell interaction patterns across epithelial and immune cells in lung tissues for patients with varying disease severity, we illustrate diverse communication patterns across individuals, and discover heterogeneous communication patterns among moderate and severe patients. We further illustrate patterns derived from cell-cell interactions are potential signatures for discriminating between moderate and severe patients. Overall, this workflow can be generalized and scaled to combine multiple scRNA-seq datasets to uncover cell-cell interactions.

| S-EPMC9534414 | biostudies-literature

nf-LO: A Scalable, Containerized Workflow for Genome-to-Genome Lift Over.

Project description:The increasing availability of new genome assemblies often comes with a paucity of associated genomic annotations, limiting the range of studies that can be performed. A common workaround is to lift over annotations from better annotated genomes. However, generating the files required to perform a lift over is computationally and labor intensive and only a limited number are currently publicly available. Here we present nf-LO (nextflow-LiftOver), a containerized and scalable Nextflow pipeline that enables lift overs within and between any species for which assemblies are available. nf-LO will consequently facilitate data interpretation across a broad range of genomic studies.

| S-EPMC8412297 | biostudies-literature

Evaluation of serverless computing for scalable execution of a joint variant calling workflow.

Project description:Advances in whole-genome sequencing have greatly reduced the cost and time of obtaining raw genetic information, but the computational requirements of analysis remain a challenge. Serverless computing has emerged as an alternative to using dedicated compute resources, but its utility has not been widely evaluated for standardized genomic workflows. In this study, we define and execute a best-practice joint variant calling workflow using the SWEEP workflow management system. We present an analysis of performance and scalability, and discuss the utility of the serverless paradigm for executing workflows in the field of genomics research. The GATK best-practice short germline joint variant calling pipeline was implemented as a SWEEP workflow comprising 18 tasks. The workflow was executed on Illumina paired-end read samples from the European and African super populations of the 1000 Genomes project phase III. Cost and runtime increased linearly with increasing sample size, although runtime was driven primarily by a single task for larger problem sizes. Execution took a minimum of around 3 hours for 2 samples, up to nearly 13 hours for 62 samples, with costs ranging from $2 to $70.

| S-EPMC8270184 | biostudies-literature

An approach to quantum-computational hydrologic inverse analysis.

Project description:Making predictions about flow and transport in an aquifer requires knowledge of the heterogeneous properties of the aquifer such as permeability. Computational methods for inverse analysis are commonly used to infer these properties from quantities that are more readily observable such as hydraulic head. We present a method for computational inverse analysis that utilizes a type of quantum computer called a quantum annealer. While quantum computing is in an early stage compared to classical computing, we demonstrate that it is sufficiently developed that it can be used to solve certain subsurface flow problems. We utilize a D-Wave 2X quantum annealer to solve 1D and 2D hydrologic inverse problems that, while small by modern standards, are similar in size and sometimes larger than hydrologic inverse problems that were solved with early classical computers. Our results and the rapid progress being made with quantum computing hardware indicate that the era of quantum-computational hydrology may not be too far in the future.

| S-EPMC5931973 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data