SPIM workflow manager for HPC.
Ontology highlight
ABSTRACT: Summary
Here we introduce a Fiji plugin utilizing the HPC-as-a-Service concept, significantly mitigating the challenges life scientists face when delegating complex data-intensive processing workflows to HPC clusters. We demonstrate on a common Selective Plane Illumination Microscopy image processing task that execution of a Fiji workflow on a remote supercomputer leads to improved turnaround time despite the data transfer overhead. The plugin allows the end users to conveniently transfer image data to remote HPC resources, manage pipeline jobs and visualize processed results directly from the Fiji graphical user interface.Availability and implementation
The code is distributed free and open source under the MIT license. Source code: https://github.com/fiji-hpc/hpc-workflow-manager/, documentation: https://imagej.net/SPIM_Workflow_Manager_For_HPC.Supplementary information
Supplementary data are available at Bioinformatics online.
SUBMITTER: Kozusznik J
PROVIDER: S-EPMC6761944 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:UnlabelledSelective Plane Illumination Microscopy (SPIM) allows to image developing organisms in 3D at unprecedented temporal resolution over long periods of time. The resulting massive amounts of raw image data requires extensive processing interactively via dedicated graphical user interface (GUI) applications. The consecutive processing steps can be easily automated and the individual time points can be processed independently, which lends itself to trivial parallelization on a high performance computing (HPC) cluster. Here, we introduce an automated workflow for processing large multiview, multichannel, multiillumination time-lapse SPIM data on a single workstation or in parallel on a HPC cluster. The pipeline relies on snakemake to resolve dependencies among consecutive processing steps and can be easily adapted to any cluster environment for processing SPIM data in a fraction of the time required to collect it.Availability and implementationThe code is distributed free and open source under the MIT license http://opensource.org/licenses/MIT The source code can be downloaded from github: https://github.com/mpicbg-scicomp/snakemake-workflows Documentation can be found here: http://fiji.sc/Automated_workflow_for_parallel_Multiview_ReconstructionContact: schmied@mpi-cbg.deSupplementary informationSupplementary data are available at Bioinformatics online.
| S-EPMC4896369 | biostudies-literature
Project description:BACKGROUND:Massive growth in the amount of research data and computational analysis has led to increased use of pipeline managers in biomedical computational research. However, each of the >100 such managers uses its own way to describe pipelines, leading to difficulty porting workflows to different environments and therefore poor reproducibility of computational studies. For this reason, the Common Workflow Language (CWL) was recently introduced as a specification for platform-independent workflow description, and work began to transition existing pipelines and workflow managers to CWL. FINDINGS:Herein, we present CWL-Airflow, a package that adds support for CWL to the Apache Airflow pipeline manager. CWL-Airflow uses CWL version 1.0 specification and can run workflows on stand-alone MacOS/Linux servers, on clusters, or on a variety of cloud platforms. A sample CWL pipeline for processing of chromatin immunoprecipitation sequencing data is provided. CONCLUSIONS:CWL-Airflow will provide users with the features of a fully fledged pipeline manager and the ability to execute CWL workflows anywhere Airflow can run-from a laptop to a cluster or cloud environment. CWL-Airflow is available under Apache License, version 2.0 (Apache-2.0), and can be downloaded from https://barski-lab.github.io/cwl-airflow, https://scicrunch.org/resolver/RRID:SCR_017196.
| S-EPMC6639121 | biostudies-literature
Project description:Rapid improvements in DNA sequencing technology have resulted in long genome sequences for a large number of similar isolates with a wide range of single nucleotide polymorphism (SNP) rates, where some isolates can have thousands of times lower SNP rates than others. Genome sequences of this kind are a challenge to existing methods for construction of phylogenetic trees. We address the issues by developing a hierarchical approach to phylogeny construction. In this method, the construction is performed at multiple levels, where at each level, groups of isolates with similar levels of similarity are identified and their phylogenetic trees are constructed. Time savings are achieved by using a sufficiently large number of columns from the input alignment, instead of all its columns. Our results show that the new approach is 20-60 times more efficient than existing programs and more accurate in situations where highly similar isolates have a wide range of SNP rates.
| S-EPMC6705828 | biostudies-literature
Project description:BackgroundRNA editing is a widespread co-/post-transcriptional mechanism that alters primary RNA sequences through the modification of specific nucleotides and it can increase both the transcriptome and proteome diversity. The automatic detection of RNA-editing from RNA-seq data is computational intensive and limited to small data sets, thus preventing a reliable genome-wide characterisation of such process.ResultsIn this work we introduce HPC-REDItools, an upgraded tool for accurate RNA-editing events discovery from large dataset repositories.Availabilityhttps://github.com/BioinfoUNIBA/REDItools2 .ConclusionsHPC-REDItools is dramatically faster than the previous version, REDItools, enabling big-data analysis by means of a MPI-based implementation and scaling almost linearly with the number of available cores.
| S-EPMC7446188 | biostudies-literature
Project description:BackgroundPhylogenetic analysis of large, multiple-gene datasets, assembled from public sequence databases, is rapidly becoming a popular way to approach difficult phylogenetic problems. Supermatrices (concatenated multiple sequence alignments of multiple genes) can yield more phylogenetic signal than individual genes. However, manually assembling such datasets for a large taxonomic group is time-consuming and error-prone. Additionally, sequence curation, alignment and assessment of the results of phylogenetic analysis are made particularly difficult by the potential for a given gene in a given species to be unrepresented, or to be represented by multiple or partial sequences. We have developed a software package, TaxMan, that largely automates the processes of sequence acquisition, consensus building, alignment and taxon selection to facilitate this type of phylogenetic study.ResultsTaxMan uses freely available tools to allow rapid assembly, storage and analysis of large, aligned DNA and protein sequence datasets for user-defined sets of species and genes. The user provides GenBank format files and a list of gene names and synonyms for the loci to analyse. Sequences are extracted from the GenBank files on the basis of annotation and sequence similarity. Consensus sequences are built automatically. Alignment is carried out (where possible, at the protein level) and aligned sequences are stored in a database. TaxMan can automatically determine the best subset of taxa to examine phylogeny at a given taxonomic level. By using the stored aligned sequences, large concatenated multiple sequence alignments can be generated rapidly for a subset and output in analysis-ready file formats. Trees resulting from phylogenetic analysis can be stored and compared with a reference taxonomy.ConclusionTaxMan allows rapid automated assembly of a multigene datasets of aligned sequences for large taxonomic groups. By extracting sequences on the basis of both annotation and BLAST similarity, it ensures that all available sequence data can be brought to bear on a phylogenetic problem, but remains fast enough to cope with many thousands of records. By automatically assisting in the selection of the best subset of taxa to address a particular phylogenetic problem, TaxMan greatly speeds up the process of generating multiple sequence alignments for phylogenetic analysis. Our results indicate that an automated phylogenetic workbench can be a useful tool when correctly guided by user knowledge.
| S-EPMC1766369 | biostudies-literature
Project description:Intermediate lung nodules are frequently discovered in CT imaging as either incidental or part of cancer screening. The Lung Nodule Manager allows for a quick retrieval of guidelines by physicians and health care professionals to determine the proper management and follow-up for patients.
| S-EPMC6456644 | biostudies-literature
Project description:Reference genome assemblies are essential for high-throughput sequencing analysis projects. Typically, genome assemblies are stored on disk alongside related resources; e.g., many sequence aligners require the assembly to be indexed. The resulting indexes are broadly applicable for downstream analysis, so it makes sense to share them. However, there is no simple tool to do this. Here, we introduce refgenie, a reference genome assembly asset manager. Refgenie makes it easier to organize, retrieve, and share genome analysis resources. In addition to genome indexes, refgenie can manage any files related to reference genomes, including sequences and annotation files. Refgenie includes a command line interface and a server application that provides a RESTful API, so it is useful for both tool development and analysis. Refgenie streamlines sharing genome analysis resources among groups and across computing environments. Refgenie is available at https://refgenie.databio.org.
| S-EPMC6988606 | biostudies-literature
Project description:ObjectiveFaster engraftment of G-CSF-mobilized peripheral blood (MPB) transplants compared to steady-state bone marrow (ssBM) is well documented and clinically relevant. A number of different factors likely contribute to this outcome. In the present study we explored whether independent of cell number there are intrinsic differences in the efficiency of progenitor cell homing to marrow between MPB and ssBM.MethodsMobilization was achieved by continuous infusion of G-CSF alone or in combination with other mobilizing agents. In vivo homing assays, in vitro migration assays, gene expression analysis, and flow cytometry were utilized to compare homing-related in vivo and in vitro properties of MPB and ssBM HPC.ResultsMarrow homing of murine MPB HPC, generated by different mobilizing schemes, was reproducibly significantly superior to that of ssBM, in lethally irradiated as well as in nonirradiated hosts. This phenotype was independent of MMP9, selectins, and beta2- and alpha4-integrins. Superior homing was also observed for human MPB HPC transplanted into NOD/SCIDbeta2microglobulin(-/-) recipients. Inhibition of HPC migration abrogated the homing advantage of MPB but did not affect homing of ssBM HPC, whereas enhancement of motility by CD26 inhibition improved marrow homing only of ssBM HPC. Enhanced SDF-1-dependent chemotaxis and low CD26 expression on MPB HPC were identified as potential contributing factors. Significant contributions of the putative alternative SDF-1 receptor, RDC1, were unlikely based on gene expression data.ConclusionThe data suggest increased motility as a converging endpoint of complex changes seen in MPB HPC which is likely responsible for their favorable homing.
| S-EPMC1847625 | biostudies-literature
Project description:We designed an epi-illumination SPIM system that uses a single objective and has a sample interface identical to that of an inverted fluorescence microscope with no additional reflection elements. It achieves subcellular resolution and single-molecule sensitivity, and is compatible with common biological sample holders, including multi-well plates. We demonstrated multicolor fast volumetric imaging, single-molecule localization microscopy, parallel imaging of 16 cell lines and parallel recording of cellular responses to perturbations.
| S-EPMC6557432 | biostudies-literature
Project description:We provide evidence that a personality trait, aggression, has a first-order effect on group financial decision making. In a laboratory experiment on group portfolio choice, highly aggressive subjects (measured by a standard psychology test) were much more likely to recommend risky investment strategies consistent with their own personal information, regardless of the information received by other group members. Outside of this group context, aggression had no effect on subject behavior. Thus, our aggression measure appears to capture an aggressive disposition, which seeks to dominate group decisions, rather than simply reflect risk attitudes or cognitive biases.
| S-EPMC5809062 | biostudies-literature