SPIM workflow manager for HPC.
Ontology highlight
ABSTRACT: Summary
Here we introduce a Fiji plugin utilizing the HPC-as-a-Service concept, significantly mitigating the challenges life scientists face when delegating complex data-intensive processing workflows to HPC clusters. We demonstrate on a common Selective Plane Illumination Microscopy image processing task that execution of a Fiji workflow on a remote supercomputer leads to improved turnaround time despite the data transfer overhead. The plugin allows the end users to conveniently transfer image data to remote HPC resources, manage pipeline jobs and visualize processed results directly from the Fiji graphical user interface.Availability and implementation
The code is distributed free and open source under the MIT license. Source code: https://github.com/fiji-hpc/hpc-workflow-manager/, documentation: https://imagej.net/SPIM_Workflow_Manager_For_HPC.Supplementary information
Supplementary data are available at Bioinformatics online.
SUBMITTER: Kozusznik J
PROVIDER: S-EPMC6761944 | biostudies-literature | 2019 Oct
REPOSITORIES: biostudies-literature
Publications
SPIM workflow manager for HPC.
Kožusznik Jan J Bainar Petr P Klímová Jana J Krumnikl Michal M Moravec Pavel P Svatoň Václav V Tomančák Pavel P
Bioinformatics (Oxford, England) 20191001 19
<h4>Summary</h4>Here we introduce a Fiji plugin utilizing the HPC-as-a-Service concept, significantly mitigating the challenges life scientists face when delegating complex data-intensive processing workflows to HPC clusters. We demonstrate on a common Selective Plane Illumination Microscopy image processing task that execution of a Fiji workflow on a remote supercomputer leads to improved turnaround time despite the data transfer overhead. The plugin allows the end users to conveniently transfe ...[more]
Similar Datasets
Project description:UnlabelledSelective Plane Illumination Microscopy (SPIM) allows to image developing organisms in 3D at unprecedented temporal resolution over long periods of time. The resulting massive amounts of raw image data requires extensive processing interactively via dedicated graphical user interface (GUI) applications. The consecutive processing steps can be easily automated and the individual time points can be processed independently, which lends itself to trivial parallelization on a high performance computing (HPC) cluster. Here, we introduce an automated workflow for processing large multiview, multichannel, multiillumination time-lapse SPIM data on a single workstation or in parallel on a HPC cluster. The pipeline relies on snakemake to resolve dependencies among consecutive processing steps and can be easily adapted to any cluster environment for processing SPIM data in a fraction of the time required to collect it.Availability and implementationThe code is distributed free and open source under the MIT license http://opensource.org/licenses/MIT The source code can be downloaded from github: https://github.com/mpicbg-scicomp/snakemake-workflows Documentation can be found here: http://fiji.sc/Automated_workflow_for_parallel_Multiview_ReconstructionContact: schmied@mpi-cbg.deSupplementary informationSupplementary data are available at Bioinformatics online.
| S-EPMC4896369 | biostudies-literature
Project description:Massive growth in the amount of research data and computational analysis has led to increased use of pipeline managers in biomedical computational research. However, each of the >100 such managers uses its own way to describe pipelines, leading to difficulty porting workflows to different environments and therefore poor reproducibility of computational studies. For this reason, the Common Workflow Language (CWL) was recently introduced as a specification for platform-independent workflow description, and work began to transition existing pipelines and workflow managers to CWL. Herein, we present CWL-Airflow, a package that adds support for CWL to the Apache Airflow pipeline manager. CWL-Airflow uses CWL version 1.0 specification and can run workflows on stand-alone MacOS/Linux servers, on clusters, or on a variety of cloud platforms. A sample CWL pipeline for processing of chromatin immunoprecipitation sequencing data is provided. CWL-Airflow will provide users with the features of a fully fledged pipeline manager and the ability to execute CWL workflows anywhere Airflow can run-from a laptop to a cluster or cloud environment. CWL-Airflow is available under Apache License, version 2.0 (Apache-2.0), and can be downloaded from https://barski-lab.github.io/cwl-airflow, https://scicrunch.org/resolver/RRID:SCR_017196.
| S-EPMC6639121 | biostudies-literature
Project description:BackgroundThe availability of transcriptomic data for species without a reference genome enables the construction of de novo transcriptome assemblies as alternative reference resources from RNA-Seq data. A transcriptome provides direct information about a species' protein-coding genes under specific experimental conditions. The de novo assembly process produces a unigenes file in FASTA format, subsequently targeted for the annotation. Homology-based annotation, a method to infer the function of sequences by estimating similarity with other sequences in a reference database, is a computationally demanding procedure.ResultsTo mitigate the computational burden, we introduce HPC-T-Annotator, a tool for de novo transcriptome homology annotation on high performance computing (HPC) infrastructures, designed for straightforward configuration via a Web interface. Once the configuration data are given, the entire parallel computing software for annotation is automatically generated and can be launched on a supercomputer using a simple command line. The output data can then be easily viewed using post-processing utilities in the form of Python notebooks integrated in the proposed software.ConclusionsHPC-T-Annotator expedites homology-based annotation in de novo transcriptome assemblies. Its efficient parallelization strategy on HPC infrastructures significantly reduces computational load and execution times, enabling large-scale transcriptome analysis and comparison projects, while its intuitive graphical interface extends accessibility to users without IT skills.
| S-EPMC11340092 | biostudies-literature
Project description:Rapid improvements in DNA sequencing technology have resulted in long genome sequences for a large number of similar isolates with a wide range of single nucleotide polymorphism (SNP) rates, where some isolates can have thousands of times lower SNP rates than others. Genome sequences of this kind are a challenge to existing methods for construction of phylogenetic trees. We address the issues by developing a hierarchical approach to phylogeny construction. In this method, the construction is performed at multiple levels, where at each level, groups of isolates with similar levels of similarity are identified and their phylogenetic trees are constructed. Time savings are achieved by using a sufficiently large number of columns from the input alignment, instead of all its columns. Our results show that the new approach is 20-60 times more efficient than existing programs and more accurate in situations where highly similar isolates have a wide range of SNP rates.
| S-EPMC6705828 | biostudies-literature
Project description:BackgroundRNA editing is a widespread co-/post-transcriptional mechanism that alters primary RNA sequences through the modification of specific nucleotides and it can increase both the transcriptome and proteome diversity. The automatic detection of RNA-editing from RNA-seq data is computational intensive and limited to small data sets, thus preventing a reliable genome-wide characterisation of such process.ResultsIn this work we introduce HPC-REDItools, an upgraded tool for accurate RNA-editing events discovery from large dataset repositories.Availabilityhttps://github.com/BioinfoUNIBA/REDItools2 .ConclusionsHPC-REDItools is dramatically faster than the previous version, REDItools, enabling big-data analysis by means of a MPI-based implementation and scaling almost linearly with the number of available cores.
| S-EPMC7446188 | biostudies-literature
Project description:Intermediate lung nodules are frequently discovered in CT imaging as either incidental or part of cancer screening. The Lung Nodule Manager allows for a quick retrieval of guidelines by physicians and health care professionals to determine the proper management and follow-up for patients.
| S-EPMC6456644 | biostudies-literature
Project description:BackgroundPhylogenetic analysis of large, multiple-gene datasets, assembled from public sequence databases, is rapidly becoming a popular way to approach difficult phylogenetic problems. Supermatrices (concatenated multiple sequence alignments of multiple genes) can yield more phylogenetic signal than individual genes. However, manually assembling such datasets for a large taxonomic group is time-consuming and error-prone. Additionally, sequence curation, alignment and assessment of the results of phylogenetic analysis are made particularly difficult by the potential for a given gene in a given species to be unrepresented, or to be represented by multiple or partial sequences. We have developed a software package, TaxMan, that largely automates the processes of sequence acquisition, consensus building, alignment and taxon selection to facilitate this type of phylogenetic study.ResultsTaxMan uses freely available tools to allow rapid assembly, storage and analysis of large, aligned DNA and protein sequence datasets for user-defined sets of species and genes. The user provides GenBank format files and a list of gene names and synonyms for the loci to analyse. Sequences are extracted from the GenBank files on the basis of annotation and sequence similarity. Consensus sequences are built automatically. Alignment is carried out (where possible, at the protein level) and aligned sequences are stored in a database. TaxMan can automatically determine the best subset of taxa to examine phylogeny at a given taxonomic level. By using the stored aligned sequences, large concatenated multiple sequence alignments can be generated rapidly for a subset and output in analysis-ready file formats. Trees resulting from phylogenetic analysis can be stored and compared with a reference taxonomy.ConclusionTaxMan allows rapid automated assembly of a multigene datasets of aligned sequences for large taxonomic groups. By extracting sequences on the basis of both annotation and BLAST similarity, it ensures that all available sequence data can be brought to bear on a phylogenetic problem, but remains fast enough to cope with many thousands of records. By automatically assisting in the selection of the best subset of taxa to address a particular phylogenetic problem, TaxMan greatly speeds up the process of generating multiple sequence alignments for phylogenetic analysis. Our results indicate that an automated phylogenetic workbench can be a useful tool when correctly guided by user knowledge.
| S-EPMC1766369 | biostudies-literature
Project description:We designed an epi-illumination SPIM system that uses a single objective and has a sample interface identical to that of an inverted fluorescence microscope with no additional reflection elements. It achieves subcellular resolution and single-molecule sensitivity, and is compatible with common biological sample holders, including multi-well plates. We demonstrated multicolor fast volumetric imaging, single-molecule localization microscopy, parallel imaging of 16 cell lines and parallel recording of cellular responses to perturbations.
| S-EPMC6557432 | biostudies-literature
Project description:During their formal studies medical students acquire extensive medical expertise. However, the medical profession demands additional competencies, such as those involved in efficient resource allocation, business administration, development, organization, and process management in the healthcare system. At present students are not sufficiently prepared for the physician's role as manager. In response, we designed the seminar course, MeCuM-SiGma, to impart basic knowledge about healthcare policy and management to students of medicine. This project report describes our teaching strategies and the initial evaluation of this educational project.In this semester-long, seminar course introduced in 2010, medical students gather experience with the competencies mentioned above as well as learn basic management skills. The course is offered each winter semester, and students sign up to attend voluntarily; course coordination and organization is done on a voluntary basis by physicians and employees of the Mentoring Office (MeCuM-Mentor) at the Medical School of the Ludwig Maximilian University (LMU) in Munich, Germany. The course is open to all students enrolled at the two medical schools in Munich. During the first part of this elective, students learn about the basic principles of the German political and healthcare systems in case-based, problem-based tutorials led by trained tutors and in lectures held by experts. In the second part of the course students take on the roles of the University Hospital's executive board of directors and supervisory board to work on an existing hospital project as a group within the scope of a simulation. This phase of the course is accompanied by workshops conducted in cooperation with university-based and off-campus partners that address the procedural learning objectives (teamwork, project management, negotiation strategies, etc.). A suitable, authentic issue currently facing the hospital is selected in advance by the course organizers in coordination with the hospital's executive board. Students then work on this issue in the third and final phase of the course under the supervision of tutors and with assistance from hospital employees. At the end of the course the students formally present the results of their work to the hospital's executive and supervisory boards.The course undergoes written student evaluation, a round of oral feedback, evaluation of the final projects, and feedback from the hospital's executive and supervisory boards. All attendees to date have reported a substantial gain in general knowledge and increased knowledge about the healthcare system, and rate the relevance of the course as being high. The majority felt the content was important for their future practice of medicine. Overall, students evaluated the course very positively [overall rating on a six-point grading scale (1=excellent; 6=unsatisfactory): 1.28 (mean)±0.45 (standard deviation)].The importance of the physician's role as manager in medical organizations and as a guiding force in the healthcare system is neglected in medical degree programs. Our seminar course attempts to address this shortcoming, is the object of great interest and receives positive evaluations from seminar participants, our cooperative partners and the executive and supervisory boards of the University Hospital in Munich.
| S-EPMC4894355 | biostudies-other
Project description:ObjectiveFaster engraftment of G-CSF-mobilized peripheral blood (MPB) transplants compared to steady-state bone marrow (ssBM) is well documented and clinically relevant. A number of different factors likely contribute to this outcome. In the present study we explored whether independent of cell number there are intrinsic differences in the efficiency of progenitor cell homing to marrow between MPB and ssBM.MethodsMobilization was achieved by continuous infusion of G-CSF alone or in combination with other mobilizing agents. In vivo homing assays, in vitro migration assays, gene expression analysis, and flow cytometry were utilized to compare homing-related in vivo and in vitro properties of MPB and ssBM HPC.ResultsMarrow homing of murine MPB HPC, generated by different mobilizing schemes, was reproducibly significantly superior to that of ssBM, in lethally irradiated as well as in nonirradiated hosts. This phenotype was independent of MMP9, selectins, and beta2- and alpha4-integrins. Superior homing was also observed for human MPB HPC transplanted into NOD/SCIDbeta2microglobulin(-/-) recipients. Inhibition of HPC migration abrogated the homing advantage of MPB but did not affect homing of ssBM HPC, whereas enhancement of motility by CD26 inhibition improved marrow homing only of ssBM HPC. Enhanced SDF-1-dependent chemotaxis and low CD26 expression on MPB HPC were identified as potential contributing factors. Significant contributions of the putative alternative SDF-1 receptor, RDC1, were unlikely based on gene expression data.ConclusionThe data suggest increased motility as a converging endpoint of complex changes seen in MPB HPC which is likely responsible for their favorable homing.
| S-EPMC1847625 | biostudies-literature