BioMake: a GNU make-compatible utility for declarative workflow management.
Ontology highlight
ABSTRACT: Motivation
The Unix 'make' program is widely used in bioinformatics pipelines, but suffers from problems that limit its application to large analysis datasets. These include reliance on file modification times to determine whether a target is stale, lack of support for parallel execution on clusters, and restricted flexibility to extend the underlying logic program.Results
We present BioMake, a make-like utility that is compatible with most features of GNU Make and adds support for popular cluster-based job-queue engines, MD5 signatures as an alternative to timestamps, and logic programming extensions in Prolog.Availability and implementation
BioMake is available for MacOSX and Linux systems from https://github.com/evoldoers/biomake under the BSD3 license. The only dependency is SWI-Prolog (version 7), available from http://www.swi-prolog.org/.Contact
ihholmes?+?biomake@gmail.com or cmungall?+?biomake@gmail.com.Supplementary information
Feature table comparing BioMake to similar tools. Supplementary data are available at Bioinformatics online.
SUBMITTER: Holmes IH
PROVIDER: S-EPMC5860158 | biostudies-literature | 2017 Nov
REPOSITORIES: biostudies-literature
Publications
BioMake: a GNU make-compatible utility for declarative workflow management.
Holmes Ian H IH Mungall Christopher J CJ
Bioinformatics (Oxford, England) 20171101 21
<h4>Motivation</h4>The Unix 'make' program is widely used in bioinformatics pipelines, but suffers from problems that limit its application to large analysis datasets. These include reliance on file modification times to determine whether a target is stale, lack of support for parallel execution on clusters, and restricted flexibility to extend the underlying logic program.<h4>Results</h4>We present BioMake, a make-like utility that is compatible with most features of GNU Make and adds support f ...[more]
Similar Datasets
Project description:The contribution of this paper is to describe how we can program neuroimaging workflow using Make, a software development tool designed for describing how to build executables from source files. A makefile (or a file of instructions for Make) consists of a set of rules that create or update target files if they have not been modified since their dependencies were last modified. These rules are processed to create a directed acyclic dependency graph that allows multiple entry points from which to execute the workflow. We show that using Make we can achieve many of the features of more sophisticated neuroimaging pipeline systems, including reproducibility, parallelization, fault tolerance, and quality assurance reports. We suggest that Make permits a large step toward these features with only a modest increase in programming demands over shell scripts. This approach reduces the technical skill and time required to write, debug, and maintain neuroimaging workflows in a dynamic environment, where pipelines are often modified to accommodate new best practices or to study the effect of alternative preprocessing steps, and where the underlying packages change frequently. This paper has a comprehensive accompanying manual with lab practicals and examples (see Supplemental Materials) and all data, scripts, and makefiles necessary to run the practicals and examples are available in the "makepipelines" project at NITRC.
| S-EPMC4735413 | biostudies-literature
Project description:Extending proteomics to smaller samples can enable the mapping of protein expression across tissues with high spatial resolution and can reveal sub-group heterogeneity. However, despite the continually improving sensitivity of LC-MS instrumentation, in-depth profiling of samples containing low-nanogram amounts of protein has remained challenging due to analyte losses incurred during preparation and analysis. To address this, we recently developed nanodroplet processing in one pot for trace samples (nanoPOTS), a robotic/microfluidic platform that generates ready-to-analyze peptides from cellular material in ~200 nL droplets with greatly reduced sample losses. In combination with ultrasensitive LC-MS, nanoPOTS has enabled >3000 proteins to be confidently identified from as few as 10 cultured human cells and ~700 proteins from single cells. However, the nanoPOTS platform requires a highly skilled operator and a costly in-house-built robotic nanopipetting instrument. In this work, we sought to evaluate the extent to which the benefits of nanodroplet processing could be preserved when upscaling reagent dispensing volumes by a factor of 10 to those addressable by commercial micropipette. We characterized the resulting platform, termed microdroplet processing in one pot for trace samples (μPOTS), for the analysis of as few as ~25 cultured HeLa cells (4 ng total protein) or 50 μm square mouse liver tissue thin sections and found that ~1800 and ~1200 unique proteins were respectively identified with high reproducibility. The reduced equipment requirements should facilitate broad dissemination of nanoproteomics workflows by obviating the need for a capital-intensive custom liquid handling system.
| S-EPMC6527493 | biostudies-literature
Project description:BACKGROUND: In bioinformatics projects, scientific workflow systems are widely used to manage computational procedures. Full-featured workflow systems have been proposed to fulfil the demand for workflow management. However, such systems tend to be over-weighted for actual bioinformatics practices. We realize that quick deployment of cutting-edge software implementing advanced algorithms and data formats, and continuous adaptation to changes in computational resources and the environment are often prioritized in scientific workflow management. These features have a greater affinity with the agile software development method through iterative development phases after trial and error.Here, we show the application of a scientific workflow system Pwrake to bioinformatics workflows. Pwrake is a parallel workflow extension of Ruby's standard build tool Rake, the flexibility of which has been demonstrated in the astronomy domain. Therefore, we hypothesize that Pwrake also has advantages in actual bioinformatics workflows. FINDINGS: We implemented the Pwrake workflows to process next generation sequencing data using the Genomic Analysis Toolkit (GATK) and Dindel. GATK and Dindel workflows are typical examples of sequential and parallel workflows, respectively. We found that in practice, actual scientific workflow development iterates over two phases, the workflow definition phase and the parameter adjustment phase. We introduced separate workflow definitions to help focus on each of the two developmental phases, as well as helper methods to simplify the descriptions. This approach increased iterative development efficiency. Moreover, we implemented combined workflows to demonstrate modularity of the GATK and Dindel workflows. CONCLUSIONS: Pwrake enables agile management of scientific workflows in the bioinformatics domain. The internal domain specific language design built on Ruby gives the flexibility of rakefiles for writing scientific workflows. Furthermore, readability and maintainability of rakefiles may facilitate sharing workflows among the scientific community. Workflows for GATK and Dindel are available at http://github.com/misshie/Workflows.
| S-EPMC3180464 | biostudies-literature
Project description:In bioinformatics, as well as other computationally intensive research fields, there is a need for workflows that can reliably produce consistent output, from known sources, independent of the software environment or configuration settings of the machine on which they are executed. Indeed, this is essential for controlled comparison between different observations and for the wider dissemination of workflows. However, providing this type of reproducibility and traceability is often complicated by the need to accommodate the myriad dependencies included in a larger body of software, each of which generally comes in various versions. Moreover, in many fields (bioinformatics being a prime example), these versions are subject to continual change due to rapidly evolving technologies, further complicating problems related to reproducibility. Here, we propose a principled approach for building analysis pipelines and managing their dependencies with GNU Guix. As a case study to demonstrate the utility of our approach, we present a set of highly reproducible pipelines called PiGx for the analysis of RNA sequencing, chromatin immunoprecipitation sequencing, bisulfite-treated DNA sequencing, and single-cell resolution RNA sequencing. All pipelines process raw experimental data and generate reports containing publication-ready plots and figures, with interactive report elements and standard observables. Users may install these highly reproducible packages and apply them to their own datasets without any special computational expertise beyond the use of the command line. We hope such a toolkit will provide immediate benefit to laboratory workers wishing to process their own datasets or bioinformaticians seeking to automate all, or parts of, their analyses. In the long term, we hope our approach to reproducibility will serve as a blueprint for reproducible workflows in other areas. Our pipelines, along with their corresponding documentation and sample reports, are available at http://bioinformatics.mdc-berlin.de/pigx.
| S-EPMC6275446 | biostudies-literature
Project description:Over the last 10 years, the number of approved disease modifying drugs acting on the focal inflammatory process in Multiple Sclerosis (MS) has increased from 3 to 10. This wide choice offers the opportunity of a personalized medicine with the objective of no clinical and radiological activity for each patient. This new paradigm requires the optimization of the detection of new FLAIR lesions on longitudinal MRI. In this paper, we describe a complete workflow-that we developed, implemented, deployed, and evaluated-to facilitate the monitoring of new FLAIR lesions on longitudinal MRI of MS patients. This workflow has been designed to be usable by both hospital and private neurologists and radiologists in France. It consists of three main components: (i) a software component that allows for automated and secured anonymization and transfer of MRI data from the clinical Picture Archive and Communication System (PACS) to a processing server (and vice-versa); (ii) a fully automated segmentation core that enables detection of focal longitudinal changes in patients from T1-weighted, T2-weighted and FLAIR brain MRI scans, and (iii) a dedicated web viewer that provides an intuitive visualization of new lesions to radiologists and neurologists. We first present these different components. Then, we evaluate the workflow on 54 pairs of longitudinal MRI scans that were analyzed by 3 experts (1 neuroradiologist, 1 radiologist, and 1 neurologist) with and without the proposed workflow. We show that our workflow provided a valuable aid to clinicians in detecting new MS lesions both in terms of accuracy (mean number of detected lesions per patient and per expert 1.8 without the workflow vs. 2.3 with the workflow, p = 5.10-4) and of time dedicated by the experts (mean time difference 2'45″, p = 10-4). This increase in the number of detected lesions has implications in the classification of MS patients as stable or active, even for the most experienced neuroradiologist (mean sensitivity was 0.74 without the workflow and 0.90 with the workflow, p-value for no difference = 0.003). It therefore has potential consequences on the therapeutic management of MS patients.
| S-EPMC8595265 | biostudies-literature
Project description:BACKGROUND: The completion of the Human Genome Project has resulted in large quantities of biological data which are proving difficult to manage and integrate effectively. There is a need for a system that is able to automate accesses to remote sites and to "understand" the information that it is managing in order to link data properly. Workflow management systems combined with Web Services are promising Information and Communication Technologies (ICT) tools. Some have already been proposed and are being increasingly applied to the biomedical domain, especially as many biology-related Web Services are now becoming available. Information on biological resources and on genomic sequences mutations are two examples of very specialized datasets that are useful for specific research domains. RESULTS: The architecture of a system that is able to access and execute predefined workflows is presented in this paper. Web Services allowing access to the IARC TP53 Mutation Database and CABRI catalogues of biological resources have been implemented and are available on-line. Example workflows which retrieve data from these Web Services have also been created and are available on-line. CONCLUSION: We present a general architecture and some building blocks for the implementation of a system that is able to remotely execute workflows of biomedical interest and show how this approach can effectively produce useful outputs. The further development and implementation of Web Services allowing access to an exhaustive set of biomedical databases and the creation of effective and useful workflows will improve the automation of in-silico analysis.
| S-EPMC1866383 | biostudies-literature
Project description:Revealed preference theory provides axiomatic tools for assessing whether individuals make observable choices "as if" they are maximizing an underlying utility function. The theory evokes a tradeoff between goods whereby individuals improve themselves by trading one good for another good to obtain the best combination. Preferences revealed in these choices are modeled as curves of equal choice (indifference curves) and reflect an underlying process of optimization. These notions have far-reaching applications in consumer choice theory and impact the welfare of human and animal populations. However, they lack the empirical implementation in animals that would be required to establish a common biological basis. In a design using basic features of revealed preference theory, we measured in rhesus monkeys the frequency of repeated choices between bundles of two liquids. For various liquids, the animals' choices were compatible with the notion of giving up a quantity of one good to gain one unit of another good while maintaining choice indifference, thereby implementing the concept of marginal rate of substitution. The indifference maps consisted of nonoverlapping, linear, convex, and occasionally concave curves with typically negative, but also sometimes positive, slopes depending on bundle composition. Out-of-sample predictions using homothetic polynomials validated the indifference curves. The animals' preferences were internally consistent in satisfying transitivity. Change of option set size demonstrated choice optimality and satisfied the Weak Axiom of Revealed Preference (WARP). These data are consistent with a version of revealed preference theory in which preferences are stochastic; the monkeys behaved "as if" they had well-structured preferences and maximized utility.
| S-EPMC5347590 | biostudies-literature
Project description:Nurses are at the forefront of providing healthcare services to individuals of all age groups and with varying medical conditions. Aside from the critical knowledge and technical skills from nursing science, advancement in technology has assisted nurses in delivering quality nursing care by streamlining workflow processes and ensuring that data can easily be retrieved or modified. Electronic health records dramatically changed the landscape of the healthcare practice by providing an electronic means to store data and for healthcare professionals to retrieve and manipulate health information in a secured and collaborative environment. But with the nature of data being stored in the electronic health records, nurses still need to organize and process these data into relevant information, knowledge or wisdom so they can provide better holistic care to patients. This discussion paper details the role of content management systems in addressing nursing workflow by providing a mechanism for nurses to be developers themselves, and not just users or consumers of health innovative technologies. By using content management systems as platform for application development, nurses or other healthcare professionals, may be able to address problems with internal workflow without having to incur huge amounts in software development, or having to extensively learn programming languages.
| S-EPMC6839280 | biostudies-literature
Project description:Workflow management systems represent, manage, and execute multistep computational analyses and offer many benefits to bioinformaticians. They provide a common language for describing analysis workflows, contributing to reproducibility and to building libraries of reusable components. They can support both incremental build and re-entrancy-the ability to selectively re-execute parts of a workflow in the presence of additional inputs or changes in configuration and to resume execution from where a workflow previously stopped. Many workflow management systems enhance portability by supporting the use of containers, high-performance computing (HPC) systems, and clouds. Most importantly, workflow management systems allow bioinformaticians to delegate how their workflows are run to the workflow management system and its developers. This frees the bioinformaticians to focus on what these workflows should do, on their data analyses, and on their science. RiboViz is a package to extract biological insight from ribosome profiling data to help advance understanding of protein synthesis. At the heart of RiboViz is an analysis workflow, implemented in a Python script. To conform to best practices for scientific computing which recommend the use of build tools to automate workflows and to reuse code instead of rewriting it, the authors reimplemented this workflow within a workflow management system. To select a workflow management system, a rapid survey of available systems was undertaken, and candidates were shortlisted: Snakemake, cwltool, Toil, and Nextflow. Each candidate was evaluated by quickly prototyping a subset of the RiboViz workflow, and Nextflow was chosen. The selection process took 10 person-days, a small cost for the assurance that Nextflow satisfied the authors' requirements. The use of prototyping can offer a low-cost way of making a more informed selection of software to use within projects, rather than relying solely upon reviews and recommendations by others.
| S-EPMC7906312 | biostudies-literature
Project description:Collaborative incident learning initiatives in radiation therapy promise to improve and standardize the quality of care provided by participating institutions. However, the software interfaces provided with such initiatives must accommodate all participants and thus are not optimized for the workflows of individual radiation therapy centers. This article describes the development and implementation of a radiation therapy incident learning system that is optimized for a clinical workflow and uses the taxonomy of the Canadian National System for Incident Reporting - Radiation Treatment (NSIR-RT).The described incident learning system is a novel version of an open-source software called the Safety and Incident Learning System (SaILS). A needs assessment was conducted prior to development to ensure SaILS (a) was intuitive and efficient (b) met changing staff needs and (c) accommodated revisions to NSIR-RT. The core functionality of SaILS includes incident reporting, investigations, tracking, and data visualization. Postlaunch modifications of SaILS were informed by discussion and a survey of radiation therapy staff.There were 240 incidents detected and reported using SaILS in 2016 and the number of incidents per month tended to increase throughout the year. An increase in incident reporting occurred after switching to fully online incident reporting from an initial hybrid paper-electronic system. Incident templating functionality and a connection with our center's oncology information system were incorporated into the investigation interface to minimize repetitive data entry. A taskable actions feature was also incorporated to document outcomes of incident reports and has since been utilized for 36% of reported incidents.Use of SaILS and the NSIR-RT taxonomy has improved the structure of, and staff engagement with, incident learning in our center. Software and workflow modifications informed by staff feedback improved the utility of SaILS and yielded an efficient and transparent solution to categorize incidents with the NSIR-RT taxonomy.
| S-EPMC5767999 | biostudies-literature