Reproducible and replicable comparisons using SummarizedBenchmark.
Ontology highlight
ABSTRACT:
SUBMITTER: Kimes PK
PROVIDER: S-EPMC6298041 | biostudies-literature | 2019 Jan
REPOSITORIES: biostudies-literature
Publications
Reproducible and replicable comparisons using SummarizedBenchmark.
Kimes Patrick K PK Reyes Alejandro A
Bioinformatics (Oxford, England) 20190101 1
<h4>Summary</h4>Benchmark studies are widely used to compare and evaluate tools developed for answering various biological questions. Despite the popularity of these comparisons, the implementation is often ad hoc, with little consistency across studies. To address this problem, we developed SummarizedBenchmark, an R package and framework for organizing and structuring benchmark comparisons. SummarizedBenchmark defines a general grammar for benchmarking and allows for easier setup and execution ...[more]
Similar Datasets
Project description: Not available
| S-EPMC6742427 | biostudies-literature
Project description:A lack of replicable test systems that realistically simulate hot water premise plumbing conditions at the laboratory-scale is an obstacle to identifying key factors that support growth of opportunistic pathogens (OPs) and opportunities to stem disease transmission. Here we developed the convectively-mixed pipe reactor (CMPR) as a simple reproducible system, consisting of off-the-shelf plumbing materials, that self-mixes through natural convective currents and enables testing of multiple, replicated, and realistic premise plumbing conditions in parallel. A 10-week validation study was conducted, comparing three pipe materials (PVC, PVC-copper, and PVC-iron; n = 18 each) to stagnant control pipes without convective mixing (n = 3 each). Replicate CMPRs were found to yield consistent water chemistry as a function of pipe material, with differences becoming less discernable by week 9. Temperature, an overarching factor known to control OP growth, was consistently maintained across all 54 CMPRs, with a coefficient of variation <2%. Dissolved oxygen (DO) remained lower in PVC-iron (1.96 ± 0.29 mg/L) than in PVC (5.71 ± 0.22 mg/L) or PVC-copper (5.90 ± 0.38 mg/L) CMPRs as expected due to corrosion. Further, DO in PVC-iron CMPRs was 33% of that observed in corresponding stagnant pipes (6.03 ± 0.33 mg/L), demonstrating the important role of internal convective mixing in stimulating corrosion and microbiological respiration. 16S rRNA gene amplicon sequencing indicated that both bulk water (Padonis = 0.001, R2 = 0.222, Pbetadis = 0.785) and biofilm (Padonis = 0.001, R2 = 0.119, Pbetadis = 0.827) microbial communities differed between CMPR versus stagnant pipes, consistent with creation of a distinct ecological niche. Overall, CMPRs can provide a more realistic simulation of certain aspects of premise plumbing than reactors commonly applied in prior research, at a fraction of the cost, space, and water demand of large pilot-scale rigs.
| S-EPMC7494094 | biostudies-literature
Project description:ImportanceNeurodevelopmental conditions, such as autism spectrum disorder (ASD), attention-deficit/hyperactivity disorder (ADHD), and obsessive-compulsive disorder (OCD), have highly heterogeneous and overlapping phenotypes and neurobiology. Data-driven approaches are beginning to identify homogeneous transdiagnostic subgroups of children; however, findings have yet to be replicated in independently collected data sets, a necessity for translation into clinical settings.ObjectiveTo identify subgroups of children with and without neurodevelopmental conditions with shared functional brain characteristics using data from 2 large, independent data sets.Design, setting, and participantsThis case-control study used data from the Province of Ontario Neurodevelopmental (POND) network (study recruitment began June 2012 and is ongoing; data were extracted April 2021) and the Healthy Brain Network (HBN; study recruitment began May 2015 and is ongoing; data were extracted November 2020). POND and HBN data are collected from institutions across Ontario and New York, respectively. Participants who had diagnoses of ASD, ADHD, and OCD or were typically developing (TD); were aged between 5 and 19 years; and successfully completed the resting-state and anatomical neuroimaging protocol were included in the current study.Main outcomes and measuresThe analyses consisted of a data-driven clustering procedure on measures derived from each participant's resting-state functional connectome, performed independently on each data set. Differences between each pair of leaves in the resulting clustering decision trees in the demographic and clinical characteristics were tested.ResultsOverall, 551 children and adolescents were included from each data set. POND included 164 participants with ADHD; 217 with ASD; 60 with OCD; and 110 with TD (median [IQR] age, 11.87 [9.51-14.76] years; 393 [71.2%] male participants; 20 [3.6%] Black, 28 [5.1%] Latino, and 299 [54.2%] White participants) and HBN included 374 participants with ADHD; 66 with ASD; 11 with OCD; and 100 with TD (median [IQR] age, 11.50 [9.22-14.20] years; 390 [70.8%] male participants; 82 [14.9%] Black, 57 [10.3%] Hispanic, and 257 [46.6%] White participants). In both data sets, subgroups with similar biology that differed significantly in intelligence as well as hyperactivity and impulsivity problems were identified, yet these groups showed no consistent alignment with current diagnostic categories. For example, there was a significant difference in Strengths and Weaknesses ADHD Symptoms and Normal Behavior Hyperactivity/Impulsivity subscale (SWAN-HI) between 2 subgroups in the POND data (C and D), with subgroup D having increased hyperactivity and impulsivity traits compared with subgroup C (median [IQR], 2.50 [0.00-7.00] vs 1.00 [0.00-5.00]; U = 1.19 × 104; P = .01; η2 = 0.02). A significant difference in SWAN-HI scores between subgroups g and d in the HBN data was also observed (median [IQR], 1.00 [0.00-4.00] vs 0.00 [0.00-2.00]; corrected P = .02). There were no differences in the proportion of each diagnosis between the subgroups in either data set.Conclusions and relevanceThe findings of this study suggest that homogeneity in the neurobiology of neurodevelopmental conditions transcends diagnostic boundaries and is instead associated with behavioral characteristics. This work takes an important step toward translating neurobiological subgroups into clinical settings by being the first to replicate our findings in independently collected data sets.
| S-EPMC10011941 | biostudies-literature
Project description:SummaryWe present a new R package admixr, which provides a convenient interface for performing reproducible population genetic analyses (f3, D, f4, f4-ratio, qpWave and qpAdm), as implemented by command-line programs in the ADMIXTOOLS software suite. In a traditional ADMIXTOOLS workflow, the user must first generate a set of text configuration files tailored to each individual analysis, often using a combination of shell scripting and manual text editing. The non-tabular output files then need to be parsed to extract values of interest prior to further analyses. Our package simplifies this process by automating all low-level configuration and parsing steps, making analyses as simple as running a single R command. Furthermore, we provide a set of R functions for processing, filtering and manipulating datasets in the EIGENSTRAT format. By unifying all steps of the workflow under a single R framework, this package enables the automation of analytic pipelines, significantly improving the reproducibility of population genetic studies.Availability and implementationThe source code of the R package is available under the MIT license. Installation instructions, reference manual and a tutorial can be found on the package website at https://bioinf.eva.mpg.de/admixr.Supplementary informationSupplementary data are available at Bioinformatics online.
| S-EPMC6736366 | biostudies-literature
Project description:A modern biomedical research project can easily contain hundreds of analysis steps and lack of reproducibility of the analyses has been recognized as a severe issue. While thorough documentation enables reproducibility, the number of analysis programs used can be so large that in reality reproducibility cannot be easily achieved. Literate programming is an approach to present computer programs to human readers. The code is rearranged to follow the logic of the program, and to explain that logic in a natural language. The code executed by the computer is extracted from the literate source code. As such, literate programming is an ideal formalism for systematizing analysis steps in biomedical research. We have developed the reproducible computing tool Lir (literate, reproducible computing) that allows a tool-agnostic approach to biomedical data analysis. We demonstrate the utility of Lir by applying it to a case study. Our aim was to investigate the role of endosomal trafficking regulators to the progression of breast cancer. In this analysis, a variety of tools were combined to interpret the available data: a relational database, standard command-line tools, and a statistical computing environment. The analysis revealed that the lipid transport related genes LAPTM4B and NDRG1 are coamplified in breast cancer patients, and identified genes potentially cooperating with LAPTM4B in breast cancer progression. Our case study demonstrates that with Lir, an array of tools can be combined in the same data analysis to improve efficiency, reproducibility, and ease of understanding. Lir is an open-source software available at github.com/borisvassilev/lir.
| S-EPMC5053501 | biostudies-literature
Project description:We present an educational unit to teach computational modeling, a vital part of chemical engineering curricula, through the lens of synthetic biology. Lectures, code, and homework questions provide conceptual and practical introductions to each computational method involved in the model development process, along with perspectives on how methods can be iterated upon to arrive at a final model. Ultimately, this content can be applied broadly to address questions in synthetic biology and classical chemical engineering.
| S-EPMC10588546 | biostudies-literature
Project description:This article considers replicability of the performance of predictors across studies. We suggest a general approach to investigating this issue, based on ensembles of prediction models trained on different studies. We quantify how the common practice of training on a single study accounts in part for the observed challenges in replicability of prediction performance. We also investigate whether ensembles of predictors trained on multiple studies can be combined, using unique criteria, to design robust ensemble learners trained upfront to incorporate replicability into different contexts and populations.
| S-EPMC5856504 | biostudies-literature
Project description:Chemotaxis is the process by which motile bacteria sense their chemical environment and move towards more favourable conditions. Escherichia coli utilises a single sensory pathway, but little is known about signalling pathways in species with more complex systems.To investigate whether chemotaxis pathways in other bacteria follow the E. coli paradigm, we analysed 206 species encoding at least 1 homologue of each of the 5 core chemotaxis proteins (CheA, CheB, CheR, CheW and CheY). 61 species encode more than one of all of these 5 proteins, suggesting they have multiple chemotaxis pathways. Operon information is not available for most bacteria, so we developed a novel statistical approach to cluster che genes into putative operons. Using operon-based models, we reconstructed putative chemotaxis pathways for all 206 species. We show that cheA-cheW and cheR-cheB have strong preferences to occur in the same operon as two-gene blocks, which may reflect a functional requirement for co-transcription. However, other che genes, most notably cheY, are more dispersed on the genome. Comparison of our operons with shuffled equivalents demonstrates that specific patterns of genomic location may be a determining factor for the observed in vivo chemotaxis pathways. We then examined the chemotaxis pathways of Rhodobacter sphaeroides. Here, the PpfA protein is known to be critical for correct partitioning of proteins in the cytoplasmically-localised pathway. We found ppfA in che operons of many species, suggesting that partitioning of cytoplasmic Che protein clusters is common. We also examined the apparently non-typical chemotaxis components, CheA3, CheA4 and CheY6. We found that though variants of CheA proteins are rare, the CheY6 variant may be a common type of CheY, with a significantly disordered C-terminal region which may be functionally significant.We find that many bacterial species potentially have multiple chemotaxis pathways, with grouping of che genes into operons likely to be a major factor in keeping signalling pathways distinct. Gene order is highly conserved with cheA-cheW and cheR-cheB blocks, perhaps reflecting functional linkage. CheY behaves differently to other Che proteins, both in its genomic location and its putative protein interactions, which should be considered when modelling chemotaxis pathways.
| S-EPMC2829493 | biostudies-literature
Project description:In multivariate normative comparisons, a patient's profile of test scores is compared to those in a normative sample. Recently, it has been shown that these multivariate normative comparisons enhance the sensitivity of neuropsychological assessment. However, multivariate normative comparisons require multivariate normative data, which are often unavailable. In this paper, we show how a multivariate normative database can be constructed by combining healthy control group data from published neuropsychological studies. We show that three issues should be addressed to construct a multivariate normative database. First, the database may have a multilevel structure, with participants nested within studies. Second, not all tests are administered in every study, so many data may be missing. Third, a patient should be compared to controls of similar age, gender and educational background rather than to the entire normative sample. To address these issues, we propose a multilevel approach for multivariate normative comparisons that accounts for missing data and includes covariates for age, gender and educational background. Simulations show that this approach controls the number of false positives and has high sensitivity to detect genuine deviations from the norm. An empirical example is provided. Implications for other domains than neuropsychology are also discussed. To facilitate broader adoption of these methods, we provide code implementing the entire analysis in the open source software package R.
| S-EPMC5340373 | biostudies-literature
Project description:Reproducible, comprehensive phosphopeptide enrichment is essential for studying phosphorylation-regulated processes. Here, we describe the application of hyper-porous magnetic TiO2 and Ti-IMAC microspheres for uniform automated phosphopeptide enrichment. Combining magnetic microspheres with a magnetic particle-handling robot enables rapid (45 min), reproducible (r2 ? 0.80) and high-fidelity (>90% purity) phosphopeptide purification in a 96-well format. Automated phosphopeptide enrichment demonstrates reproducible synthetic phosphopeptide recovery across 2 orders of magnitude, "well-to-well" quantitative reproducibility indistinguishable to internal SILAC standards, and robust "plate-to-plate" reproducibility across 5 days of independent enrichments. As a result, automated phosphopeptide enrichment enables statistical analysis of label-free phosphoproteomic samples in a high-throughput manner. This technique uses commercially available, off-the-shelf components and can be easily adopted by any laboratory interested in phosphoproteomic analysis. We provide a free downloadable automated phosphopeptide enrichment program to facilitate uniform interlaboratory collaboration and exchange of phosphoproteomic data sets.
| S-EPMC4206527 | biostudies-literature