Swarm v3: towards tera-scale amplicon clustering.
Ontology highlight
ABSTRACT: Motivation
Previously we presented swarm, an open-source amplicon clustering program that produces fine-scale molecular operational taxonomic units (OTUs) that are free of arbitrary global clustering thresholds. Here we present swarm v3 to address issues of contemporary datasets that are growing towards tera-byte sizes.Results
When compared to previous swarm versions, swarm v3 has modernized C ++ source code, reduced memory footprint by up to 50%, optimized CPU-usage and multithreading (more than 7 times faster with default parameters), and it has been extensively tested for its robustness and logic.Availability
Source code and binaries are available at https://github.com/torognes/swarm.Supplementary information
Supplementary data are available at Bioinformatics online.
SUBMITTER: Mahe F
PROVIDER: S-EPMC8696092 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Previously we presented Swarm v1, a novel and open source amplicon clustering program that produced fine-scale molecular operational taxonomic units (OTUs), free of arbitrary global clustering thresholds and input-order dependency. Swarm v1 worked with an initial phase that used iterative single-linkage with a local clustering threshold (d), followed by a phase that used the internal abundance structures of clusters to break chained OTUs. Here we present Swarm v2, which has two important novel features: (1) a new algorithm for d = 1 that allows the computation time of the program to scale linearly with increasing amounts of data; and (2) the new fastidious option that reduces under-grouping by grafting low abundant OTUs (e.g., singletons and doubletons) onto larger ones. Swarm v2 also directly integrates the clustering and breaking phases, dereplicates sequencing reads with d = 0, outputs OTU representatives in fasta format, and plots individual OTUs as two-dimensional networks.
| S-EPMC4690345 | biostudies-literature
Project description:Popular de novo amplicon clustering methods suffer from two fundamental flaws: arbitrary global clustering thresholds, and input-order dependency induced by centroid selection. Swarm was developed to address these issues by first clustering nearly identical amplicons iteratively using a local threshold, and then by using clusters' internal structure and amplicon abundances to refine its results. This fast, scalable, and input-order independent approach reduces the influence of clustering parameters and produces robust operational taxonomic units.
| S-EPMC4178461 | biostudies-literature
Project description:Multi-objective clustering has received widespread attention recently, as it can obtain more accurate and reasonable solution. In this paper, an improved multi-objective clustering framework using particle swarm optimization (IMCPSO) is proposed. Firstly, a novel particle representation for clustering problem is designed to help PSO search clustering solutions in continuous space. Secondly, the distribution of Pareto set is analyzed. The analysis results are applied to the leader selection strategy, and make algorithm avoid trapping in local optimum. Moreover, a clustering solution-improved method is proposed, which can increase the efficiency in searching clustering solution greatly. In the experiments, 28 datasets are used and nine state-of-the-art clustering algorithms are compared, the proposed method is superior to other approaches in the evaluation index ARI.
| S-EPMC5716574 | biostudies-literature
Project description:This paper puts forward a new automatic clustering algorithm based on Multi-Objective Particle Swarm Optimization and Simulated Annealing, "MOPSOSA". The proposed algorithm is capable of automatic clustering which is appropriate for partitioning datasets to a suitable number of clusters. MOPSOSA combines the features of the multi-objective based particle swarm optimization (PSO) and the Multi-Objective Simulated Annealing (MOSA). Three cluster validity indices were optimized simultaneously to establish the suitable number of clusters and the appropriate clustering for a dataset. The first cluster validity index is centred on Euclidean distance, the second on the point symmetry distance, and the last cluster validity index is based on short distance. A number of algorithms have been compared with the MOPSOSA algorithm in resolving clustering problems by determining the actual number of clusters and optimal clustering. Computational experiments were carried out to study fourteen artificial and five real life datasets.
| S-EPMC4488466 | biostudies-literature
Project description:Database peptide search algorithms deduce peptides from mass spectrometry (MS) data. There has been substantial effort in improving their computational efficiency to achieve larger and more complex systems biology studies. However, modern serial and high-performance computing (HPC) algorithms exhibit sub-optimal performance mainly due to their ineffective parallel designs (low resource utilization), and high overhead costs. We present an HPC framework, called HiCOPS, for efficient acceleration of the database peptide search algorithms on distributed-memory supercomputers. HiCOPS provides, on average, more than 10-fold improvement in speed, and superior parallel performance over several existing HPC database search software. We also formulate a mathematical model for performance analysis and optimization, and report near-optimal results for several key metrics including strong-scale efficiency, hardware utilization, load-balance, inter-process communication and I/O overheads. The core parallel design, techniques, and optimizations presented in HiCOPS are search-algorithm independent and can be extended to efficiently accelerate the existing and future algorithms and software.
| S-EPMC8554525 | biostudies-literature
Project description:How can we detect fraudulent lockstep behavior in large-scale multi-aspect data (i.e., tensors)? Can we detect it when data are too large to fit in memory or even on a disk? Past studies have shown that dense subtensors in real-world tensors (e.g., social media, Wikipedia, TCP dumps, etc.) signal anomalous or fraudulent behavior such as retweet boosting, bot activities, and network attacks. Thus, various approaches, including tensor decomposition and search, have been proposed for detecting dense subtensors rapidly and accurately. However, existing methods suffer from low accuracy, or they assume that tensors are small enough to fit in main memory, which is unrealistic in many real-world applications such as social media and web. To overcome these limitations, we propose D-Cube, a disk-based dense-subtensor detection method, which also can run in a distributed manner across multiple machines. Compared to state-of-the-art methods, D-Cube is (1) Memory Efficient: requires up to 1,561× less memory and handles 1,000× larger data (2.6TB), (2) Fast: up to 7× faster due to its near-linear scalability, (3) Provably Accurate: gives a guarantee on the densities of the detected subtensors, and (4) Effective: spotted network attacks from TCP dumps and synchronized behavior in rating data most accurately.
| S-EPMC8118605 | biostudies-literature
Project description:Data clustering is commonly employed in many disciplines. The aim of clustering is to partition a set of data into clusters, in which objects within the same cluster are similar and dissimilar to other objects that belong to different clusters. Over the past decade, the evolutionary algorithm has been commonly used to solve clustering problems. This study presents a novel algorithm based on simplified swarm optimization, an emerging population-based stochastic optimization approach with the advantages of simplicity, efficiency, and flexibility. This approach combines variable vibrating search (VVS) and rapid centralized strategy (RCS) in dealing with clustering problem. VVS is an exploitation search scheme that can refine the quality of solutions by searching the extreme points nearby the global best position. RCS is developed to accelerate the convergence rate of the algorithm by using the arithmetic average. To empirically evaluate the performance of the proposed algorithm, experiments are examined using 12 benchmark datasets, and corresponding results are compared with recent works. Results of statistical analysis indicate that the proposed algorithm is competitive in terms of the quality of solutions.
| S-EPMC4562660 | biostudies-literature
Project description:A clustering algorithm is a solution for grouping a set of objects and for distribution centre location problems. But the common K-means clustering algorithm may give local optimal solutions. Swarm intelligent algorithms simulate the social behaviours of animals and avoid local optimal solutions. We employ three swarm intelligent algorithms to avoid these solutions. We propose a new algorithm for the clustering problem, the fruit-fly optimization K-means algorithm (FOA K-means). We designed a distribution centre location problem and three clustering indicators to evaluate the performance of algorithms. We compare the algorithms of K-means with the ant colony optimization algorithm (ACO K-means), particle swarm optimization algorithm (PSO K-means), and fruit-fly optimization algorithm. We find K-Means modified by the fruit-fly optimization algorithm (FOA K-means) has the best performance on convergence speed and three clustering indicators, compactness, separation, and integration. Thus, we can apply FOA K-means to improve the distribution centre location solution and the efficiency for distribution in the future.
| S-EPMC9409525 | biostudies-literature
Project description:BackgroundThe V3 loop of the glycoprotein gp120 of HIV-1 plays an important role in viral entry into cells by utilizing as coreceptor CCR5 or CXCR4, and is implicated in the phenotypic tropisms of HIV viruses. It has been hypothesized that the interaction between the V3 loop and CCR5 or CXCR4 is mediated by electrostatics. We have performed hierarchical clustering analysis of the spatial distributions of electrostatic potentials and charges of V3 loop structures containing consensus sequences of HIV-1 subtypes.ResultsAlthough the majority of consensus sequences have a net charge of +3, the spatial distribution of their electrostatic potentials and charges may be a discriminating factor for binding and infectivity. This is demonstrated by the formation of several small subclusters, within major clusters, which indicates common origin but distinct spatial details of electrostatic properties. Some of this information may be present, in a coarse manner, in clustering of sequences, but the spatial details are largely lost. We show the effect of ionic strength on clustering of electrostatic potentials, information that is not present in clustering of charges or sequences. We also make correlations between clustering of electrostatic potentials and net charge, coreceptor selectivity, global prevalence, and geographic distribution. Finally, we interpret coreceptor selectivity based on the N6X7T8|S8X9 sequence glycosylation motif, the specific positive charge location according to the 11/24/25 rule, and the overall charge and electrostatic potential distribution.ConclusionsWe propose that in addition to the sequence and the net charge of the V3 loop of each subtype, the spatial distributions of electrostatic potentials and charges may also be important factors for receptor recognition and binding and subsequent viral entry into cells. This implies that the overall electrostatic potential is responsible for long-range recognition of the V3 loop with coreceptors CCR5/CXCR4, whereas the charge distribution contributes to the specific short-range interactions responsible for the formation of the bound complex. We also propose a scheme for coreceptor selectivity based on the sequence glycosylation motif, the 11/24/25 rule, and net charge.
| S-EPMC3295656 | biostudies-literature
Project description:Genomic data analysis across multiple cloud platforms is an ongoing challenge, especially when large amounts of data are involved. Here, we present Swarm, a framework for federated computation that promotes minimal data motion and facilitates crosstalk between genomic datasets stored on various cloud platforms. We demonstrate its utility via common inquiries of genomic variants across BigQuery in the Google Cloud Platform (GCP), Athena in the Amazon Web Services (AWS), Apache Presto and MySQL. Compared to single-cloud platforms, the Swarm framework significantly reduced computational costs, run-time delays and risks of security breach and privacy violation.
| S-EPMC8143397 | biostudies-literature