Dataset Information

Parallel MapReduce: Maximizing Cloud Resource Utilization and Performance Improvement Using Parallel Execution Strategies.

ABSTRACT: MapReduce is the preferred cloud computing framework used in large data analysis and application processing. MapReduce frameworks currently in place suffer performance degradation due to the adoption of sequential processing approaches with little modification and thus exhibit underutilization of cloud resources. To overcome this drawback and reduce costs, we introduce a Parallel MapReduce (PMR) framework in this paper. We design a novel parallel execution strategy of Map and Reduce worker nodes. Our strategy enables further performance improvement and efficient utilization of cloud resources execution of Map and Reduce functions to utilize multicore environments available with computing nodes. We explain in detail makespan modeling and working principle of the PMR framework in the paper. Performance of PMR is compared with Hadoop through experiments considering three biomedical applications. Experiments conducted for BLAST, CAP3, and DeepBind biomedical applications report makespan time reduction of 38.92%, 18.00%, and 34.62% considering the PMR framework against Hadoop framework. Experiments' results prove that the PMR cloud computing platform proposed is robust, cost-effective, and scalable, which sufficiently supports diverse applications on public and private cloud platforms. Consequently, overall presentation and results indicate that there is good matching between theoretical makespan modeling presented and experimental values investigated.

SUBMITTER: Al-Absi AA

PROVIDER: S-EPMC6207866 | biostudies-literature | 2018

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Parallel MapReduce: Maximizing Cloud Resource Utilization and Performance Improvement Using Parallel Execution Strategies.

Al-Absi Ahmed Abdulhakim AA Al-Sammarraie Najeeb Abbas NA Shaher Yafooz Wael Mohamed WM Kang Dae-Ki DK

BioMed research international 20181017

MapReduce is the preferred cloud computing framework used in large data analysis and application processing. MapReduce frameworks currently in place suffer performance degradation due to the adoption of sequential processing approaches with little modification and thus exhibit underutilization of cloud resources. To overcome this drawback and reduce costs, we introduce a Parallel MapReduce (<i>PMR</i>) framework in this paper. We design a novel parallel execution strategy of Map and Reduce worke ...[more]

PMID: 30417014

Similar Datasets

Project description:BackgroundThe MapReduce framework enables a scalable processing and analyzing of large datasets by distributing the computational load on connected computer nodes, referred to as a cluster. In Bioinformatics, MapReduce has already been adopted to various case scenarios such as mapping next generation sequencing data to a reference genome, finding SNPs from short read data or matching strings in genotype files. Nevertheless, tasks like installing and maintaining MapReduce on a cluster system, importing data into its distributed file system or executing MapReduce programs require advanced knowledge in computer science and could thus prevent scientists from usage of currently available and useful software solutions.ResultsHere we present Cloudgene, a freely available platform to improve the usability of MapReduce programs in Bioinformatics by providing a graphical user interface for the execution, the import and export of data and the reproducibility of workflows on in-house (private clouds) and rented clusters (public clouds). The aim of Cloudgene is to build a standardized graphical execution environment for currently available and future MapReduce programs, which can all be integrated by using its plug-in interface. Since Cloudgene can be executed on private clusters, sensitive datasets can be kept in house at all time and data transfer times are therefore minimized.ConclusionsOur results show that MapReduce programs can be integrated into Cloudgene with little effort and without adding any computational overhead to existing programs. This platform gives developers the opportunity to focus on the actual implementation task and provides scientists a platform with the aim to hide the complexity of MapReduce. In addition to MapReduce programs, Cloudgene can also be used to launch predefined systems (e.g. Cloud BioLinux, RStudio) in public clouds. Currently, five different bioinformatic programs using MapReduce and two systems are integrated and have been successfully deployed. Cloudgene is freely available at http://cloudgene.uibk.ac.at.

Project description:BACKGROUND: High-throughput molecular profiling data has been used to improve clinical decision making by stratifying subjects based on their molecular profiles. Unsupervised clustering algorithms can be used for stratification purposes. However, the current speed of the clustering algorithms cannot meet the requirement of large-scale molecular data due to poor performance of the correlation matrix calculation. With high-throughput sequencing technologies promising to produce even larger datasets per subject, we expect the performance of the state-of-the-art statistical algorithms to be further impacted unless efforts towards optimisation are carried out. MapReduce is a widely used high performance parallel framework that can solve the problem. RESULTS: In this paper, we evaluate the current parallel modes for correlation calculation methods and introduce an efficient data distribution and parallel calculation algorithm based on MapReduce to optimise the correlation calculation. We studied the performance of our algorithm using two gene expression benchmarks. In the micro-benchmark, our implementation using MapReduce, based on the R package RHIPE, demonstrates a 3.26-5.83 fold increase compared to the default Snowfall and 1.56-1.64 fold increase compared to the basic RHIPE in the Euclidean, Pearson and Spearman correlations. Though vanilla R and the optimised Snowfall outperforms our optimised RHIPE in the micro-benchmark, they do not scale well with the macro-benchmark. In the macro-benchmark the optimised RHIPE performs 2.03-16.56 times faster than vanilla R. Benefiting from the 3.30-5.13 times faster data preparation, the optimised RHIPE performs 1.22-1.71 times faster than the optimised Snowfall. Both the optimised RHIPE and the optimised Snowfall successfully performs the Kendall correlation with TCGA dataset within 7 hours. Both of them conduct more than 30 times faster than the estimated vanilla R. CONCLUSIONS: The performance evaluation found that the new MapReduce algorithm and its implementation in RHIPE outperforms vanilla R and the conventional parallel algorithms implemented in R Snowfall. We propose that MapReduce framework holds great promise for large molecular data analysis, in particular for high-dimensional genomic data such as that demonstrated in the performance evaluation described in this paper. We aim to use this new algorithm as a basis for optimising high-throughput molecular data correlation calculation for Big Data.

Project description:Located in the Eurasian continent's hinterland, Xinjiang is a typical arid and resource-developing region in China's northwest. Problems such as excessive resource consumption, environmental pollution, and ecological imbalance are becoming severe, which have become the bottleneck that further restricts Xinjiang's sustainable development. Due to its outstanding quantitative advantages, ecological efficiency has become a significant indicator and analytical tool for measuring the green economy and sustainable development. In this study, we analyzed ecological efficiency variation for Xinjiang's 14 prefectures between 2001 and 2015 using a super-efficient data envelopment model (DEA), Malmquist Index, and Tobit model. These analyses indicated that: (1) The overall ecological efficiency level of Xinjiang is low, and development among regions is unbalanced, out of sync, lacks sustainability. (2) From 2001 to 2015, Xinjiang's ecological efficiency showed a W-shaped rising trend and finally increased by 5.7%. It is due to the substantial improvement in environmental efficiency. (3) By analyzing the environmental efficiency and resource efficiency, 14 prefectures in Xinjiang consist of four development modes: low energy consumption and low emission, high energy consumption and low emission, low energy consumption and high emission, and high energy consumption and high emission. (4) Water resources are restricting factors of arid regions. In most prefectures, there exist excessive water resource investment, excessive COD, and NH3-N emissions. (5) By analyzing the Malmquist index, it shows that the technical progress index(TC) restricted ecological efficiency. In contrast, the technical efficiency index (EC) promoted ecological efficiency.(6)The ecological efficiency was positively correlated with the utilization of foreign capital, urbanization rate, and average education degree but negatively correlated with the marketization degree. The study has guidance and reference function for the sustainable development of Xinjiang-a vital corridor of the Silk Road Economic Belt, and also provides a reference to the research work of other arid resource-based regions.

Dataset Information

Parallel MapReduce: Maximizing Cloud Resource Utilization and Performance Improvement Using Parallel Execution Strategies.

Publications

Parallel MapReduce: Maximizing Cloud Resource Utilization and Performance Improvement Using Parallel Execution Strategies.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets