Browse
Submit Data
Databases
API
Help

Dataset Information

0 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Enabling Hardware Affinity in JVM-Based Applications: A Case Study for Big Data

ABSTRACT: Java has been the backbone of Big Data processing for more than a decade due to its interesting features such as object orientation, cross-platform portability and good programming productivity. In fact, most popular Big Data frameworks such as Hadoop and Spark are implemented in Java or using other languages designed to run on the Java Virtual Machine (JVM) such as Scala. However, modern computing hardware is increasingly complex, featuring multiple processing cores aggregated into one or more CPUs that are usually organized as a Non-Uniform Memory Access (NUMA) architecture. The platform-independent features of the JVM come at the cost of hardware abstraction, which makes it more difficult for Big Data developers to take advantage of hardware-aware optimizations based on managing CPU or NUMA affinities. In this paper we introduce jhwloc, a Java library for easily managing such affinities in JVM-based applications and gathering information about the underlying hardware topology. To demonstrate the functionality and benefits of our proposal, we have extended Flame-MR, our Java-based MapReduce framework, to provide support for setting CPU affinities through jhwloc. The experimental evaluation using representative Big Data workloads has shown that performance can be improved by up to 17% when efficiently exploiting the hardware. jhwloc is publicly available to download at https://github.com/rreye/jhwloc.

SUBMITTER: Krzhizhanovskaya V

PROVIDER: S-EPMC7302232 | biostudies-literature | 2020 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Similar Datasets

Enabling phenotypic big data with PheNorm.

Project description:Objective:Electronic health record (EHR)-based phenotyping infers whether a patient has a disease based on the information in his or her EHR. A human-annotated training set with gold-standard disease status labels is usually required to build an algorithm for phenotyping based on a set of predictive features. The time intensiveness of annotation and feature curation severely limits the ability to achieve high-throughput phenotyping. While previous studies have successfully automated feature curation, annotation remains a major bottleneck. In this paper, we present PheNorm, a phenotyping algorithm that does not require expert-labeled samples for training. Methods:The most predictive features, such as the number of International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes or mentions of the target phenotype, are normalized to resemble a normal mixture distribution with high area under the receiver operating curve (AUC) for prediction. The transformed features are then denoised and combined into a score for accurate disease classification. Results:We validated the accuracy of PheNorm with 4 phenotypes: coronary artery disease, rheumatoid arthritis, Crohn's disease, and ulcerative colitis. The AUCs of the PheNorm score reached 0.90, 0.94, 0.95, and 0.94 for the 4 phenotypes, respectively, which were comparable to the accuracy of supervised algorithms trained with sample sizes of 100-300, with no statistically significant difference. Conclusion:The accuracy of the PheNorm algorithms is on par with algorithms trained with annotated samples. PheNorm fully automates the generation of accurate phenotyping algorithms and demonstrates the capacity for EHR-driven annotations to scale to the next level - phenotypic big data.

| S-EPMC6251688 | biostudies-literature

The ProteomeXchange consortium in 2020: enabling 'big data' approaches in proteomics.

Project description:The ProteomeXchange (PX) consortium of proteomics resources (http://www.proteomexchange.org) has standardized data submission and dissemination of mass spectrometry proteomics data worldwide since 2012. In this paper, we describe the main developments since the previous update manuscript was published in Nucleic Acids Research in 2017. Since then, in addition to the four PX existing members at the time (PRIDE, PeptideAtlas including the PASSEL resource, MassIVE and jPOST), two new resources have joined PX: iProX (China) and Panorama Public (USA). We first describe the updated submission guidelines, now expanded to include six members. Next, with current data submission statistics, we demonstrate that the proteomics field is now actively embracing public open data policies. At the end of June 2019, more than 14 100 datasets had been submitted to PX resources since 2012, and from those, more than 9 500 in just the last three years. In parallel, an unprecedented increase of data re-use activities in the field, including 'big data' approaches, is enabling novel research and new data resources. At last, we also outline some of our future plans for the coming years.

| S-EPMC7145525 | biostudies-literature

Reproducible big data science: A case study in continuous FAIRness.

Project description:Big biomedical data create exciting opportunities for discovery, but make it difficult to capture analyses and outputs in forms that are findable, accessible, interoperable, and reusable (FAIR). In response, we describe tools that make it easy to capture, and assign identifiers to, data and code throughout the data lifecycle. We illustrate the use of these tools via a case study involving a multi-step analysis that creates an atlas of putative transcription factor binding sites from terabytes of ENCODE DNase I hypersensitive sites sequencing data. We show how the tools automate routine but complex tasks, capture analysis algorithms in understandable and reusable forms, and harness fast networks and powerful cloud computers to process data rapidly, all without sacrificing usability or reproducibility-thus ensuring that big data are not hard-to-(re)use data. We evaluate our approach via a user study, and show that 91% of participants were able to replicate a complex analysis involving considerable data volumes.

| S-EPMC6459504 | biostudies-literature

IoT Big-Data Centred Knowledge Granule Analytic and Cluster Framework for BI Applications: A Case Base Analysis.

Project description:The current rapid growth of Internet of Things (IoT) in various commercial and non-commercial sectors has led to the deposition of large-scale IoT data, of which the time-critical analytic and clustering of knowledge granules represent highly thought-provoking application possibilities. The objective of the present work is to inspect the structural analysis and clustering of complex knowledge granules in an IoT big-data environment. In this work, we propose a knowledge granule analytic and clustering (KGAC) framework that explores and assembles knowledge granules from IoT big-data arrays for a business intelligence (BI) application. Our work implements neuro-fuzzy analytic architecture rather than a standard fuzzified approach to discover the complex knowledge granules. Furthermore, we implement an enhanced knowledge granule clustering (e-KGC) mechanism that is more elastic than previous techniques when assembling the tactical and explicit complex knowledge granules from IoT big-data arrays. The analysis and discussion presented here show that the proposed framework and mechanism can be implemented to extract knowledge granules from an IoT big-data array in such a way as to present knowledge of strategic value to executives and enable knowledge users to perform further BI actions.

| S-EPMC4657997 | biostudies-literature

Big Trajectory Data Mining: A Survey of Methods, Applications, and Services.

Project description:The increasingly wide usage of smart infrastructure and location-aware terminals has helped increase the availability of trajectory data with rich spatiotemporal information. The development of data mining and analysis methods has allowed researchers to use these trajectory datasets to identify urban reality (e.g., citizens' collective behavior) in order to solve urban problems in transportation, environment, public security, etc. However, existing studies in this field have been relatively isolated, and an integrated and comprehensive review is lacking the problems that have been tackled, methods that have been tested, and services that have been generated from existing research. In this paper, we first discuss the relationships among the prevailing trajectory mining methods and then, classify the applications of trajectory data into three major groups: social dynamics, traffic dynamics, and operational dynamics. Finally, we briefly discuss the services that can be developed from studies in this field. Practical implications are also delivered for participants in trajectory data mining. With a focus on relevance and association, our review is aimed at inspiring researchers to identify gaps among tested methods and guiding data analysts and planners to select the most suitable methods for specific problems.

| S-EPMC7472055 | biostudies-literature

Using big data in pediatric oncology: Current applications and future directions.

Project description:Pediatric cancer is a rare disease with a low annual incidence, which presents a significant challenge in being able to collect enough data to fuel clinical discoveries. Big data registry trials hold promise to advance the study of pediatric cancers by allowing for the combination of traditional randomized controlled trials with the power of larger cohort sizes. The emergence of big data resources and data-sharing initiatives are becoming transformative for pediatric cancer diagnosis and treatment. This review discusses the uses of big data in pediatric cancer, existing pediatric cancer registry initiatives and research, the challenges in harmonizing these data to improve accessibility for study, and building pediatric data commons and other important future endeavors.

| S-EPMC9036548 | biostudies-literature

Ultra-fast data-mining hardware architecture based on stochastic computing.

Project description:Minimal hardware implementations able to cope with the processing of large amounts of data in reasonable times are highly desired in our information-driven society. In this work we review the application of stochastic computing to probabilistic-based pattern-recognition analysis of huge database sets. The proposed technique consists in the hardware implementation of a parallel architecture implementing a similarity search of data with respect to different pre-stored categories. We design pulse-based stochastic-logic blocks to obtain an efficient pattern recognition system. The proposed architecture speeds up the screening process of huge databases by a factor of 7 when compared to a conventional digital implementation using the same hardware area.

| S-EPMC4425430 | biostudies-literature

Big data in corneal diseases and cataract: Current applications and future directions.

Project description:The accelerated growth in electronic health records (EHR), Internet-of-Things, mHealth, telemedicine, and artificial intelligence (AI) in the recent years have significantly fuelled the interest and development in big data research. Big data refer to complex datasets that are characterized by the attributes of "5 Vs"-variety, volume, velocity, veracity, and value. Big data analytics research has so far benefitted many fields of medicine, including ophthalmology. The availability of these big data not only allow for comprehensive and timely examinations of the epidemiology, trends, characteristics, outcomes, and prognostic factors of many diseases, but also enable the development of highly accurate AI algorithms in diagnosing a wide range of medical diseases as well as discovering new patterns or associations of diseases that are previously unknown to clinicians and researchers. Within the field of ophthalmology, there is a rapidly expanding pool of large clinical registries, epidemiological studies, omics studies, and biobanks through which big data can be accessed. National corneal transplant registries, genome-wide association studies, national cataract databases, and large ophthalmology-related EHR-based registries (e.g., AAO IRIS Registry) are some of the key resources. In this review, we aim to provide a succinct overview of the availability and clinical applicability of big data in ophthalmology, particularly from the perspective of corneal diseases and cataract, the synergistic potential of big data, AI technologies, internet of things, mHealth, and wearable smart devices, and the potential barriers for realizing the clinical and research potential of big data in this field.

| S-EPMC9929069 | biostudies-literature

Biomedical Big Data Technologies, Applications, and Challenges for Precision Medicine: A Review.

Project description:The explosive growth of biomedical Big Data presents both significant opportunities and challenges in the realm of knowledge discovery and translational applications within precision medicine. Efficient management, analysis, and interpretation of big data can pave the way for groundbreaking advancements in precision medicine. However, the unprecedented strides in the automated collection of large-scale molecular and clinical data have also introduced formidable challenges in terms of data analysis and interpretation, necessitating the development of novel computational approaches. Some potential challenges include the curse of dimensionality, data heterogeneity, missing data, class imbalance, and scalability issues. This overview article focuses on the recent progress and breakthroughs in the application of big data within precision medicine. Key aspects are summarized, including content, data sources, technologies, tools, challenges, and existing gaps. Nine fields-Datawarehouse and data management, electronic medical record, biomedical imaging informatics, Artificial intelligence-aided surgical design and surgery optimization, omics data, health monitoring data, knowledge graph, public health informatics, and security and privacy-are discussed.

| S-EPMC10784210 | biostudies-literature

Food composition databases in the era of Big Data: Vegetable oils as a case study.

Project description:Understanding the population's dietary patterns and their impacts on health requires many different sources of information. The development of reliable food composition databases is a key step in this pursuit. With them, nutrition and health care professionals can provide better public health advice and guide society toward achieving a better and healthier life. Unfortunately, these databases are full of caveats. Focusing on the specific case of vegetable oils, we analyzed the possible obsolescence of the information and the differences or inconsistencies among databases. We show that in many cases, the information is limited, incompletely documented, old or unreliable. More importantly, despite the many efforts carried out in the last decades, there is still much work to be done. As such, institutions should develop long-standing programs that can ensure the quality of the information on what we eat in the long term. In the face of climate change and complex societal challenges in an interconnected world, the full diversity of the food system needs to be recognized and more efforts should be put toward achieving a data-driven food system.

| S-EPMC9851468 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data