Meteorological data analysis using MapReduce.
Ontology highlight
ABSTRACT:
SUBMITTER: Fang W
PROVIDER: S-EPMC3953661 | biostudies-other | 2014
REPOSITORIES: biostudies-other
Similar Datasets
Project description:BackgroundHigh-throughput molecular profiling data has been used to improve clinical decision making by stratifying subjects based on their molecular profiles. Unsupervised clustering algorithms can be used for stratification purposes. However, the current speed of the clustering algorithms cannot meet the requirement of large-scale molecular data due to poor performance of the correlation matrix calculation. With high-throughput sequencing technologies promising to produce even larger datasets per subject, we expect the performance of the state-of-the-art statistical algorithms to be further impacted unless efforts towards optimisation are carried out. MapReduce is a widely used high performance parallel framework that can solve the problem.ResultsIn this paper, we evaluate the current parallel modes for correlation calculation methods and introduce an efficient data distribution and parallel calculation algorithm based on MapReduce to optimise the correlation calculation. We studied the performance of our algorithm using two gene expression benchmarks. In the micro-benchmark, our implementation using MapReduce, based on the R package RHIPE, demonstrates a 3.26-5.83 fold increase compared to the default Snowfall and 1.56-1.64 fold increase compared to the basic RHIPE in the Euclidean, Pearson and Spearman correlations. Though vanilla R and the optimised Snowfall outperforms our optimised RHIPE in the micro-benchmark, they do not scale well with the macro-benchmark. In the macro-benchmark the optimised RHIPE performs 2.03-16.56 times faster than vanilla R. Benefiting from the 3.30-5.13 times faster data preparation, the optimised RHIPE performs 1.22-1.71 times faster than the optimised Snowfall. Both the optimised RHIPE and the optimised Snowfall successfully performs the Kendall correlation with TCGA dataset within 7 hours. Both of them conduct more than 30 times faster than the estimated vanilla R.ConclusionsThe performance evaluation found that the new MapReduce algorithm and its implementation in RHIPE outperforms vanilla R and the conventional parallel algorithms implemented in R Snowfall. We propose that MapReduce framework holds great promise for large molecular data analysis, in particular for high-dimensional genomic data such as that demonstrated in the performance evaluation described in this paper. We aim to use this new algorithm as a basis for optimising high-throughput molecular data correlation calculation for Big Data.
| S-EPMC4246436 | biostudies-literature
Project description:Next-generation DNA sequencing (NGS) projects, such as the 1000 Genomes Project, are already revolutionizing our understanding of genetic variation among individuals. However, the massive data sets generated by NGS--the 1000 Genome pilot alone includes nearly five terabases--make writing feature-rich, efficient, and robust analysis tools difficult for even computationally sophisticated individuals. Indeed, many professionals are limited in the scope and the ease with which they can answer scientific questions by the complexity of accessing and manipulating the data produced by these machines. Here, we discuss our Genome Analysis Toolkit (GATK), a structured programming framework designed to ease the development of efficient and robust analysis tools for next-generation DNA sequencers using the functional programming philosophy of MapReduce. The GATK provides a small but rich set of data access patterns that encompass the majority of analysis tool needs. Separating specific analysis calculations from common data management infrastructure enables us to optimize the GATK framework for correctness, stability, and CPU and memory efficiency and to enable distributed and shared memory parallelization. We highlight the capabilities of the GATK by describing the implementation and application of robust, scale-tolerant tools like coverage calculators and single nucleotide polymorphism (SNP) calling. We conclude that the GATK programming framework enables developers and analysts to quickly and easily write efficient and robust NGS tools, many of which have already been incorporated into large-scale sequencing projects like the 1000 Genomes Project and The Cancer Genome Atlas.
| S-EPMC2928508 | biostudies-literature
Project description:MotivationPost-sequencing DNA analysis typically consists of read mapping followed by variant calling. Especially for whole genome sequencing, this computational step is very time-consuming, even when using multithreading on a multi-core machine.ResultsWe present Halvade, a framework that enables sequencing pipelines to be executed in parallel on a multi-node and/or multi-core compute infrastructure in a highly efficient manner. As an example, a DNA sequencing analysis pipeline for variant calling has been implemented according to the GATK Best Practices recommendations, supporting both whole genome and whole exome sequencing. Using a 15-node computer cluster with 360 CPU cores in total, Halvade processes the NA12878 dataset (human, 100 bp paired-end reads, 50× coverage) in <3 h with very high parallel efficiency. Even on a single, multi-core machine, Halvade attains a significant speedup compared with running the individual tools with multithreading.
| S-EPMC4514927 | biostudies-literature
Project description:Retrieving historical fine particulate matter (PM2.5) data is key for evaluating the long-term impacts of PM2.5 on the environment, human health and climate change. Satellite-based aerosol optical depth has been used to estimate PM2.5, but estimations have largely been undermined by massive missing values, low sampling frequency and weak predictive capability. Here, using a novel feature engineering approach to incorporate spatial effects from meteorological data, we developed a robust LightGBM model that predicts PM2.5 at an unprecedented predictive capacity on hourly (R2 = 0.75), daily (R2 = 0.84), monthly (R2 = 0.88) and annual (R2 = 0.87) timescales. By taking advantage of spatial features, our model can also construct hourly gridded networks of PM2.5. This capability would be further enhanced if meteorological observations from regional stations were incorporated. Our results show that this model has great potential in reconstructing historical PM2.5 datasets and real-time gridded networks at high spatial-temporal resolutions. The resulting datasets can be assimilated into models to produce long-term re-analysis that incorporates interactions between aerosols and physical processes.
| S-EPMC8566180 | biostudies-literature
Project description:ObjectivesThe main purpose of this publication is to help users (students, researchers, farmers, advisors, etc.) of weather data with agronomic purposes (e.g. crop yield forecast) to retrieve and process gridded weather data from different Application Programming Interfaces (API client) sources using R software.Data descriptionThis publication consists of a code-tutorial developed in R that is part of the data-curation process from numerous research projects carried out by the Ciampitti's Lab, Department of Agronomy, Kansas State University. We make use of three weather databases for which specific libraries were developed in R language: (i) DAYMET (Thornton et al. in https://daymet.ornl.gov/ , 2019; https://github.com/bluegreen-labs/daymetr ), (ii) NASA-POWER (Sparks in J Open Source Softw 3:1035, 2018; https://github.com/ropensci/nasapower ), and (iii) Climate Hazards Group InfraRed Precipitation with Station Data (CHIRPS) (Funk et al. in Sci Data 2:150066, 2015; https://github.com/ropensci/chirps ). The databases offer different weather variables, and vary in terms of spatio-temporal coverage and resolution. The tutorial shows and explain how to retrieve weather data from multiple locations at once using latitude and longitude coordinates. Additionally, it offers the possibility to create relevant variables and summaries that are of agronomic interest such as Shannon Diversity Index (SDI) of precipitation, abundant and well distributed rainfall (AWDR), growing degree days (GDD), crop heat units (CHU), extreme precipitation (EPE) and temperature events (ETE), reference evapotranspiration (ET0), among others.
| S-EPMC8157746 | biostudies-literature
Project description:MotivationPublic archives contain thousands of trillions of bases of valuable sequencing data. More than 40% of the Sequence Read Archive is human data protected by provisions such as dbGaP. To analyse dbGaP-protected data, researchers must typically work with IT administrators and signing officials to ensure all levels of security are implemented at their institution. This is a major obstacle, impeding reproducibility and reducing the utility of archived data.ResultsWe present a protocol and software tool for analyzing protected data in a commercial cloud. The protocol, Rail-dbGaP, is applicable to any tool running on Amazon Web Services Elastic MapReduce. The tool, Rail-RNA v0.2, is a spliced aligner for RNA-seq data, which we demonstrate by running on 9662 samples from the dbGaP-protected GTEx consortium dataset. The Rail-dbGaP protocol makes explicit for the first time the steps an investigator must take to develop Elastic MapReduce pipelines that analyse dbGaP-protected data in a manner compliant with NIH guidelines. Rail-RNA automates implementation of the protocol, making it easy for typical biomedical investigators to study protected RNA-seq data, regardless of their local IT resources or expertise.Availability and implementationRail-RNA is available from http://rail.bio Technical details on the Rail-dbGaP protocol as well as an implementation walkthrough are available at https://github.com/nellore/rail-dbgap Detailed instructions on running Rail-RNA on dbGaP-protected data using Amazon Web Services are available at http://docs.rail.bio/dbgap/Contacts: anellore@gmail.com or langmea@cs.jhu.eduSupplementary informationSupplementary data are available at Bioinformatics online.
| S-EPMC4978928 | biostudies-literature
Project description:Central African countries may bear high climate change-related infectious disease burdens because of preexisting high rates of disease, poor healthcare infrastructure, land use changes, and high environmental change vulnerabilities. However, making connections between climate and infectious diseases in this region is hampered by the paucity of high-quality meteorological data. This review analyzes the sources and quality of meteorological data used to study the interactions between weather and infectious diseases in Central African countries. Results show that 23% of studies used meteorological data that mismatched with the disease spatial scale of interest. Use of inappropriate weather data was most frequently identified in analyses using meteorological station data or gridded data products. These findings have implications for the interpretation of existing analyses and provide guidance for the use of climate data in future analyses of the connections between meteorology and infectious diseases in Central Africa.
| S-EPMC5184398 | biostudies-literature
Project description:Droughts have severely affected Afghanistan over the last four decades, leading to critical food shortages where two-thirds of the country’s population are in a food crisis. Long years of conflict have lowered the country’s ability to deal with hazards such as drought which can rapidly escalate into disasters. Understanding the spatial and temporal distribution of droughts is needed to be able to respond effectively to disasters and plan for future occurrences. This study used Standardized Precipitation Evapotranspiration Index (SPEI) at monthly, seasonal and annual temporal scales to map the spatiotemporal change dynamics of drought characteristics (distribution, frequency, duration and severity) in Afghanistan. SPEI indices were mapped for river basins, disaggregated into 189 sub-catchments, using monthly precipitation and potential evapotranspiration derived from temperature station observations from 1980 to 2017. The results show these multi-dimensional drought characteristics vary along different years, change among sub-catchments, and differ across temporal scales. During the 38 years, the driest decade and period are 2000s and 1999–2022, respectively. The 2000–01 water year is the driest with the whole country experiencing ‘severe’ to ‘extreme’ drought, more than 53% (87 sub-catchments) suffering the worst drought in history, and about 58% (94 sub-catchments) having ‘very frequent’ drought (7 to 8 months) or ‘extremely frequent’ drought (9 to 10 months). The estimated seasonal duration and severity present significant variations across the study area and among the study period. The nation also suffers from recurring droughts with varying length and intensity in 2004, 2006, 2008 and most recently 2011. There is a trend towards increasing drought with longer duration and higher severity extending all over sub-catchments from southeast to north and central regions. These datasets and maps help to fill the knowledge gap on detailed sub-catchment scale meteorological drought characteristics in Afghanistan. The study findings improve our understanding of the influences of climate change on the drought dynamics and can guide catchment planning for reliable adaptation to and mitigation against future droughts.
| S-EPMC9901756 | biostudies-literature
Project description:BackgroundStorage of genomic data is a major cost for the Life Sciences, effectively addressed via specialized data compression methods. For the same reasons of abundance in data production, the use of Big Data technologies is seen as the future for genomic data storage and processing, with MapReduce-Hadoop as leaders. Somewhat surprisingly, none of the specialized FASTA/Q compressors is available within Hadoop. Indeed, their deployment there is not exactly immediate. Such a State of the Art is problematic.ResultsWe provide major advances in two different directions. Methodologically, we propose two general methods, with the corresponding software, that make very easy to deploy a specialized FASTA/Q compressor within MapReduce-Hadoop for processing files stored on the distributed Hadoop File System, with very little knowledge of Hadoop. Practically, we provide evidence that the deployment of those specialized compressors within Hadoop, not available so far, results in better space savings, and even in better execution times over compressed data, with respect to the use of generic compressors available in Hadoop, in particular for FASTQ files. Finally, we observe that these results hold also for the Apache Spark framework, when used to process FASTA/Q files stored on the Hadoop File System.ConclusionsOur Methods and the corresponding software substantially contribute to achieve space and time savings for the storage and processing of FASTA/Q files in Hadoop and Spark. Being our approach general, it is very likely that it can be applied also to FASTA/Q compression methods that will appear in the future.AvailabilityThe software and the datasets are available at https://github.com/fpalini/fastdoopc.
| S-EPMC7986029 | biostudies-literature
Project description:This study aimed to evaluate the impact of weather conditions and air pollution on the onset of sudden sensorineural hearing loss (SSNHL). The Korean Health Insurance Review and Assessment Service - National Sample Cohort (HIRA-NSC) from 2002 through 2013 was used. A total of 5,200 participants with SSNHL were matched 1:4 for age, sex, income, region of residence, hypertension, diabetes, and dyslipidemia with 20,800 control participants. Meteorological data included daily mean temperature (°C), daily highest temperature (°C), daily lowest temperature (°C), daily temperature difference (°C), relative humidity (%), ambient atmospheric pressure (hPa), pressure, SO2 (ppm), NO2 (ppm), O3 (ppm), CO (ppm), and PM10 (μg/m3) of a mean of 60 days, 30 days, 14 days, 7 days, and 3 days before SSNHL were analyzed. Hourly measurements were taken from 94 places to assess the temperature, humidity, and atmospheric pressure and from 273 places to determine SO2, NO2, O3, CO, and PM10. Crude and adjusted odds ratios (ORs) and 95% confidence intervals (CIs) of meteorological data for SSNHL were analyzed using unconditional logistic regression analyses. Subgroup analyses were conducted by age and sex. The mean NO2 and O3 concentrations 14 days before the index date were different in the SSNHL group compared to those in the control group (P < 0.001 for NO2 and P = 0.021 for O3). The adjusted 14-day OR for NO2 (0.1 ppm) exposure was 3.12 in the SSNHL group compared to that in the control group (95% CI = 2.16-4.49, P < 0.001). The increased odds of NO2 exposure for 14 days in the SSNHL group persisted in the age group older than 30 years for both sexes. Other meteorological conditions did not show differences between the SSNHL and control groups. SSNHL was associated with high concentrations of NO2.
| S-EPMC6547844 | biostudies-literature