Dataset Information

An efficient approach for low latency processing in stream data.

ABSTRACT: Stream data is the data that is generated continuously from the different data sources and ideally defined as the data that has no discrete beginning or end. Processing the stream data is a part of big data analytics that aims at querying the continuously arriving data and extracting meaningful information from the stream. Although earlier processing of such stream was using batch analytics, nowadays there are applications like the stock market, patient monitoring, and traffic analysis which can cause a drastic difference in processing, if the output is generated in levels of hours and minutes. The primary goal of any real-time stream processing system is to process the stream data as soon as it arrives. Correspondingly, analytics of the stream data also needs consideration of surrounding dependent data. For example, stock market analytics results are often useless if we do not consider their associated or dependent parameters which affect the result. In a real-world application, these dependent stream data usually arrive from the distributed environment. Hence, the stream processing system has to be designed, which can deal with the delay in the arrival of such data from distributed sources. We have designed the stream processing model which can deal with all the possible latency and provide an end-to-end low latency system. We have performed the stock market prediction by considering affecting parameters, such as USD, OIL Price, and Gold Price with an equal arrival rate. We have calculated the Normalized Root Mean Square Error (NRMSE) which simplifies the comparison among models with different scales. A comparative analysis of the experiment presented in the report shows a significant improvement in the result when considering the affecting parameters. In this work, we have used the statistical approach to forecast the probability of possible data latency arrives from distributed sources. Moreover, we have performed preprocessing of stream data to ensure at-least-once delivery semantics. In the direction towards providing low latency in processing, we have also implemented exactly-once processing semantics. Extensive experiments have been performed with varying sizes of the window and data arrival rate. We have concluded that system latency can be reduced when the window size is equal to the data arrival rate.

SUBMITTER: Bhatt N

PROVIDER: S-EPMC7959664 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:In recent years, the application and wide adoption of Internet of Things (IoT)-based technologies have increased the proliferation of monitoring systems, which has consequently exponentially increased the amounts of heterogeneous data generated. Processing and analysing the massive amount of data produced is cumbersome and gradually moving from classical 'batch' processing-extract, transform, load (ETL) technique to real-time processing. For instance, in environmental monitoring and management domain, time-series data and historical dataset are crucial for prediction models. However, the environmental monitoring domain still utilises legacy systems, which complicates the real-time analysis of the essential data, integration with big data platforms and reliance on batch processing. Herein, as a solution, a distributed stream processing middleware framework for real-time analysis of heterogeneous environmental monitoring and management data is presented and tested on a cluster using open source technologies in a big data environment. The system ingests datasets from legacy systems and sensor data from heterogeneous automated weather systems irrespective of the data types to Apache Kafka topics using Kafka Connect APIs for processing by the Kafka streaming processing engine. The stream processing engine executes the predictive numerical models and algorithms represented in event processing (EP) languages for real-time analysis of the data streams. To prove the feasibility of the proposed framework, we implemented the system using a case study scenario of drought prediction and forecasting based on the Effective Drought Index (EDI) model. Firstly, we transform the predictive model into a form that could be executed by the streaming engine for real-time computing. Secondly, the model is applied to the ingested data streams and datasets to predict drought through persistent querying of the infinite streams to detect anomalies. As a conclusion of this study, a performance evaluation of the distributed stream processing middleware infrastructure is calculated to determine the real-time effectiveness of the framework.

Project description:The oil and gas industries (OGI) are the primary global energy source, with pipelines as vital components for OGI transportation. However, pipeline leaks pose significant risks, including fires, injuries, environmental harm, and property damage. Therefore, maintaining an effective pipeline maintenance system is critical for ensuring a safe and sustainable energy supply. The Internet of Things (IoT) has emerged as a cutting-edge technology for efficient OGI pipeline leak detection. However, deploying IoT in OGI monitoring faces significant challenges due to hazardous environments and limited communication infrastructure. Energy efficiency and fault tolerance, typical IoT concerns, gain heightened importance in the OGI context. In OGI monitoring, IoT devices are linearly deployed with no alternative communication mechanism available along OGI pipelines. Thus, the absence of both communication routes can disrupt crucial data transmission. Therefore, ensuring energy-efficient and fault-tolerant communication for OGI data is paramount. Critical data needs to reach the control center on time for faster actions to avoid loss. Low latency communication for critical data is another challenge of the OGI monitoring environment. Moreover, IoT devices gather a plethora of OGI parameter data including redundant values that hold no relevance for transmission to the control center. Thus, optimizing data transmission is essential to conserve energy in OGI monitoring. This article presents the Priority-Based, Energy-Efficient, and Optimal Data Routing Protocol (PO-IMRP) to tackle these challenges. The energy model and congestion control mechanism optimize data packets for an energy-efficient and congestion-free network. In PO-IMRP, nodes are aware of their energy status and communicate node's depletion status timely for network robustness. Priority-based routing selects low-latency routes for critical data to avoid OGI losses. Comparative analysis against linear LEACH highlights PO-IMRP's superior performance in terms of total packet transmission by completing fewer rounds with more packet's transmissions, attributed to the packet optimization technique implemented at each hop, which helps mitigate network congestion. MATLAB simulations affirm the effectiveness of the protocol in terms of energy efficiency, fault-tolerance, and low latency communication.

Dataset Information

An efficient approach for low latency processing in stream data.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets