Dataset Information

LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things.

ABSTRACT: Log anomaly detection is an efficient method to manage modern large-scale Internet of Things (IoT) systems. More and more works start to apply natural language processing (NLP) methods, and in particular word2vec, in the log feature extraction. Word2vec can extract the relevance between words and vectorize the words. However, the computing cost of training word2vec is high. Anomalies in logs are dependent on not only an individual log message but also on the log message sequence. Therefore, the vector of words from word2vec can not be used directly, which needs to be transformed into the vector of log events and further transformed into the vector of log sequences. To reduce computational cost and avoid multiple transformations, in this paper, we propose an offline feature extraction model, named LogEvent2vec, which takes the log event as input of word2vec to extract the relevance between log events and vectorize log events directly. LogEvent2vec can work with any coordinate transformation methods and anomaly detection models. After getting the log event vector, we transform log event vector to log sequence vector by bary or tf-idf and three kinds of supervised models (Random Forests, Naive Bayes, and Neural Networks) are trained to detect the anomalies. We have conducted extensive experiments on a real public log dataset from BlueGene/L (BGL). The experimental results demonstrate that LogEvent2vec can significantly reduce computational time by 30 times and improve accuracy, comparing with word2vec. LogEvent2vec with bary and Random Forest can achieve the best F1-score and LogEvent2vec with tf-idf and Naive Bayes needs the least computational time.

SUBMITTER: Wang J

PROVIDER: S-EPMC7249657 | biostudies-literature | 2020 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things.

Wang Jin J Tang Yangning Y He Shiming S Zhao Changqing C Sharma Pradip Kumar PK Alfarraj Osama O Tolba Amr A

Sensors (Basel, Switzerland) 20200426 9

Log anomaly detection is an efficient method to manage modern large-scale Internet of Things (IoT) systems. More and more works start to apply natural language processing (NLP) methods, and in particular word2vec, in the log feature extraction. Word2vec can extract the relevance between words and vectorize the words. However, the computing cost of training word2vec is high. Anomalies in logs are dependent on not only an individual log message but also on the log message sequence. Therefore, the ...[more]

PMID: 32357404

Similar Datasets

Project description:BackgroundDemographic changes and a low birth rate have led to a workforce shortage in Japan. To address this issue, the government has promoted engagement of female employment. However, increased female employment can impact women's health. Using Internet of Things (IoT) and apps to manage women's health has gained attention, but few studies have focused on working women.ObjectiveThis study aimed to clarify the current situation of working women and their use of IoT or apps to manage their health.MethodsA large-scale, nationwide internet survey was conducted among 10,000 female participants aged from 20 years to 64 years in Japan. Participants were recruited from a marketing research company's active survey panel of 5.24 million members. The survey included questions about health status, sociodemographic factors, psychological characteristics, and the use of IoT or apps for health management. We compared perceived health status and reasons for current IoT use using t tests and assessed participant characteristics that predicted IoT use using the C5.0 decision tree algorithm. Ethical approval was granted by St. Luke's International University.ResultsAmong participants, 14.6% (1455/10,000) currently used IoT or apps, 7% (695/10,000) used them previously, and 78.5% (7850/10,000) had never used them. Current users (42.7 years old) were older than past users (39.7 years old). Discrepancies were observed between participants' perceived health problems and the purpose for using IoT or apps, with 21.3% (2130/10,000) of all women reporting they experienced menstrual symptoms or disorders but only 3.5% (347/10,000) used IoT or apps to manage the same symptom. On the other hand, current users were more likely to use IoT or apps to manage nutrition-related problems such as underweight or obesity (405/1455, 27.8%). Device use was highest among current users, with 87.3% (1270/1455) using smartphones, 19.7% (287/1455) using smartwatches, and 13.3% (194/1455) using PCs. Decision tree analysis identified 6 clusters, the largest consisting of 81.6% (5323/6523) of non-IoT users who did not exercise regularly, while pregnant women were more likely to use IoT or apps.ConclusionsOur findings highlight the idea that woman with particular health problems (ie, menstrual symptoms or disorders and premenstrual syndrome) have lower use of IoT or apps, suggesting an unmet need for IoT and apps in specific areas.

Project description:BackgroundOne of the main concerns of public health surveillance is to preserve the physical and mental health of older adults while supporting their independence and privacy. On the other hand, to better assist those individuals with essential health care services in the event of an emergency, their regular activities should be monitored. Internet of Things (IoT) sensors may be employed to track the sequence of activities of individuals via ambient sensors, providing real-time insights on daily activity patterns and easy access to the data through the connected ecosystem. Previous surveys to identify the regular activity patterns of older adults were deficient in the limited number of participants, short period of activity tracking, and high reliance on predefined normal activity.ObjectiveThe objective of this study was to overcome the aforementioned challenges by performing a pilot study to evaluate the utilization of large-scale data from smart home thermostats that collect the motion status of individuals for every 5-minute interval over a long period of time.MethodsFrom a large-scale dataset, we selected a group of 30 households who met the inclusion criteria (having at least 8 sensors, being connected to the system for at least 355 days in 2018, and having up to 4 occupants). The indoor activity patterns were captured through motion sensors. We used the unsupervised, time-based, deep neural-network architecture long short-term memory-variational autoencoder to identify the regular activity pattern for each household on 2 time scales: annual and weekday. The results were validated using 2019 records. The area under the curve as well as loss in 2018 were compatible with the 2019 schedule. Daily abnormal behaviors were identified based on deviation from the regular activity model.ResultsThe utilization of this approach not only enabled us to identify the regular activity pattern for each household but also provided other insights by assessing sleep behavior using the sleep time and wake-up time. We could also compare the average time individuals spent at home for the different days of the week. From our study sample, there was a significant difference in the time individuals spent indoors during the weekend versus on weekdays.ConclusionsThis approach could enhance individual health monitoring as well as public health surveillance. It provides a potentially nonobtrusive tool to assist public health officials and governments in policy development and emergency personnel in the event of an emergency by measuring indoor behavior while preserving privacy and using existing commercially available thermostat equipment.

Dataset Information

LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things.

Publications

LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets