Network anomaly detection system with optimized DS evidence theory.
Ontology highlight
ABSTRACT:
SUBMITTER: Liu Y
PROVIDER: S-EPMC4165327 | biostudies-other | 2014
REPOSITORIES: biostudies-other
Similar Datasets
Project description:In this paper, to solve the problem of detecting network anomalies, a method of forming a set of informative features formalizing the normal and anomalous behavior of the system on the basis of evaluating the Hurst (H) parameter of the network traffic has been proposed. Criteria to detect and prevent various types of network anomalies using the Three Sigma Rule and Hurst parameter have been defined. A rescaled range (RS) method to evaluate the Hurst parameter has been chosen. The practical value of the proposed method is conditioned by a set of the following factors: low time spent on calculations, short time required for monitoring, the possibility of self-training, as well as the possibility of observing a wide range of traffic types. For new DPI (Deep Packet Inspection) system implementation, algorithms for analyzing and captured traffic with protocol detection and determining statistical load parameters have been developed. In addition, algorithms that are responsible for flow regulation to ensure the QoS (Quality of Services) based on the conducted static analysis of flows and the proposed method of detection of anomalies using the parameter Hurst have been developed. We compared the proposed software DPI system with the existing SolarWinds Deep Packet Inspection for the possibility of network traffic anomaly detection and prevention. The created software components of the proposed DPI system increase the efficiency of using standard intrusion detection and prevention systems by identifying and taking into account new non-standard factors and dependencies. The use of the developed system in the IoT communication infrastructure will increase the level of information security and significantly reduce the risks of its loss.
| S-EPMC7146318 | biostudies-literature
Project description:Currently, expert systems and applied machine learning algorithms are widely used to automate network intrusion detection. In critical infrastructure applications of communication technologies, the interaction among various industrial control systems and the Internet environment intrinsic to the IoT technology makes them susceptible to cyber-attacks. Given the existence of the enormous network traffic in critical Cyber-Physical Systems (CPSs), traditional methods of machine learning implemented in network anomaly detection are inefficient. Therefore, recently developed machine learning techniques, with the emphasis on deep learning, are finding their successful implementations in the detection and classification of anomalies at both the network and host levels. This paper presents an ensemble method that leverages deep models such as the Deep Neural Network (DNN) and Long Short-Term Memory (LSTM) and a meta-classifier (i.e., logistic regression) following the principle of stacked generalization. To enhance the capabilities of the proposed approach, the method utilizes a two-step process for the apprehension of network anomalies. In the first stage, data pre-processing, a Deep Sparse AutoEncoder (DSAE) is employed for the feature engineering problem. In the second phase, a stacking ensemble learning approach is utilized for classification. The efficiency of the method disclosed in this work is tested on heterogeneous datasets, including data gathered in the IoT environment, namely IoT-23, LITNET-2020, and NetML-2020. The results of the evaluation of the proposed approach are discussed. Statistical significance is tested and compared to the state-of-the-art approaches in network anomaly detection.
| S-EPMC7472141 | biostudies-literature
Project description:Relative scores such as Local Outlying Factor and mass ratio have been shown to be better scores than global scores in detecting anomalies. While this is true, our analysis reveals for the first time that these relative scores have a key shortcoming: anomalies have greatly different relative scores if they are located in different regions where the curvatures of the density surface are very different. As a result, the low-score anomalies could be ranked lower than some normal points. This revelation motivates (i) a new score called Neighbourhood Contrast (NC) which produces approximately the same high scores for all anomalies, regardless of varying curvatures of the density surface in different regions; and (ii) an anomaly detection method based on NC. Our experiments show that the proposed method which employs the new score significantly outperforms methods using the aforementioned relative scores on benchmark datasets.
| S-EPMC7206296 | biostudies-literature
Project description:We introduce a network statistic that measures structural properties at the micro-, meso-, and macroscopic scales, while still being easy to compute and interpretable at a glance. Our statistic, the onion spectrum, is based on the onion decomposition, which refines the k-core decomposition, a standard network fingerprinting method. The onion spectrum is exactly as easy to compute as the k-cores: It is based on the stages at which each vertex gets removed from a graph in the standard algorithm for computing the k-cores. Yet, the onion spectrum reveals much more information about a network, and at multiple scales; for example, it can be used to quantify node heterogeneity, degree correlations, centrality, and tree- or lattice-likeness. Furthermore, unlike the k-core decomposition, the combined degree-onion spectrum immediately gives a clear local picture of the network around each node which allows the detection of interesting subgraphs whose topological structure differs from the global network organization. This local description can also be leveraged to easily generate samples from the ensemble of networks with a given joint degree-onion distribution. We demonstrate the utility of the onion spectrum for understanding both static and dynamic properties on several standard graph models and on many real-world networks.
| S-EPMC4989159 | biostudies-literature
Project description:BackgroundStatistical analysis, which has become an integral part of evidence-based medicine, relies heavily on data quality that is of critical importance in modern clinical research. Input data are not only at risk of being falsified or fabricated, but also at risk of being mishandled by investigators.ObjectiveThe urgent need to assure the highest data quality possible has led to the implementation of various auditing strategies designed to monitor clinical trials and detect errors of different origin that frequently occur in the field. The objective of this study was to describe a machine learning-based algorithm to detect anomalous patterns in data created as a consequence of carelessness, systematic error, or intentionally by entering fabricated values.MethodsA particular electronic data capture (EDC) system, which is used for data management in clinical registries, is presented including its architecture and data structure. This EDC system features an algorithm based on machine learning designed to detect anomalous patterns in quantitative data. The detection algorithm combines clustering with a series of 7 distance metrics that serve to determine the strength of an anomaly. For the detection process, the thresholds and combinations of the metrics were used and the detection performance was evaluated and validated in the experiments involving simulated anomalous data and real-world data.ResultsFive different clinical registries related to neuroscience were presented-all of them running in the given EDC system. Two of the registries were selected for the evaluation experiments and served also to validate the detection performance on an independent data set. The best performing combination of the distance metrics was that of Canberra, Manhattan, and Mahalanobis, whereas Cosine and Chebyshev metrics had been excluded from further analysis due to the lowest performance when used as single distance metric-based classifiers.ConclusionsThe experimental results demonstrate that the algorithm is universal in nature, and as such may be implemented in other EDC systems, and is capable of anomalous data detection with a sensitivity exceeding 85%.
| S-EPMC8140384 | biostudies-literature
Project description:Event-based cameras are bio-inspired novel sensors that asynchronously record changes in illumination in the form of events. This principle results in significant advantages over conventional cameras, such as low power utilization, high dynamic range, and no motion blur. Moreover, by design, such cameras encode only the relative motion between the scene and the sensor and not the static background to yield a very sparse data structure. In this paper, we leverage these advantages of an event camera toward a critical vision application-video anomaly detection. We propose an anomaly detection solution in the event domain with a conditional Generative Adversarial Network (cGAN) made up of sparse submanifold convolution layers. Video analytics tasks such as anomaly detection depend on the motion history at each pixel. To enable this, we also put forward a generic unsupervised deep learning solution to learn a novel memory surface known as Deep Learning (DL) memory surface. DL memory surface encodes the temporal information readily available from these sensors while retaining the sparsity of event data. Since there is no existing dataset for anomaly detection in the event domain, we also provide an anomaly detection event dataset with a set of anomalies. We empirically validate our anomaly detection architecture, composed of sparse convolutional layers, on this proposed and online dataset. Careful analysis of the anomaly detection network reveals that the presented method results in a massive reduction in computational complexity with good performance compared to previous state-of-the-art conventional frame-based anomaly detection networks.
| S-EPMC8358807 | biostudies-literature
Project description:The purposes of this study are to propose an unsupervised anomaly detection method based on a deep neural network (DNN) model, which requires only normal images for training, and to evaluate its performance with a large chest radiograph dataset. We used the auto-encoding generative adversarial network (α-GAN) framework, which is a combination of a GAN and a variational autoencoder, as a DNN model. A total of 29,684 frontal chest radiographs from the Radiological Society of North America Pneumonia Detection Challenge dataset were used for this study (16,880 male and 12,804 female patients; average age, 47.0 years). All these images were labeled as "Normal," "No Opacity/Not Normal," or "Opacity" by board-certified radiologists. About 70% (6,853/9,790) of the Normal images were randomly sampled as the training dataset, and the rest were randomly split into the validation and test datasets in a ratio of 1:2 (7,610 and 15,221). Our anomaly detection system could correctly visualize various lesions including a lung mass, cardiomegaly, pleural effusion, bilateral hilar lymphadenopathy, and even dextrocardia. Our system detected the abnormal images with an area under the receiver operating characteristic curve (AUROC) of 0.752. The AUROCs for the abnormal labels Opacity and No Opacity/Not Normal were 0.838 and 0.704, respectively. Our DNN-based unsupervised anomaly detection method could successfully detect various diseases or anomalies in chest radiographs by training with only the normal images.
| S-EPMC8289984 | biostudies-literature
Project description:Cellular frustrated models have been developed to describe how the adaptive immune system works. They are composed by independent agents that continuously pair and unpair depending on the information that one sub-set of these agents display. The emergent dynamics is sensitive to changes in the displayed information and can be used to detect anomalies, which can be important to accomplish the immune system main function of protecting the host. Therefore, it has been hypothesized that these models could be adequate to model the immune system activation. Likewise it has been hypothesized that these models could provide inspiration to develop new artificial intelligence algorithms for data mining applications. However, computational algorithms do not need to follow strictly the immunological reality. Here, we investigate efficient implementation strategies of these immune inspired ideas for anomaly detection applications and use real data to compare the performance of cellular frustration algorithms with standard implementations of one-class support vector machines and deep autoencoders. Our results demonstrate that more efficient implementations of cellular frustration algorithms are possible and also that cellular frustration algorithms can be advantageous for semi-supervised anomaly detection applications given their robustness and accuracy.
| S-EPMC6613704 | biostudies-literature
Project description:Anomaly detection is a challenging task that frequently arises in practically all areas of industry and science, from fraud detection and data quality monitoring to finding rare cases of diseases and searching for new physics. Most of the conventional approaches to anomaly detection, such as one-class SVM and Robust Auto-Encoder, are one-class classification methods, i.e., focus on separating normal data from the rest of the space. Such methods are based on the assumption of separability of normal and anomalous classes, and subsequently do not take into account any available samples of anomalies. Nonetheless, in practical settings, some anomalous samples are often available; however, usually in amounts far lower than required for a balanced classification task, and the separability assumption might not always hold. This leads to an important task-incorporating known anomalous samples into training procedures of anomaly detection models. In this work, we propose a novel model-agnostic training procedure to address this task. We reformulate one-class classification as a binary classification problem with normal data being distinguished from pseudo-anomalous samples. The pseudo-anomalous samples are drawn from low-density regions of a normalizing flow model by feeding tails of the latent distribution into the model. Such an approach allows to easily include known anomalies into the training process of an arbitrary classifier. We demonstrate that our approach shows comparable performance on one-class problems, and, most importantly, achieves comparable or superior results on tasks with variable amounts of known anomalies.
| S-EPMC8627226 | biostudies-literature
Project description:To effectively and accurately detect and classify network intrusion data, this paper introduces a general regression neural network (GRNN) based on the artificial immune algorithm with elitist strategies (AIAE). The elitist archive and elitist crossover were combined with the artificial immune algorithm (AIA) to produce the AIAE-GRNN algorithm, with the aim of improving its adaptivity and accuracy. In this paper, the mean square errors (MSEs) were considered the affinity function. The AIAE was used to optimize the smooth factors of the GRNN; then, the optimal smooth factor was solved and substituted into the trained GRNN. Thus, the intrusive data were classified. The paper selected a GRNN that was separately optimized using a genetic algorithm (GA), particle swarm optimization (PSO), and fuzzy C-mean clustering (FCM) to enable a comparison of these approaches. As shown in the results, the AIAE-GRNN achieves a higher classification accuracy than PSO-GRNN, but the running time of AIAE-GRNN is long, which was proved first. FCM and GA-GRNN were eliminated because of their deficiencies in terms of accuracy and convergence. To improve the running speed, the paper adopted principal component analysis (PCA) to reduce the dimensions of the intrusive data. With the reduction in dimensionality, the PCA-AIAE-GRNN decreases in accuracy less and has better convergence than the PCA-PSO-GRNN, and the running speed of the PCA-AIAE-GRNN was relatively improved. The experimental results show that the AIAE-GRNN has a higher robustness and accuracy than the other algorithms considered and can thus be used to classify the intrusive data.
| S-EPMC4373783 | biostudies-literature